读取大文本文件时出现 Java OutOfMemoryError

问题描述

我是 Java 新手，正在阅读非常大的文件，需要一些帮助来理解问题并解决它.我们有一些遗留代码，必须对其进行优化才能使其正常运行.文件大小可以从 10mb 到 10gb 不等.只有当文件开始超过 800mb 大小时才会开始.

I'm new to Java and working on reading very large files, need some help to understand the problem and solve it. We have got some legacy code which have to be optimized to make it run properly.The file size can vary from 10mb to 10gb only. only trouble start when file starting beyond 800mb size.

InputStream inFileReader = channelSFtp.get(path); // file reading from ssh.
byte[] localbuffer = new byte[2048];
ByteArrayOutputStream bArrStream = new ByteArrayOutputStream();

int i = 0;
while (-1 != (i = inFileReader.read(buffer))) {
bArrStream.write(localbuffer, 0, i);
}

byte[] data = bArrStream.toByteArray();
inFileReader.close();
bos.close();

我们收到错误

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2271)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)

任何帮助将不胜感激?

推荐答案

尝试使用 java.nio.MappedByteBuffer.

http://docs.oracle.com/javase/7/docs/api/java/nio/MappedByteBuffer.html

您无需手动复制即可将文件内容映射到内存中.高级操作系统提供内存映射，Java 有 API 来利用该功能.

You can map a file's content onto memory without copying it manually. High-level Operating Systems offer memory-mapping and Java has API to utilize the feature.

如果我的理解是正确的，内存映射不会将文件的全部内容加载到内存中(意思是根据需要加载和卸载部分内容")，所以我猜 10GB 的文件不会占用你的内存.

If my understanding is correct, memory-mapping does not load a file's entire content onto memory (meaning "loaded and unloaded partially as necessary"), so I guess a 10GB file won't eat up your memory.

这篇关于读取大文本文件时出现 Java OutOfMemoryError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！