java - Java，您如何有效地对大型输入流进行分块？

我的输入流可能为20-30mb。我正在尝试将块作为多部分文件上传到S3。

我有可用的内容长度，也有可用的输入流。我如何有效地做到这一点并记住记忆。

我看到有人做了这样的事情，但不确定我是否完全理解：

    int contentLength = inputStreamMetadata.getContentLength();
    int partSize = 512 * 1024; // Set part size to 2 MB
    int filePosition = 0;

    ByteArrayInputStream bais = inputStreamMetadata.getInputStream();
    List<PartETag> partETags = new ArrayList<>();
    byte[] chunkedFileBytes = new byte[partSize];
    for (int i = 1; filePosition < contentLength; i++) {
      // Because the last part could be less than 5 MB, adjust the part size as needed.
      partSize = Math.min(partSize, (contentLength - filePosition));

      filePosition += bais.read(chunkedFileBytes, filePosition, partSize);

      // Create the request to upload a part.
      UploadPartRequest uploadRequest = new UploadPartRequest()
          .withBucketName(bucketName)
          .withUploadId(uploadId)
          .withKey(fileName)
          .withPartNumber(i)
          .withInputStream(new ByteArrayInputStream(chunkedFileBytes, 0, partSize))
          .withPartSize(partSize);

      UploadPartResult uploadResult = client.uploadPart(uploadRequest);
      partETags.add(uploadResult.getPartETag());
    }
}

具体来说就是：.withInputStream(new ByteArrayInputStream(bytes, 0, bytesRead))

最佳答案

抱歉，我无法（轻松）对其进行测试，但是我认为您真的很接近，...只需“修复”并“安排”您的循环！

将https://stackoverflow.com/a/22128215/592355与您的最新代码结合在一起：

int partSize = 5 * 1024 * 1024; // Set part size to 5 MB
ByteArrayInputStream bais = inputStreamMetadata.getInputStream();
List<PartETag> partETags = new ArrayList<>();
byte[] buff = new byte[partSize];
int partNumber = 1;
while (true) {//!
    int readBytes = bais.read(buff);// readBytes in [-1 .. partSize]!
    if (readBytes == -1) { //EOF
        break;
    }
    // Create the request to upload a part.
    UploadPartRequest uploadRequest = new UploadPartRequest()
                .withBucketName(bucketName)
                .withUploadId(uploadId)
                .withKey(fileName)
                .withPartNumber(partNumber++)
                .withInputStream(new ByteArrayInputStream(buff, 0, readBytes))
                .withPartSize(readBytes);

    UploadPartResult uploadResult = client.uploadPart(uploadRequest);
    partETags.add(uploadResult.getPartETag());
}
// Complete the multipart upload....
// https://docs.aws.amazon.com/AmazonS3/latest/dev/llJavaUploadFile.html