如何使用 boto3 将 S3 对象保存到文件

问题描述

我正在尝试使用适用于 AWS 的新 boto3 客户端创建hello world".

I'm trying to do a "hello world" with new boto3 client for AWS.

我的用例相当简单:从 S3 获取对象并将其保存到文件中.

The use-case I have is fairly simple: get object from S3 and save it to the file.

在 boto 2.X 中，我会这样做:

In boto 2.X I would do it like this:

import boto
key = boto.connect_s3().get_bucket('foo').get_key('foo')
key.get_contents_to_filename('/tmp/foo')

在 boto 3 中.我找不到做同样事情的干净方法，所以我手动迭代Streaming"对象:

In boto 3 . I can't find a clean way to do the same thing, so I'm manually iterating over the "Streaming" object:

import boto3
key = boto3.resource('s3').Object('fooo', 'docker/my-image.tar.gz').get()
with open('/tmp/my-image.tar.gz', 'w') as f:
    chunk = key['Body'].read(1024*8)
    while chunk:
        f.write(chunk)
        chunk = key['Body'].read(1024*8)

或

import boto3
key = boto3.resource('s3').Object('fooo', 'docker/my-image.tar.gz').get()
with open('/tmp/my-image.tar.gz', 'w') as f:
    for chunk in iter(lambda: key['Body'].read(4096), b''):
        f.write(chunk)

而且它工作正常.我想知道是否有任何本机"boto3 函数可以完成相同的任务?

And it works fine. I was wondering is there any "native" boto3 function that will do the same task?

推荐答案

Boto3 最近进行了一项定制，有助于解决此问题(除其他外).它目前暴露在低级 S3 客户端上，可以这样使用:

There is a customization that went into Boto3 recently which helps with this (among other things). It is currently exposed on the low-level S3 client, and can be used like this:

s3_client = boto3.client('s3')
open('hello.txt').write('Hello, world!')

# Upload the file to S3
s3_client.upload_file('hello.txt', 'MyBucket', 'hello-remote.txt')

# Download the file from S3
s3_client.download_file('MyBucket', 'hello-remote.txt', 'hello2.txt')
print(open('hello2.txt').read())

这些函数将自动处理读取/写入文件以及对大文件并行执行分段上传.

These functions will automatically handle reading/writing files as well as doing multipart uploads in parallel for large files.

请注意，s3_client.download_file 不会创建目录.它可以创建为 pathlib.Path('/path/to/file.txt').parent.mkdir(parents=True,exist_ok=True).

Note that s3_client.download_file won't create a directory. It can be created as pathlib.Path('/path/to/file.txt').parent.mkdir(parents=True, exist_ok=True).

这篇关于如何使用 boto3 将 S3 对象保存到文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！