如何从S3加载一个pickle文件以在AWS Lambda中使用?

问题描述

我目前正在尝试将S3中的腌制文件加载到AWS Lambda中并将其存储到列表中(腌制是列表).

I am currently trying to load a pickled file from S3 into AWS lambda and store it to a list (the pickle is a list).

这是我的代码:

import pickle
import boto3

s3 = boto3.resource('s3')
with open('oldscreenurls.pkl', 'rb') as data:
    old_list = s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)

即使文件存在，我也会收到以下错误消息:

I get the following error even though the file exists:

FileNotFoundError: [Errno 2] No such file or directory: 'oldscreenurls.pkl'

有什么想法吗?

推荐答案

如 download_fileobj ，您需要以二进制 write 模式打开文件并首先保存到文件.下载文件后，您可以将其打开以进行读取和修复.

As shown in the documentation for download_fileobj, you need to open the file in binary write mode and save to the file first. Once the file is downloaded, you can open it for reading and unpickle.

import pickle
import boto3

s3 = boto3.resource('s3')
with open('oldscreenurls.pkl', 'wb') as data:
    s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)

with open('oldscreenurls.pkl', 'rb') as data:
    old_list = pickle.load(data)

download_fileobj将S3中的对象名称加到本地文件的句柄中，然后将该对象的内容保存到文件中.此功能还有一个版本，称为 ，它将使用文件名而不是打开的文件句柄，并为您打开文件.

download_fileobj takes the name of an object in S3 plus a handle to a local file, and saves the contents of that object to the file. There is also a version of this function called download_file that takes a filename instead of an open file handle and handles opening it for you.

在这种情况下，最好使用 S3Client.get_object ，但要避免先写入然后立即读取文件.您还可以写入内存中的BytesIO对象，该对象的行为类似于文件，但实际上并不接触磁盘.看起来像这样:

In this case it would probably be better to use S3Client.get_object though, to avoid having to write and then immediately read a file. You could also write to an in-memory BytesIO object, which acts like a file but doesn't actually touch a disk. That would look something like this:

import pickle
import boto3
from io import BytesIO

s3 = boto3.resource('s3')
with BytesIO() as data:
    s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)
    data.seek(0)    # move back to the beginning after writing
    old_list = pickle.load(data)

这篇关于如何从S3加载一个pickle文件以在AWS Lambda中使用?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！