问题描述
我正在尝试将 CSV 文件从私有 S3 存储桶读取到 Pandas 数据帧:
I'm trying to read a CSV file from a private S3 bucket to a pandas dataframe:
df = pandas.read_csv('s3://mybucket/file.csv')
我可以从公共存储桶读取文件,但从私有存储桶读取文件会导致 HTTP 403: Forbidden 错误.
I can read a file from a public bucket, but reading a file from a private bucket results in HTTP 403: Forbidden error.
我已经使用 aws configure 配置了 AWS 凭证.
I have configured the AWS credentials using aws configure.
我可以使用 boto3 从私有存储桶下载文件,它使用 aws 凭据.看来我需要配置 pandas 才能使用 AWS 凭证,但不知道如何.
I can download a file from a private bucket using boto3, which uses aws credentials. It seems that I need to configure pandas to use AWS credentials, but don't know how.
推荐答案
Pandas 在 read_csv
中使用 boto
(不是 boto3
).您或许可以安装 boto 并使其正常工作.
Pandas uses boto
(not boto3
) inside read_csv
. You might be able to install boto and have it work correctly.
boto 和 python 3.4.4/python3.5.1 存在一些问题.如果您在这些平台上,并且在修复这些平台之前,您可以将 boto 3 用作
There's some troubles with boto and python 3.4.4 / python3.5.1. If you're on those platforms, and until those are fixed, you can use boto 3 as
import boto3
import pandas as pd
s3 = boto3.client('s3')
obj = s3.get_object(Bucket='bucket', Key='key')
df = pd.read_csv(obj['Body'])
那个 obj
有一个 .read
方法(它返回一个字节流),这对于 Pandas 来说已经足够了.
That obj
had a .read
method (which returns a stream of bytes), which is enough for pandas.
这篇关于从私有 S3 存储桶读取文件到 Pandas 数据帧的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!