从私有 S3 存储桶读取文件到 Pandas 数据帧

问题描述

我正在尝试将 CSV 文件从私有 S3 存储桶读取到 Pandas 数据帧:

I'm trying to read a CSV file from a private S3 bucket to a pandas dataframe:

df = pandas.read_csv('s3://mybucket/file.csv')

我可以从公共存储桶读取文件，但从私有存储桶读取文件会导致 HTTP 403: Forbidden 错误.

I can read a file from a public bucket, but reading a file from a private bucket results in HTTP 403: Forbidden error.

我已经使用 aws configure 配置了 AWS 凭证.

I have configured the AWS credentials using aws configure.

我可以使用 boto3 从私有存储桶下载文件，它使用 aws 凭据.看来我需要配置 pandas 才能使用 AWS 凭证，但不知道如何.

I can download a file from a private bucket using boto3, which uses aws credentials. It seems that I need to configure pandas to use AWS credentials, but don't know how.

推荐答案

Pandas 在 read_csv 中使用 boto(不是 boto3).您或许可以安装 boto 并使其正常工作.

Pandas uses boto (not boto3) inside read_csv. You might be able to install boto and have it work correctly.

boto 和 python 3.4.4/python3.5.1 存在一些问题.如果您在这些平台上，并且在修复这些平台之前，您可以将 boto 3 用作

There's some troubles with boto and python 3.4.4 / python3.5.1. If you're on those platforms, and until those are fixed, you can use boto 3 as

import boto3
import pandas as pd

s3 = boto3.client('s3')
obj = s3.get_object(Bucket='bucket', Key='key')
df = pd.read_csv(obj['Body'])

那个 obj 有一个 .read 方法(它返回一个字节流)，这对于 Pandas 来说已经足够了.

That obj had a .read method (which returns a stream of bytes), which is enough for pandas.

这篇关于从私有 S3 存储桶读取文件到 Pandas 数据帧的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！