如何使用包 AWSS3.jl 连接到在 EC2 实例上运行的 S3 存储桶?

问题描述

我正在尝试在 AWS 计算集群上使用 Julia，处理来自 Amazon S3 存储桶的数据.

I am trying to use Julia on an AWS computing cluster, processing data from Amazon S3 buckets.

如何在 Julia 中使用 AWSS3.jl 打开与 S3 存储桶的连接?

How do I use in Julia AWSS3.jl to open a connection to an S3 bucket?

我特别想知道配置连接并随后使用 Julia 的配置的最佳方法是什么.

In particular I would like to know what is the best approach to configure the connectivity and subsequently use that configuration from Julia.

推荐答案

1. 创建具有 S3 访问权限的 AWS 策略(特别是 PutObject、GetObject 等)

创建一个面向 EC2 的 IAM 角色并将第一步中的策略添加到该角色

Create an EC2 oriented IAM role and add the policy from the first step to that role

将 IAM 角色分配给 EC2 实例(或者在为集群创建 EC2 实例时将其配置为在实例配置文件中使用)

Assign the IAM role to the EC2 instance (or perhaps configure it to use in an instance profile when creating EC2 instances for your cluster)

现在你已经准备好在 Julia 中工作了.下面是一个将任何 Julia 对象序列化和反序列化到 S3 存储桶的简单示例.

Now you are ready to do your work in Julia. Below a simple example that serializes and deserializes any Julia object to an S3 bucket.

using AWS, AWSS3, Serialization
struct SampleData
  a::Int
  b::String
end

d=SampleData(1,"sss")
aws = global_aws_config(; region="us-east-1")
b = IOBuffer()
serialize(b, d)

s3_put(aws, "your-s3-bucket-name","myfile.bin", b.data)

ddat = s3_get(aws, "your-s3-bucket-name","myfile.bin")
d2 = deserialize(IOBuffer(ddat))

@assert d == d2

如果您对第 1-3 点有任何问题，这里是教程:https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-access-s3-bucket/

Should you have any problems with points 1-3, here is the tutorial: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-access-s3-bucket/

这篇关于如何使用包 AWSS3.jl 连接到在 EC2 实例上运行的 S3 存储桶?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！