s4cmd的问题

问题描述

我有大约50GB的数据要上传到S3存储桶，但是s3cmd并不可靠且非常慢.由于超时错误，同步似乎无法正常工作.

I have about 50GB data to upload to S3 bucket but s3cmd is unreliable and very slow. the sync doesn't seem to work because of the timeout error.

我切换到了 s4cmd ，它运行良好，多线程且运行速度很快.

I switched to s4cmd it works great, multi threaded and fast.

     s4cmd dsync -r -t 1000 --ignore-empty-source forms/ s3://bucket/J/M/

上面的代码上传了一组文件，然后抛出错误- [线程失败]无法从以下源读取数据:/home/ubuntu/文件路径源文件包含一个图像文件，因此没有任何错误.

The above uploads a set of files and then throws error -[Thread Failure] Unable to read data from source: /home/ubuntu/path to fileThe source file contains an image file so there is nothing wrong there.

s4cmd具有--retry之类的选项，可以在命令失败时重新启动该命令，但这似乎也不起作用. 如果您遇到了防止此错误的解决方案，请分享.

s4cmd has options like --retry for the command to restart if it fails but this also doesn't seem to work. If you have come across a solution to prevent this error, Please share.

推荐答案

我知道它工作正常.我很高兴我的文件上传速度超快.如果您仍在使用s3cmd，我强烈建议您切换到s4cmd！

I got it working fine. I'm glad my file uploads are super fast. If you are still using s3cmd I highly recommend you switch to s4cmd!

下载并安装s4cmd.找到s4cmd.py并替换为以下内容-

Download and install s4cmd. Find s4cmd.py and replace with the following -

    @log_calls
  def read_file_chunk(self, source, pos, chunk):
    '''Read local file cunks'''
    data = None
    with open(source, 'rb') as f:
      f.seek(pos)
      data = f.read(chunk)
    if not f:
      raise Failure('Unable to read data from source: %s' % source)
    return StringIO(data)

然后将s4cmd.py调用为上载命令，例如

then call s4cmd.py into the upload command like

/pathtodir/s4cmd.py dsync -r forms/ s3://bucket/J/M/

这篇关于s4cmd的问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！