Boto3:需要Future或协程

问题描述

我正在尝试使用boto3从S3下载2个文件，等到下载2个文件后，将继续处理2个文件

I am trying to download 2 files from S3, using boto3, and wait until 2 files are downloaded, will continue to process 2 files

我做了什么

async def download_files():
    await client.download_file(const.bucket_name, 'file/name.txt', '/tmp/name.txt')
    print('file 1 downloaded')
    await client.download_file(const.bucket_name, 'file/class.txt', '/tmp/class.txt')
    print('file 2 downloaded')
    return True

def main():
    ...
    loop = asyncio.get_event_loop()
    loop.run_until_complete(download_files())
    loop.close()
    ...
main()

我收到一个错误

A Future or coroutine is required

这是我第一次使用 asyncio ，请告诉我.

It's first time that I am using asyncio, please advise me.

推荐答案

boto3不支持异步...它的功能正在阻塞，而不是可以等待.因此，这里要做的绝对最低限度是从对 download_file 的调用中删除 await ，它应该可以正常工作.

boto3 isn't asyncio-aware... its functions are blocking, rather than being awaitable. So, the absolute minimum to do here would be to just remove await from the calls to download_file, and it should work.

client.download_file(const.bucket_name, 'file/name.txt', '/tmp/name.txt')
print('file 1 downloaded')
client.download_file(const.bucket_name, 'file/class.txt', '/tmp/class.txt')
print('file 2 downloaded')
return True

但是，如果线程的事件循环中还有任何其他并发任务，则这将具有较差的并发属性:它们将被阻塞并且在下载期间不会继续进行[这将使asyncio的使用变得不必要...并发任务的进行有点不同步...]

However, this would have poor concurrency properties if there were to be any other concurrent tasks in the thread's event loop: they would be blocked and wouldn't proceed during the downloads [this would make the usage of asyncio a bit unnecessary... concurrent tasks proceeding is sort of the point of asyncio...]

要获得更好的并发属性，您应该能够通过 run_in_executor ，默认情况下，它将在另一个线程中运行传递的函数.

To have better concurrency properties, you should be able to call the functions via run_in_executor, which by default will run the passed function in another thread.

async def download_files(loop):
  await loop.run_in_executor(None, client.download_file, const.bucket_name, 'file/name.txt', '/tmp/name.txt')
  print('file 1 downloaded')
  await loop.run_in_executor(None, client.download_file, const.bucket_name, 'file/class.txt', '/tmp/class.txt')
  print('file 2 downloaded')
  return True

或者，您可以使用 aiohttp 和滚动拥有的AWS认证(完整披露:我自己撰写的有关滚动式AWS身份验证的帖子)

Alternatively, rather than using boto3 and threads, you could use aiohttp, and roll-you-own AWS authentication (full disclosure: the post on rolling-your-own AWS authentication was written by me)

这篇关于Boto3:需要Future或协程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！