如何限制并发异步I / O操作的数量？

问题描述

// let's say there is a list of 1000+ URLs
string[] urls = { "http://google.com", "http://yahoo.com", ... };

// now let's send HTTP requests to each of these URLs in parallel
urls.AsParallel().ForAll(async (url) => {
    var client = new HttpClient();
    var html = await client.GetStringAsync(url);
});

这是问题所在，它开始1000+并发Web请求。有没有一种简单的方法来限制​​这些异步HTTP请求的并发量？以便不超过20的网页是在任何给定时间下载。如何做到这一点的最有效的方式？

Here is the problem, it starts 1000+ simultaneous web requests. Is there an easy way to limit the concurrent amount of these async http requests? So that no more than 20 web pages are downloaded at any given time. How to do it in the most efficient manner?

推荐答案

您绝对可以做到这一点的异步for .NET的最新版本，使用.NET 4.5 Beta版。从USR点previous岗位的好文章由Stephen Toub写的，但少宣布的消息是异步信号实际上是由它变成的.NET 4.5 Beta版

You can definitely do this in the latest versions of async for .NET, using .NET 4.5 Beta. The previous post from 'usr' points to a good article written by Stephen Toub, but the less announced news is that the async semaphore actually made it into the Beta release of .NET 4.5

如果你看看我们敬爱的<$c$c>SemaphoreSlim类（你应该使用，因为它是比原来的<$c$c>Semaphore),它现在拥有的<$c$c>WaitAsync(...)系列重载，所有的预期参数 - 超时时间间隔，取消标记，所有你平时的日程安排的朋友：）

If you look at our beloved SemaphoreSlim class (which you should be using since it's more performant than the original Semaphore), it now boasts the WaitAsync(...) series of overloads, with all of the expected arguments - timeout intervals, cancellation tokens, all of your usual scheduling friends :)

斯蒂芬也写了最近的博客文章，与测试版出来了新的.NET 4.5超值服务：http://blogs.msdn.com/b/pfxteam/archive/2012/02/29/10274035.aspx

Stephen's also written a more recent blog post about the new .NET 4.5 goodies that came out with Beta: http://blogs.msdn.com/b/pfxteam/archive/2012/02/29/10274035.aspx

最后，这里是关于如何使用SemaphoreSlim为异步方法节流一些样本code：

Last, here's some sample code about how to use SemaphoreSlim for async method throttling:

async Task MyOuterMethod() {

    // let's say there is a list of 1000+ URLs
    string[] urls = { "http://google.com", "http://yahoo.com", ... };

    // now let's send HTTP requests to each of these URLs in parallel
    List<Task> allTasks = new List<Task>();
    SemaphoreSlim throttler = new SemaphoreSlim(initialCount: 20);
    foreach (var url in urls) {

        // do an async wait until we can schedule again
        await throttler.WaitAsync();

        // using Task.Run(...) to run the lambda in its own parallel
        // flow on the threadpool
        allTasks.Add(Task.Run(async () => {
            try {
                var client = new HttpClient();
                var html = await client.GetStringAsync(url);
            } finally {
                throttler.Release();
            }
        }));
    }

    // won't get here until all urls have been put into tasks
    await Task.WhenAll(allTasks);

    // won't get here until all tasks have completed in some way
    // (either success or exception)
}

最后，但可能是一个值得一提的是，使用基于TPL-调度的解决方案。您可以在那些尚未启动的TPL创建委托密集型任务，并允许自定义任务调度程序来限制并发性。事实上，这里是它的MSDN示例：

Last, but probably a worthy mention is a solution that uses TPL-based scheduling. You can create delegate-bound tasks on the TPL that have not yet been started, and allow for a custom task scheduler to limit the concurrency. In fact, there's an MSDN sample for it here:

http://msdn.microsoft.com/en-us/library/ee789351

这篇关于如何限制并发异步I / O操作的数量？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！