本文介绍了使用Rx和SelectMany限制并发请求的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我有一个要使用 HttpClient 同时下载的页面的URL列表。 URL列表可能很大(100个或更多!)

I have a list of URLs of pages I want to download concurrently using HttpClient. The list of URLs can be large (100 or more!)

我目前有以下代码:

var urls = new List<string>
            {
                @"http:\\www.amazon.com",
                @"http:\\www.bing.com",
                @"http:\\www.facebook.com",
                @"http:\\www.twitter.com",
                @"http:\\www.google.com"
            };

var client = new HttpClient();

var contents = urls
    .ToObservable()
    .SelectMany(uri => client.GetStringAsync(new Uri(uri, UriKind.Absolute)));

contents.Subscribe(Console.WriteLine);

问题:由于使用 SelectMany ,几乎同时创建了大量任务。似乎,如果URL列表足够大,则很多Task都会超时(我收到一个任务已取消 例外)。

The problem: due to the usage of SelectMany, a big bunch of Tasks are created almost at the same time. It seems that if the list of URLs is big enough, a lot Tasks give timeouts (I'm getting "A Task was cancelled" exceptions).

所以,我认为应该有一种方法,也许使用某种调度程序,来限制并发任务的数量,在给定时间不允许超过5个或6个。

So, I thought there should be a way, maybe using some kind of Scheduler, to limit the number of concurrent Tasks, not allowing more than 5 or 6 at a given time.

通过这种方式,我可以像不立即执行任务那样进行并发下载,而不必启动太多可能导致停顿的任务。

This way I could get concurrent downloads without launching too many tasks that may get stall, like they do right now.

该怎么做,这样我才不会被很多超时任务所饱和?

How to do that so I don't saturate with lots of timed-out Tasks?

推荐答案

记住 SelectMany()实际上是 Select()。Merge()。尽管 SelectMany 没有 maxConcurrent 参数,但可以。因此,您可以使用它。

Remember SelectMany() is actually Select().Merge(). While SelectMany does not have a maxConcurrent paramter, Merge() does. So you can use that.

在您的示例中,您可以这样做:

From your example, you can do this:

var urls = new List<string>
    {
        @"http:\\www.amazon.com",
        @"http:\\www.bing.com",
        @"http:\\www.facebook.com",
        @"http:\\www.twitter.com",
        @"http:\\www.google.com"
    };

var client = new HttpClient();

var contents = urls
    .ToObservable()
    .Select(uri => Observable.FromAsync(() => client.GetStringAsync(uri)))
    .Merge(2); // 2 maximum concurrent requests!

contents.Subscribe(Console.WriteLine);

这篇关于使用Rx和SelectMany限制并发请求的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

09-21 02:31