猫鼬光标:http集合中的批量请求

本文介绍了猫鼬光标:http集合中的批量请求的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我遇到一个与rxJS和大量集合中的大量HTTP请求(1M +文档)相关的问题

I have a problem which is relevant to rxJS and bulk HTTP requests from the huge collection (1M+ docs)

我有以下代码，逻辑很简单.我将集合中的所有文档推送到allplayers数组，并一次向API发出20个批量HTTP请求(猜测您了解其局限性)，因此，代码可以正常工作，但我想是时候从此重构它了:

I have the following code, with quite simple logic. I push all the docs from the collection to allplayers array and making bulk 20 HTTP requests to API at once (guess you understand why it's limited) So, the code works fine, but I guess it's time to refactor it from this:

        const cursor = players_db.find(query).lean().cursor();
        cursor.on('data', function(player) { allPlayers.push(player); });
        cursor.on('end', function() {
            logger.log('warng',`S,${allPlayers.length}`);
            from(allPlayers).pipe(
                mergeMap(player => getPlayer(player.name, player.realm),20),
            ).subscribe({
                next: player => console.log(`${player.name}@${player.realm}`),
                error: error => console.error(error),
                complete: () => console.timeEnd(`${updatePlayer.name}`),
            });
        });

就目前而言，我正在将find与cursor与(batchSize)配合使用，但是如果我理解此权限(通过.length)，并根据以下问题进行了解答:{} batchSize只是一种优化方式，它并没有返回我的X文档数组.

As for now, I'm using find with cursor with (batchSize), but if I understood this right (via .length), and according to this question: {mongoose cursor batchSize} batchSize is just a way of optimization and it's not return me array of X docs.

例如，我可以形成具有必要长度(如20)的数组，并将其传输到rxJS，就像我以前使用的那样.但我想应该有另一种方法，可以在for promise loop

For example I could form arrays with necessary length (like 20) and transfer it to rxJS as I use it before. But I guess there should be another way, where I could use rxJS inside this for promise loop

    const players = await players_db.find(query).lean().cursor({batchSize: 10});
    for (let player = await players.next(); player != null; player = await players.next()) {
        //do something via RxJS inside for loop
    }

我也发现了这个问题{}，这也与我的问题有关，但我理解逻辑，但不了解它的语法.我也知道cursor变量不是文档，我不能对它做任何有用的的操作.或者实际上我可以?

Also I found this question {Best way to query all documents from a mongodb collection in a reactive way w/out flooding RAM} which also relevant to my problem and I understand the logic, but don't the syntax of it. I also know that cursor variable isn't a doc I cann't do anything useful with it. Or actually I could?

rxJS的bufferCount是一个非常有趣的运算符
https://gist.github.com/wellcaffeinated/f908094998edf54dc5840c8c3ad734d3 可能的解决方案?

rxJS's bufferCount is a quite interesting operator
https://gist.github.com/wellcaffeinated/f908094998edf54dc5840c8c3ad734d3 probable solution?

推荐答案

因此，最终我发现在这种情况下不需要rxJS.

So, in the end I found that rxJS isn't needed (but can be used) for this case.

解决方案非常简单，只需使用MongoCursor:

The solution was quite simple and using just MongoCursor:

async function BulkRequest (bulkSize = 10) {
    try {
       let BulkRequest_Array = [];
       const cursor = collection_db.find({}).lean().cursor({batchSize: bulkSize});
       cursor.on('data', async (doc) => {
           BulkRequest_Array.push(/*any function or axios instance*/)
            if (BulkRequest_Array.length >= bulkSize) {
                cursor.pause();
                console.time(`========================`);;
                await Promise.all(BulkRequest_Array);
                BulkRequest_Array.length = 0;
                cursor.resume();
                console.timeEnd(`========================`);
            }
       }
    } catch (e) {
       console.error(e)
    }
}

BulkRequest();

这篇关于猫鼬光标:http集合中的批量请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！

rxjs