来自Iterator的Java ExecutorService读取任务

问题描述

全部

我正在使用Java ExecutorService并行执行任务.不幸的是，任务列表现在达到了数千万.这意味着由于内存限制，无法提前将任务提交给执行者服务.

I'm using a Java ExecutorService to perform tasks in parallel. Unfortunately, the list of tasks is now reaching the tens of millions. This means that submitting the tasks to the executor service ahead of time is infeasible due to memory constraints.

我能够生成一个迭代器，该迭代器可以根据需要动态创建任务，但是我不确定如何最好地将其应用于ExecutorService.

I am able to generate an iterator which dynamically creates the tasks as they are needed, but I'm not sure how best to apply this to the ExecutorService.

我应该创建一个从迭代器中提取下一个任务的任务，还是有更好的方法呢?

Should I create a task which pulls the next task from the iterator or is there some better way to do this?

推荐答案

一个快速的实验就完成了这种工作.当然，它应该演示一种实现方法.

A quick experiment produced this that kind of works. It certainly should demonstrate one way of doing it.

我创建并运行一个 ServiceFeeder ，该服务通过 execute 方法将 Runnables 传递给服务.

I create and run a ServiceFeeder which delivers Runnables to the service via the execute method.

ExecutorService service = Executors.newFixedThreadPool(10);

class ServiceFeeder implements Runnable {
    final Iterator<Runnable> i;

    public ServiceFeeder(Iterator<Runnable> i) {
        this.i = i;
    }

    @Override
    public void run() {
        while (i.hasNext()) {
            service.execute(i.next());
        }
    }
}

public void test() throws Exception {
    System.out.println("Hello world!");
    // Demo Iterator<Runnable> - use yours.
    Iterator<Runnable> i = new Iterator<Runnable>() {
        volatile int n = 0;

        @Override
        public boolean hasNext() {
            return n < 100;
        }

        @Override
        public Runnable next() {
            return () -> System.out.println(n++);
        }
    };

    ServiceFeeder feeder = new ServiceFeeder(i);
    Thread feederThread = new Thread(feeder);
    feederThread.start();
    // Wait for the feeder to stop.
    feederThread.join();
    // Wait for the service to stop.
    service.shutdown();
}

这种种作品，因为它的打印量远远超出了我的预期，但是作为演示恕我直言，这不是问题.

This kind of works because it prints far more than I expected but that's not a problem as a demo IMHO.

这篇关于来自Iterator的Java ExecutorService读取任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！