ValueError:太多值无法解压缩Multiprocessing Pool

问题描述

我有以下工作人员"，最初返回了一个JSON对象，但我希望它返回多个JSON对象:

I have the following 'worker' that initially returned a single JSON object, but I would like it to return multiple JSON objects:

def data_worker(data):
    _cats, index, total = data
    _breeds = {}

    try:
        url = _channels['feedUrl']
        r = get(url, timeout=5)
        rss = etree.XML(r.content)

        tags = rss.xpath('//cats/item')
        _cats['breeds'] = {}
        for t in tags:
            _cats['breeds']["".join(t.xpath('breed/@url'))] = True
            _breeds['url'] = "".join(t.xpath('breed/@url'))

        return [_cats, _breeds]
    except:
        return [_cats, _breeds]

此工作程序是多处理池的参数:

This worker is a parameter for a multiprocessing pool:

cats, breeds = pool.map(data_worker, data, chunksize=1)

当我运行池和带有一个输出(即_cats)的工作程序时，它工作得很好，但是当我尝试返回多个JSON"schemas"时，出现错误:

When I run the pool and the worker with just one output (i.e. _cats), it works just fine, but when I try to return multiple JSON "schemas," I get the error:

  File "crawl.py", line 111, in addFeedData
    [cats, breeds] = pool.map(data_worker, data, chunksize=1)
ValueError: too many values to unpack

如何在data_worker中返回2个单独的JSON对象?我需要将它们作为单独的JSON对象.请注意，我已经尝试了以下方法，这些方法不起作用:

How can I return 2 separate JSON objects in data_worker? I need them to be separate JSON objects. Note, I have already tried the following, which did not work:

[cats, breeds] = pool.map(data_worker, data, chunksize=1)
(cats, breeds) = pool.map(data_worker, data, chunksize=1)
return (_cats, _breeds)

推荐答案

首先，我认为您打算写这篇文章:

First of all I think you meant to write this:

cats, breeds = pool.map(data_worker, data, chunksize=1)

但是无论如何这是行不通的，因为data_worker返回一对，但是map()返回一个工作者返回的列表.因此，您应该这样做:

But anyway this won't work, because data_worker returns a pair, but map() returns a list of whatever the worker returns. So you should do this:

cats = []
breeds = []
for cat, breed in pool.map(data_worker, data, chunksize=1):
    cats.append(cat)
    breeds.append(breed)

这将为您提供您要查找的两个列表.

This will give you the two lists you seek.

换句话说，您希望有一对列表，但是却得到了一对列表.

In other words, you expected a pair of lists, but you got a list of pairs.

这篇关于ValueError:太多值无法解压缩Multiprocessing Pool的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！