值未使用python2.7中的pool.apply_async插入MySQL表中

问题描述

我正在尝试运行以下代码来并行填充某个应用程序的表.首先，定义以下函数，该函数应该连接到我的数据库，并使用给定的值执行sql命令(以插入表中).

I am trying to run the following code to populate a table in parallel for a certain application. First the following function is defined which is supposed to connect to my db and execute the sql command with the values given (to insert into table).

def dbWriter(sql, rows) :
   # load cnf file
    MYSQL_CNF = os.path.abspath('.') + '/mysql.cnf'
    conn = MySQLdb.connect(db='dedupe',
                       charset='utf8',
                       read_default_file = MYSQL_CNF)

    cursor = conn.cursor()
    cursor.executemany(sql, rows)
    conn.commit()
    cursor.close()

    conn.close()

然后有一块:

pool = dedupe.backport.Pool(processes=2)

done = False

while not done :
    chunks = (list(itertools.islice(b_data, step)) for step in
      [step_size]*100)


    results = []

    for chunk in chunks :
        print len(chunk)
        results.append(pool.apply_async(dbWriter,
                                    ("INSERT INTO blocking_map VALUES (%s, %s)",
                                     chunk)))

    for r in results :

        r.wait()

    if len(chunk) < step_size :
        done = True


pool.close()

一切正常，没有错误.但是最后，我的表是空的，这意味着插入不成功.在许多Google搜索之后，我尝试了很多方法来解决此问题(包括添加要插入的列名)，但未成功.任何建议，将不胜感激. (在python2.7，gcloud(ubuntu)中运行代码.请注意，在将此处粘贴后，缩进可能会有些混乱)

Everything works and there are no errors. But at the end, my table is empty, meaning somehow the insertions were not successful. I have tried so many things to fix this (including adding column names for insertion) after many google searches and have not been successful. Any suggestions would be appreciated. (running code in python2.7, gcloud (ubuntu). note that indents may be a bit messed up after pasting here)

还请注意，块"完全遵循所需的数据格式.

Please also note that "chunk" follows exactly the required data format.

注意.这是此示例的一部分请注意，在上面的示例(链接)中，我唯一要更改的是，由于我在gcloud平台上运行我的代码并强制执行GTID标准，因此我将创建表和插入表的步骤分开.

Note. This is part of this examplePlease note that the only thing I am changing in the above example (linked) is that I am separating the steps for creation of and inserting into the tables since I am running my code on gcloud platform and it enforces GTID standards.

推荐答案

解决方案将dbwriter功能更改为:

Solution was changing dbwriter function to:

conn = MySQLdb.connect(host = # host ip,
                 user = # username,
                 passwd = # password,
                 db = 'dedupe')
cursor = conn.cursor()
cursor.executemany(sql, rows)
cursor.close()
conn.commit()
conn.close()

这篇关于值未使用python2.7中的pool.apply_async插入MySQL表中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！