问题描述
我对Python还是很陌生,我正在尝试编写一些巨大的列表(内部带有随机字母).实际上,在我的机器上要花费2 50,000条线大约需要75-80秒.
I'm fairly new to Python, and I'm trying to write some huge lists (with random letters inside). Actually it takes me around 75 - 80 seconds on my machine for 2,000,000 lines.
import timeit
import random, string
global_tab = []
global_nb_loop = 2000000
print("Generate %d lines" % global_nb_loop)
global_tab = []
for x in range(global_nb_loop):
global_tab.append(("".join( [random.choice(string.ascii_letters) for i in range(15)] ), "".join( [random.choice(string.digits) for i in range(2)])))
print("%d lines generated" % len(global_tab))
以及linux time
命令的结果:
And the result with linux time
command:
$ time python3 DEV/PyETL/generateList.py
Generate 2000000 lines
2000000 lines generated
real 1m16.844s
user 1m16.609s
sys 0m0.203s
当监视系统资源时,只有1个内核处于100%的状态,而不是像我在其上测试过的Windows机器上的4个那样,使我感到惊讶.
I was surprised when monitoring system resources that only 1 core was at 100%, instead of 4 like on a Windows machine on which I've tested this too.
我当然已经尝试应用一些线程,但是我面临一个问题:它比在单个内核上运行需要更多的时间.也许线程不是解决方案,或者我可能错误地使用了它们.
Of course I've tried to apply some threads, but I'm facing a problem: it takes more time than running on a single core. Maybe threads are not the solution or I'm probably using them wrong.
这是新代码:
import random, string
import threading
global_tab = []
global_nb_threads = 4
global_nb_loop = 2000000
threadLock = threading.Lock()
class generateList(threading.Thread):
def __init__(self, name):
threading.Thread.__init__(self)
self.name = name
def run(self):
global global_tab
self.tab = []
print("[%s] Generate %d lines" % (self.name, int(global_nb_loop/global_nb_threads)))
# divide desirated lines with number of threads
for x in range(int(global_nb_loop/global_nb_threads)):
self.tab.append(("".join( [random.choice(string.ascii_letters) for i in range(15)] ), "".join( [random.choice(string.digits) for i in range(2)])))
threadLock.acquire()
global_tab += self.tab
threadLock.release()
del self.tab
print("[%s] %d lines in list" % (self.name, len(global_tab)))
for i in range(global_nb_threads):
# Create threads
t = generateList("Thread-" + str(i))
# Start
t.start()
for i in range(global_nb_threads):
# Wait for threads end
t.join()
执行:
$ time python3 DEV/PyETL/generateListThreads.py
[Thread-0] Generate 500000 lines
[Thread-1] Generate 500000 lines
[Thread-2] Generate 500000 lines
[Thread-3] Generate 500000 lines
[Thread-3] 500000 lines in list
[Thread-0] 1000000 lines in list
[Thread-2] 1500000 lines in list
[Thread-1] 2000000 lines in list
real 1m40.858s
user 1m41.208s
sys 0m0.916s
32秒比1个内核多100%,但是监视显示这8个内核同时具有20%至40%的负载.
32 seconds more than 1 core with 100%, but monitoring shows that the 8 cores were with 20 - 40% load at the same time.
由于所有线程都在同时工作,因此生成的行数更少并且仅用于更新全局变量而进行同步,因此执行时间不应该少于单个内核吗?
Since all threads are working at the same time, generating fewer rows and synchronizing only for updating a global variable, shouldn't the execution time be lower than a single core?
推荐答案
我很确定您的锁不是必需的,并且会降低您的速度. (实际上,我只是注意到锁是在 之后使用的,所以大部分工作都没有完成,所以并不是很重要.)
I am pretty sure your lock is not necessary and is slowing you down. (edit: actually, I just noticed the lock is used after the majority of the work is done, so isn't really relevant.)
global_tab += self.tab
(我认为)是通过Python GIL构成的原子. (实际上,此只能声明list.extend()
,请改用它.这是另一个参考:列表线程安全吗?
global_tab += self.tab
is (I think) atomic through the Python GIL. (Actually, this only claims list.extend()
, so use that instead. Here's another reference: Are lists thread safe?
或者,我会尝试使用大块大小的multiprocessing.imap_unordered
.不利之处在于结果是按流发送的,但是您的随机字符串处理可能会使结果黯然失色.
Alternatively, I would try multiprocessing.imap_unordered
with a large chunksize. The downside is the results are sent over by stream, but your random string processing might overshadow that.
import multiprocessing
import random
import string
def randomword(x):
return ''.join(random.choice(string.ascii_letters) for i in range(15))
pool = multiprocessing.Pool(8)
results = pool.imap_unordered(randomword, range(100))
print([r for r in results])
对于200万个字符串(我将其更改为打印长度):
For 2 million strings (I changed it to print the length):
$ time python r.py
2000000
real 0m38.305s
user 1m31.717s
sys 0m25.853s
我也尝试清理一下您的版本并得到:
I also tried cleaning up your version a bit and got:
$ time python rr.py
[Thread-0] Generate 250000 lines
[Thread-1] Generate 250000 lines
[Thread-2] Generate 250000 lines
[Thread-3] Generate 250000 lines
[Thread-4] Generate 250000 lines
[Thread-5] Generate 250000 lines
[Thread-6] Generate 250000 lines
[Thread-7] Generate 250000 lines
[Thread-4] 250000 lines in list
[Thread-1] 500000 lines in list
[Thread-7] 750000 lines in list
[Thread-0] 1000000 lines in list
[Thread-6] 1250000 lines in list
[Thread-2] 1500000 lines in list
[Thread-3] 1750000 lines in list
[Thread-5] 2000000 lines in list
real 0m22.113s
user 0m24.969s
sys 0m5.537s
几个重要的变化:
- 在大范围内使用
xrange()
(啊,python3已经做到了.) - 删除线程锁
- 在全局上使用
extend()
.
- use
xrange()
on the large ranges (ah, python3 already does this.) - remove the threadlock
- use
extend()
on the global.
(当我将结果附加到global_tab
,顺便说一句,并且不包括临时列表时,我的结果大致相同.)
(my results were about the same when just appending to the global_tab
, btw, and leaving out the temporary list.)
import random, string
import threading
global_tab = []
global_nb_threads = 8
global_nb_loop = 2000000
class generateList(threading.Thread):
def __init__(self, name):
threading.Thread.__init__(self)
self.name = name
def run(self):
global global_tab
self.tab = []
print("[%s] Generate %d lines" % (self.name, int(global_nb_loop/global_nb_threads)))
for x in range(int(global_nb_loop/global_nb_threads)):
self.tab.append(("".join( [random.choice(string.ascii_letters) for i in range(15)] ), "".join( [random.choice(string.digits) for i in range(2)])))
global_tab.extend(self.tab)
print("[%s] %d lines in list" % (self.name, len(global_tab)))
for i in range(global_nb_threads):
t = generateList("Thread-" + str(i))
t.start()
for i in range(global_nb_threads):
t.join()
...但是,单线程仍在16秒时稍微快一点.
...but, single threaded is still slightly faster at 16 seconds.
如果我调multiprocessing
,我可以将其降低到6秒:
If I tune multiprocessing
, I can get it down to 6 seconds:
size = 2000000
processes = 8
pool = multiprocessing.Pool(processes)
results = [r for r in pool.imap_unordered(randomword, range(size), chunksize=int(size/processes))]
print(len(results))
输出:
$ time python r.py
2000000
real 0m5.713s
user 0m35.594s
sys 0m0.546s
...所以我认为这是我的最终答案:使用multiprocessing
.
...so I think that's my final answer: Use multiprocessing
.
这篇关于减少生成大量列表时的执行时间的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!