您好,我有一个关于池分配器的问题。当我开始我的训练工作时,尝试执行“PoolAllocator”花了几个小时。一些日志如下所示。有没有办法调试/分析原因?我该如何改进?
谢谢!
tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 9639 get requests, put_count=4341 evicted_count=1000 eviction_rate=0.230362 and unsatisfied allocation rate=0.663762
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 100 to 110
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2013 evicted_count=2000 eviction_rate=0.993542 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 7080 get requests, put_count=6922 evicted_count=5000 eviction_rate=0.722335 and unsatisfied allocation rate=0.730791
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 176 to 193
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2025 evicted_count=2000 eviction_rate=0.987654 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=5030 evicted_count=5000 eviction_rate=0.994036 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2044 evicted_count=2000 eviction_rate=0.978474 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 9617 get requests, put_count=8892 evicted_count=5000 eviction_rate=0.562303 and unsatisfied allocation rate=0.600915
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 596 to 655
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2087 evicted_count=2000 eviction_rate=0.958313 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=3095 evicted_count=3000 eviction_rate=0.969305 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=1115 evicted_count=1000 eviction_rate=0.896861 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=1140 evicted_count=1000 eviction_rate=0.877193 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=1169 evicted_count=1000 eviction_rate=0.855432 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=1204 evicted_count=1000 eviction_rate=0.830565 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2247 evicted_count=2000 eviction_rate=0.890076 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=8272 evicted_count=8000 eviction_rate=0.967118 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2362 evicted_count=2000 eviction_rate=0.84674 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 10614 get requests, put_count=10944 evicted_count=2000 eviction_rate=0.182749 and unsatisfied allocation rate=0.198606
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 4823 to 5305
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=3705 evicted_count=3000 eviction_rate=0.809717 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 4204990 get requests, put_count=4204742 evicted_count=3000 eviction_rate=0.00071348 and unsatisfied allocation rate=0.00104257
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 16377314 get requests, put_count=16374197 evicted_count=13000 eviction_rate=0.000793932 and unsatisfied allocation rate=0.00105347
最佳答案
实际上,这项工作运行良好。问题是python的输出缓冲。在作业完成之前,它不会显示任何训练结果。您可以通过以下方式禁用缓冲:
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
或者您可以尝试以下其他方法:
Disable output buffering
关于tensorflow - 长时间在 "PoolAllocator",我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/36928391/