Pytorch 问题:当 num_workers > 时，我的 jupyter 卡住了0

问题描述

这是我在 PyTorch 中的代码片段，当我使用 num_workers 时，我的 jupiter notebook 卡住了 >0，我在这个问题上花了很多时间没有任何答案.我没有 GPU，只能使用 CPU.

This is a snippet of my code in PyTorch, my jupiter notebook stuck when I used num_workers > 0, I spent a lot on this problem without any answer. I do not have a GPU and I work only with a CPU.

class IndexedDataset(Dataset):

def __init__(self,data,targets, test=False):
    self.dataset = data
    if not test:
        self.labels = targets.numpy()
        self.mask =  np.concatenate((np.zeros(NUM_LABELED), np.ones(NUM_UNLABELED)))


    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, idx):
        image = self.dataset[idx]
        return image, self.labels[idx]

    def display(self, idx):
        plt.imshow(self.dataset[idx], cmap='gray')
        plt.show()

train_set = IndexedDataset(train_data, train_target, test = False)

test_set = IndexedDataset(test_data, test_target, test = True)

train_loader = DataLoader(train_set, batch_size=BATCH_SIZE, num_workers=2)

test_loader = DataLoader(test_set, batch_size=BATCH_SIZE, num_workers=2)

任何帮助，不胜感激.

推荐答案

当num_workers大于0时，PyTorch使用多个进程进行数据加载.

When num_workers is greater than 0, PyTorch uses multiple processes for data loading.

Jupyter 笔记本在多处理方面存在已知问题.

Jupyter notebooks have known issues with multiprocessing.

解决此问题的一种方法是不使用 Jupyter notebook - 只需编写一个普通的 .py 文件并通过命令行运行它.

One way to resolve this is not to use Jupyter notebooks - just write a normal .py file and run it via command-line.

或者尝试使用这里的建议:Jupyter notebook 永远不会完成使用多处理 (Python 3) 进行处理.

Or try use what's suggested here: Jupyter notebook never finishes processing using multiprocessing (Python 3).

这篇关于Pytorch 问题:当 num_workers > 时，我的 jupyter 卡住了0的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！