python - python中的spacy多线程训练

我试图找到一种在spacy上使用多线程来训练NER模型的方法。看起来在我的工作计算机(Ubuntu 16.04 Python3.5)上默认使用多线程，但在我的服务器上却没有。

知道为什么吗？

关于服务器上spaCy＆env的信息

Platform           Linux-3.14.32-xxxx-grs-ipv6-64-x86_64-with-Debian-8
Python version     3.4.2
Location           /home/nlp/.env/lib/python3.4/site-packages/spacy
Models             fr, fr_core_news_md
spaCy version      2.0.5

尝试过程:

安装

python3 -m venv .env
source .env/bin/activate
pip install -U spacy
pip3 install pip --upgrade
python -m spacy download fr
python -m spacy validate

脚本python3

import spacy
import random

ITERATION_NBR = 100
DROP_RATE = 0.5

TRAIN_DATA = [
    ('Who is Shaka Khan?', {
        'entities': [(7, 17, 'PERSON')]
    }),
    ('I like London and Berlin.', {
        'entities': [(7, 13, 'LOC'), (18, 24, 'LOC')]
    })
]

def main():
    try:
        nlp = spacy.load("fr")
    except:
        nlp = spacy.load("fr_core_news_sm")
    if 'ner' not in nlp.pipe_names:
        ner = nlp.create_pipe('ner')
        nlp.add_pipe(ner, last=True)
    else:
        ner = nlp.get_pipe('ner')
    for _, annotations in TRAIN_DATA:
        for ent in annotations.get('entities'):
            ner.add_label(ent[2])
    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner']
    with nlp.disable_pipes(*other_pipes):
        optimizer = nlp.begin_training()
        for itn in range(ITERATION_NBR):
            random.shuffle(TRAIN_DATA)
            losses = {}
            for text, annotations in TRAIN_DATA:
                nlp.update(
                    [text],
                    [annotations],
                    drop=DROP_RATE,
                    sgd=optimizer,
                    losses=losses)

执行

python3 <scriptName>.py

最佳答案

它必须是 python> = 3.5 ，才能在训练spacy时默认运行多线程

关于python - python中的spacy多线程训练，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/48517984/