我正在尝试建立一个Tf-Idf模型,该模型可以使用gensim对双字母组和单字组进行评分。为此,我构建了gensim词典,然后使用该词典创建用于构建模型的语料库的词袋表示。
构建字典的步骤如下所示:
dict = gensim.corpora.Dictionary(tokens)
其中
token
是像这样的字母组合和双字母组合的列表:[('restore',),
('diversification',),
('made',),
('transport',),
('The',),
('grass',),
('But',),
('distinguished', 'newspaper'),
('came', 'well'),
('produced',),
('car',),
('decided',),
('sudden', 'movement'),
('looking', 'glasses'),
('shapes', 'replaced'),
('beauties',),
('put',),
('college', 'days'),
('January',),
('sometimes', 'gives')]
但是,当我向
gensim.corpora.Dictionary()
提供诸如此类的列表时,该算法会将所有 token 减少为双字母组,例如:test = gensim.corpora.Dictionary([(('happy', 'dog'))])
[test[id] for id in test]
=> ['dog', 'happy']
有没有办法用gensim生成包含双字母组的字典?
最佳答案
from gensim.models import Phrases
from gensim.models.phrases import Phraser
from gensim import models
docs = ['new york is is united states', 'new york is most populated city in the world','i love to stay in new york']
token_ = [doc.split(" ") for doc in docs]
bigram = Phrases(token_, min_count=1, threshold=2,delimiter=b' ')
bigram_phraser = Phraser(bigram)
bigram_token = []
for sent in token_:
bigram_token.append(bigram_phraser[sent])
输出将是:
[['new york', 'is', 'is', 'united', 'states'],['new york', 'is', 'most', 'populated', 'city', 'in', 'the', 'world'],['i', 'love', 'to', 'stay', 'in', 'new york']]
#now you can make dictionary of bigram token
dict = gensim.corpora.Dictionary(bigram_token)
print(dict.token2id)
#Convert the word into vector, and now you can use tfidf model from gensim
corpus = [dict.doc2bow(text) for text in bigram_token]
tfidf_model = models.TfidfModel(corpus)
关于python - 如何建立一个包含双字母组的gensim词典?,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/51426107/