问题描述
我正在尝试这段代码
from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np
train_data = ["football is the sport","gravity is the movie", "education is imporatant"]
vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5,
stop_words='english')
print "Applying first train data"
X_train = vectorizer.fit_transform(train_data)
print vectorizer.get_feature_names()
print "\n\nApplying second train data"
train_data = ["cricket", "Transformers is a film","AIMS is a college"]
X_train = vectorizer.transform(train_data)
print vectorizer.get_feature_names()
print "\n\nApplying fit transform onto second train data"
X_train = vectorizer.fit_transform(train_data)
print vectorizer.get_feature_names()
这一个的输出是
Applying first train data
[u'education', u'football', u'gravity', u'imporatant', u'movie', u'sport']
Applying second train data
[u'education', u'football', u'gravity', u'imporatant', u'movie', u'sport']
Applying fit transform onto second train data
[u'aims', u'college', u'cricket', u'film', u'transformers']
我使用了fit_transform给矢量化器提供了第一组数据,因此它给了我类似[u'education', u'football', u'gravity', u'imporatant', u'movie', u'sport']
的特征名称,之后我将另一个训练集应用于了相同的矢量化器,但是它给了我与不使用fit或fit_transform相同的特征名称.但是我想知道如何在不覆盖以前的oncs的情况下更新矢量化器的功能.如果我再次使用fit_transform,以前的功能将被覆盖.因此,我想更新矢量化器的功能列表.所以我想要类似[u'education', u'football', u'gravity', u'imporatant', u'movie', u'sport',u'aims', u'college', u'cricket', u'film', u'transformers']
的方法.
I gave the first set of data using fit_transform to vectorizer so it gave me feature names like [u'education', u'football', u'gravity', u'imporatant', u'movie', u'sport']
after that i applied another train set to the same vectorizer but it gave me the same feature names as I didnt use fit or fit_transform. But I want to know how to update the features of a vectorizer without overwriting the previous oncs. If I use fit_transform again the previous features will get overwritten. So I want to update the feature list of the vectorizer. So i want something like [u'education', u'football', u'gravity', u'imporatant', u'movie', u'sport',u'aims', u'college', u'cricket', u'film', u'transformers']
How can I get that.
推荐答案
在sklearn术语中,这称为部分拟合,而您不能使用TfidfVectorizer
来做到这一点.有两种解决方法:
In sklearn terminology, this is called a partial fit and you can't do it with a TfidfVectorizer
. There are two ways around this:
- 连接两个训练集并重新矢量化
- 使用
HashingVectorizer
,它支持部分拟合.但是,由于具有哈希功能,因此该方法没有get_feature_names
方法,因此不会保留原始格式.另一个优点是,这样可以大大提高内存效率.
- Concatenate the two training sets and re-vectorize
- use a
HashingVectorizer
, which support partial fitting. However, that does not have aget_feature_names
method due to the fact that is hashes features, so the original isn't kept. Another advantage is that this is much more memory efficient.
第一种方法的示例:
from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np
train_data1 = ["football is the sport", "gravity is the movie", "education is important"]
vectorizer = TfidfVectorizer(stop_words='english')
print("Applying first train data")
X_train = vectorizer.fit_transform(train_data1)
print(vectorizer.get_feature_names())
print("\n\nApplying second train data")
train_data2 = ["cricket", "Transformers is a film", "AIMS is a college"]
X_train = vectorizer.transform(train_data2)
print(vectorizer.get_feature_names())
print("\n\nApplying fit transform onto second train data")
X_train = vectorizer.fit_transform(train_data1 + train_data2)
print(vectorizer.get_feature_names())
输出:
Applying first train data
['education', 'football', 'gravity', 'important', 'movie', 'sport']
Applying second train data
['education', 'football', 'gravity', 'important', 'movie', 'sport']
Applying fit transform onto second train data
['aims', 'college', 'cricket', 'education', 'film', 'football', 'gravity', 'important', 'movie', 'sport', 'transformers']
这篇关于将功能名称更新到scikit TFIdfVectorizer的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!