我正在尝试为我的 Ted Talks 自定义数据集运行 this scikit example code。
每个目录是一个主题，其下是包含每个 Ted Talk 描述的文本文件。
这就是我的数据集树结构的方式。如您所见，每个目录都是一个主题，在其下方是带有描述的文本文件。
Topics/
|-- Activism
| |-- 1149.txt
| |-- 1444.txt
| |-- 157.txt
| |-- 1616.txt
| |-- 1706.txt
| |-- 1718.txt
|-- Adventure
| |-- 1036.txt
| |-- 1777.txt
| |-- 2930.txt
| |-- 2968.txt
| |-- 3027.txt
| |-- 3290.txt
|-- Advertising
| |-- 3673.txt
| |-- 3685.txt
| |-- 6567.txt
| `-- 6925.txt
|-- Africa
| |-- 1045.txt
| |-- 1072.txt
| |-- 1103.txt
| |-- 1112.txt
|-- Aging
| |-- 1848.txt
| |-- 2495.txt
| |-- 2782.txt
|-- Agriculture
| |-- 3469.txt
| |-- 4140.txt
| |-- 4733.txt
| |-- 4939.txt
我以这种形式制作了我的数据集，以类似于树结构如下的 20news 组:
20news-18828/
|-- alt.atheism
| |-- 49960
| |-- 51060
| |-- 51119
|-- comp.graphics
| |-- 37261
| |-- 37913
| |-- 37914
| |-- 37915
| |-- 37916
| |-- 37917
| |-- 37918
|-- comp.os.ms-windows.misc
| |-- 10000
| |-- 10001
| |-- 10002
| |-- 10003
| |-- 10004
| |-- 10005
在 original code (98-124) 中，这是直接从 scikit 加载训练和测试数据的方式。
print("Loading 20 newsgroups dataset for categories:")
print(categories if categories else "all")
data_train = fetch_20newsgroups(subset='train', categories=categories,
shuffle=True, random_state=42,
remove=remove)
data_test = fetch_20newsgroups(subset='test', categories=categories,
shuffle=True, random_state=42,
remove=remove)
print('data loaded')
categories = data_train.target_names # for case categories == None
def size_mb(docs):
return sum(len(s.encode('utf-8')) for s in docs) / 1e6
data_train_size_mb = size_mb(data_train.data)
data_test_size_mb = size_mb(data_test.data)
print("%d documents - %0.3fMB (training set)" % (
len(data_train.data), data_train_size_mb))
print("%d documents - %0.3fMB (test set)" % (
len(data_test.data), data_test_size_mb))
print("%d categories" % len(categories))
print()
# split a training set and a test set
y_train, y_test = data_train.target, data_test.target
由于这个数据集在 Scikit 中可用，它的标签等都是内置的。
就我而言，我知道如何加载数据集 (Line 84) :
dataset = load_files('./TED_dataset/Topics/')
我不知道在那之后我应该做什么。我想知道我应该如何在训练和测试中拆分这些数据并从我的数据集中生成这些标签:
data_train.data, data_test.data
总而言之，我只想加载我的数据集，在此代码上无错误地运行它。我有 uploaded the dataset here 给那些想看的人。
我提到了 this question，它简要介绍了测试列车加载。我还想知道如何从我的数据集中获取 data_train.target_names。
编辑:
我试图让火车和测试返回错误:
dataset = load_files('./TED_dataset/Topics/')
train, test = train_test_split(dataset, train_size = 0.8)
更新的代码是 here 。
最佳答案
我想你正在寻找这样的东西:
In [1]: from sklearn.datasets import load_files
In [2]: from sklearn.cross_validation import train_test_split
In [3]: bunch = load_files('./Topics')
In [4]: X_train, X_test, y_train, y_test = train_test_split(bunch.data, bunch.target, test_size=.4)
# Then proceed to train your model and validate.
请注意，
bunch.target 是一个整数数组，它们是存储在 bunch.target_names 中的类别名称的索引。In [14]: X_test[:2]
Out[14]:
['Psychologist Philip Zimbardo asks, "Why are boys struggling?" He shares some stats (lower graduation rates, greater worries about intimacy and relationships) and suggests a few reasons -- and challenges the TED community to think about solutions.Philip Zimbardo was the leader of the notorious 1971 Stanford Prison Experiment -- and an expert witness at Abu Ghraib. His book The Lucifer Effect explores the nature of evil; now, in his new work, he studies the nature of heroism.',
'Human growth has strained the Earth\'s resources, but as Johan Rockstrom reminds us, our advances also give us the science to recognize this and change behavior. His research has found nine "planetary boundaries" that can guide us in protecting our planet\'s many overlapping ecosystems.If Earth is a self-regulating system, it\'s clear that human activity is capable of disrupting it. Johan Rockstrom has led a team of scientists to define the nine Earth systems that need to be kept within bounds for Earth to keep itself in balance.']
In [15]: y_test[:2]
Out[15]: array([ 84, 113])
In [16]: [bunch.target_names[idx] for idx in y_test[:2]]
Out[16]: ['Education', 'Global issues']
关于python - 在 Scikit 中加载自定义数据集(类似于 20 个新闻组集)用于文本文档的分类，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/33612296/