问题描述
我在R中玩耍以找到tf-idf
值.
I am playing around in R to find the tf-idf
values.
我有一组documents
,例如:
D1 = "The sky is blue."
D2 = "The sun is bright."
D3 = "The sun in the sky is bright."
我想创建一个像这样的矩阵:
I want to create a matrix like this:
Docs blue bright sky sun
D1 tf-idf 0.0000000 tf-idf 0.0000000
D2 0.0000000 tf-idf 0.0000000 tf-idf
D3 0.0000000 tf-idf tf-idf tf-idf
所以,我在R
中的代码:
library(tm)
docs <- c(D1 = "The sky is blue.", D2 = "The sun is bright.", D3 = "The sun in the sky is bright.")
dd <- Corpus(VectorSource(docs)) #Make a corpus object from a text vector
#Clean the text
dd <- tm_map(dd, stripWhitespace)
dd <- tm_map(dd, tolower)
dd <- tm_map(dd, removePunctuation)
dd <- tm_map(dd, removeWords, stopwords("english"))
dd <- tm_map(dd, stemDocument)
dd <- tm_map(dd, removeNumbers)
inspect(dd)
A corpus with 3 text documents
The metadata consists of 2 tag-value pairs and a data frame
Available tags are:
create_date creator
Available variables in the data frame are:
MetaID
$D1
sky blue
$D2
sun bright
$D3
sun sky bright
> dtm <- DocumentTermMatrix(dd, control = list(weighting = weightTfIdf))
> as.matrix(dtm)
Terms
Docs blue bright sky sun
D1 0.7924813 0.0000000 0.2924813 0.0000000
D2 0.0000000 0.2924813 0.0000000 0.2924813
D3 0.0000000 0.1949875 0.1949875 0.1949875
如果我进行手工计算,则矩阵应为:
If I do a hand calculation then the matrix should be:
Docs blue bright sky sun
D1 0.2385 0.0000000 0.3521 0.0000000
D2 0.0000000 0.3521 0.0000000 0.3521
D3 0.0000000 0.1949875 0.058 0.058
我正在像说blue
那样计算tf
= 1/2 = 0.5
,并且像log(3/1) = 0.477121255
那样计算idf
.因此tf-idf = tf*idf = 0.5*0.477 = 0.2385
.这样,我正在计算其他tf-idf
值.现在,我想知道为什么在手计算矩阵和R矩阵中得到不同的结果?哪个给出正确的结果?我是在手工计算中做错了什么,还是我的R代码有问题?
I am calculating like say blue
as tf
= 1/2 = 0.5
and idf
as log(3/1) = 0.477121255
. Therefore tf-idf = tf*idf = 0.5*0.477 = 0.2385
. In this way, I am calculating the other tf-idf
values. Now, I am wondering, why I am getting different results in the matrix of hand calculation and in the matrix of R? Which gives the correct results? Am I doing something wrong in hand calculation or is there something wrong in my R code?
推荐答案
您的手工计算与DocumentTermMatrix计算不同的原因是您使用的是不同的log
基数.当您说log(3/1) = 0.477121255
时,您必须使用对数为10的日志.在R中,它将为log10(3)
. R中的默认log
是自然对数,因此,如果在R中键入log(3)
,则得到〜1.10.但是weightTfIdf使用对数基数2进行计算.因此,当为蓝色"计算tf-idf时,您会得到
The reason your hand calculation doesn't agree with the DocumentTermMatrix calculation is you are using a different log
base. When you say log(3/1) = 0.477121255
you must be using log base 10. In R, that would be log10(3)
. The default log
in R is natural log so if you type log(3)
in R you get ~1.10. But the weightTfIdf uses log base 2 for its calculations. Thus when calculating tf-idf for "blue" you get
(1/2)*log2(3/1) = 0.7924813
我希望一切都可以清除.
I hope that clears things up.
这篇关于R和手计算中的不同tf-idf值的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!