我正在尝试使用tidytext使用双字母组和三字母组合。我可以为 token 使用什么代码来查找2个和3个单词。
这是仅使用bigrams的代码:
library(tidytext)
library(janeaustenr)
austen_bigrams <- austen_books() %>%
unnest_tokens(bigram, text, token = "ngrams", n = 2)
austen_bigrams
最佳答案
如果查看?unnest_tokens
,它将告诉您...
是传递给 token 生成器的参数的。对于ngram,这是tokenizers::tokenize_ngrams
,如果您查看其帮助页面,它具有n_min
参数,因此您可以执行
library(magrittr)
library(tidytext)
library(janeaustenr)
austen_bigrams <- austen_books() %>%
head(1000) %>% # otherwise this will get very large
unnest_tokens(bigram, text, token = "ngrams", n = 3, n_min = 2)
austen_bigrams
#> # A tibble: 19,801 x 2
#> book bigram
#> <fctr> <chr>
#> 1 Sense & Sensibility sense and
#> 2 Sense & Sensibility sense and sensibility
#> 3 Sense & Sensibility and sensibility
#> 4 Sense & Sensibility and sensibility by
#> 5 Sense & Sensibility sensibility by
#> 6 Sense & Sensibility sensibility by jane
#> 7 Sense & Sensibility by jane
#> 8 Sense & Sensibility by jane austen
#> 9 Sense & Sensibility jane austen
#> 10 Sense & Sensibility jane austen 1811
#> # ... with 19,791 more rows