通过 Nokogiri 获取可见的文本词

问题描述

我想用 Nokogiri 打开一个网页，提取用户在浏览器中访问该页面时看到的所有单词并分析词频.

I'd like to open a web page with Nokogiri and extract all the words that a user sees when they visit the page in a browser and analyze the word frequency.

使用 nokogiri 从 html 文档中获取所有可读单词的最简单方法是什么?理想的代码片段将采用一个 html 页面(比如一个文件)，并给出来自所有类型的可读元素的单个单词的数组.

What is the easiest way of getting all readable words out of an html document with nokogiri? The ideal code snippet would take a html page (as a file, say) and give an array of individual words that come from all types of elements that are readable.

(无需担心 javascript 或 css 隐藏元素从而隐藏单词；只要所有设计用于显示的单词都可以.)

(No need to worry about javascript or css hiding elements and thus hiding words; just all words designed for display is fine.)

推荐答案

您想要 Nokogiri::XML::Node#inner_text 方法:

You want the Nokogiri::XML::Node#inner_text method:

require 'nokogiri'
require 'open-uri'
html = Nokogiri::HTML(open 'http://stackoverflow.com/questions/6129357')

# Alternatively
html = Nokogiri::HTML(IO.read 'myfile.html')

text  = html.at('body').inner_text

# Pretend that all words we care about contain only a-z, 0-9, or underscores
words = text.scan(/\w+/)
p words.length, words.uniq.length, words.uniq.sort[0..8]
#=> 907
#=> 428
#=> ["0", "1", "100", "15px", "2", "20", "2011", "220px", "24158nokogiri"]

# How about words that are only letters?
words = text.scan(/[a-z]+/i)
p words.length, words.uniq.length, words.uniq.sort[0..5]
#=> 872
#=> 406
#=> ["Answer", "Ask", "Badges", "Browse", "DocumentFragment", "Email"]

# Find the most frequent words
require 'pp'
def frequencies(words)
  Hash[
    words.group_by(&:downcase).map{ |word,instances|
      [word,instances.length]
    }.sort_by(&:last).reverse
  ]
end
pp frequencies(words)
#=> {"nokogiri"=>34,
#=>  "a"=>27,
#=>  "html"=>18,
#=>  "function"=>17,
#=>  "s"=>13,
#=>  "var"=>13,
#=>  "b"=>12,
#=>  "c"=>11,
#=>  ...

# Hrm...let's drop the javascript code out of our words
html.css('script').remove
words = html.at('body').inner_text.scan(/\w+/)
pp frequencies(words)
#=> {"nokogiri"=>36,
#=>  "words"=>18,
#=>  "html"=>17,
#=>  "text"=>13,
#=>  "with"=>12,
#=>  "a"=>12,
#=>  "the"=>11,
#=>  "and"=>11,
#=>  ...

这篇关于通过 Nokogiri 获取可见的文本词的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！