我试图从http://www.childrenshospital.org/directory?state=%7B%22showLandingContent%22%3Afalse%2C%22model%22%3A%7B%22search_specialist%22%3Afalse%2C%22search_type%22%3A%5B%22directoryphysician%22%2C%22directorynurse%22%5D%7D%2C%22customModel%22%3A%7B%22nurses%22%3Atrue%7D%7D抓取各个提供商的网址。
我查看了页面来源并确定了感兴趣的网址。例如，我要从以下节点中刮取“ http://www.childrenshospital.org/doctors/mirna-aeschlimann”
<a data-layer-event="searchClick" data-bind="attr: {href: model.Url}" href="http://www.childrenshospital.org/doctors/mirna-aeschlimann"><!--ko text: model.FirstName-->Mirna<!--/ko--><!--ko text: ' ' + model.LastName--> Aeschlimann<!--/ko--><!--ko if: model.Suffix-->, <!--ko text: model.Suffix-->MD<!--/ko--><!--/ko--></a>
我尝试了以下代码。但是，由于某些原因，它没有返回上面的节点。
base_html <- "http://www.childrenshospital.org/directory?state=%7B%22showLandingContent%22%3Afalse%2C%22model%22%3A%7B%22search_specialist%22%3Afalse%2C%22search_type%22%3A%5B%22directoryphysician%22%2C%22directorynurse%22%5D%7D%2C%22customModel%22%3A%7B%22nurses%22%3Atrue%7D%7D"
doc <- htmlTreeParse(base_html, useInternal = TRUE)
任何帮助将不胜感激。请让我知道是否需要提供更多信息。
最佳答案
您是否尝试针对网站提出的XHR请求以获取数据？
library(httr)
library(purrr)
library(xml2)
library(dplyr)
library(jsonlite)
map_df(1:17, function(i) {
POST("http://www.childrenshospital.org/searchdirectory.ajax",
body = list(search_query = "",
search_specialties = "",
search_languages = "",
search_gender = "", search_departments = "",
search_programs = "", search_userlocation = "",
search_radius = "10", search_pcp = "true",
search_specialist = "false",
search_type = "directorynurse|directoryphysician",
search_letter = "", search_querygroup = "dirnametext",
search_page = "10"),
encode = "form") -> res
content(res, as="text") %>%
fromJSON() %>%
.$Records %>%
mutate(Address=xml2::xml_text(xml2::read_html(paste0("<x>", Address, "</x>")))) %>%
tbl_df()
}) -> tmp_df
glimpse(tmp_df)
## Observations: 408
## Variables: 21
## $ ID <chr> "{E8ECAF3B-B49C-4CD8-AB16-6CE63F0379C0}", "{1E1...
## $ FirstName <chr> "Jonathan", "Barbara", "Mark", "Maura", "Sarah"...
## $ LastName <chr> "Schwab", "Seagle", "Shapira", "Shea", "Sheldon...
## $ Image <chr> "/~/media/directory/physicians/schwab_jonathan....
## $ Suffix <chr> "MD", "MD", "MD", "MD", "MD", "MD", "MD", "MD",...
## $ Url <chr> "http://www.childrenshospital.org/doctors/jonat...
## $ Gender <chr> "male", "female", "male", "female", "female", "...
## $ Latitude <chr> "42.3344382", "42.326435", "41.559642", "42.423...
## $ Longitude <chr> "-72.6618324", "-71.149499", "-70.939315", "-71...
## $ Address <chr> "{\"practice_name\":\"Northampton Area Pediatri...
## $ Distance <chr> "", "", "", "", "", "", "", "", "", "", "", "",...
## $ OtherLocations <chr> "", "", "", "", "", "", "", "", "", "", "Westwo...
## $ AcademicTitle <chr> "", "", "", "", "", "", "", "", "", "", "", "",...
## $ HospitalTitle <chr> "Pediatrician", "Pediatrician", "Pediatrician",...
## $ Specialties <chr> "Pediatrics", "General Pediatrics, Pediatrics, ...
## $ Departments <chr> "", "General Pediatrics", "General Pediatrics",...
## $ Languages <chr> "", "English", "", "English", "English", "", "E...
## $ PPOCLink <chr> "http://www.childrenshospital.org/patient-resou...
## $ Gallery <chr> "", "", "", "", "", "", "", "", "", "", "", "",...
## $ Phone <chr> "(413) 584-8700", "(617) 731-0200", "(508) 996-...
## $ Fax <chr> "(413) 584-1714", "(617) 731-0289", "(508) 992-...
关于javascript - 在R中使用JavaScript从网页上抓取链接，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/41435049/