且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

提取“隐藏” HTML与Jsoup

更新时间:2023-12-03 20:53:16

数据似乎是用AJAX加载的。 JSoup不处理Javascript。



您需要的是一个无头浏览器API,它可以在不实际渲染任何内容的情况下处理Javascript。



HtmlUnit似乎是最知名的工具,尽管我自己从未使用它。如前所述,Selenium Webdriver也是一个选项。



我相信你将不得不加载URL,等待所有的AJAX进行处理,最终你会得到几乎您可以在Java中使用与Chrome相同的解析树来处理它!


I am trying to get at HTML data that does not appear in the source document but can be exposed, for example, by "inspect element" in Google Chrome.

Example page: http://assignment.uspto.gov/#/search?q=9000000&sort=patAssignorEarliestExDate%20desc%2C%20id%20desc&synonyms=false

There are a number of div elements containing assignment data for U.S. Patent No. 9,000,000 that appear below the line

<script async="async" type="text/javascript" src="https://components.uspto.gov/js/ais/2-2-assignment-search.js"></script>

Is there a way to extract this hidden html with Jsoup?

The data seems to loaded with AJAX. JSoup does not process Javascript.

What you need is a "headless browser" API, that processes Javascript without actually rendering anything.

HtmlUnit seems to be the best known tool, although I've never used it myself. As suggested before, Selenium Webdriver is also an option.

I believe you will have to load the URL, wait for all the AJAX to process, and you will eventually get almost the same parse tree you get in Chrome in Java to do with it as you wish!