且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

使用JSoup抓取Google结果

更新时间:2023-02-19 16:55:10

在这里.

public class ScanWebSO 
{
public static void main (String args[])
{
    Document doc;
    try{
        doc =        Jsoup.connect("https://www.google.com/search?as_q=&as_epq=%22Yorkshire+Capital%22+&as_oq=fraud+OR+allegations+OR+scam&as_eq=&as_nlo=&as_nhi=&lr=lang_en&cr=countryCA&as_qdr=all&as_sitesearch=&as_occt=any&safe=images&tbs=&as_filetype=&as_rights=").userAgent("Mozilla").ignoreHttpErrors(true).timeout(0).get();
        Elements links = doc.select("li[class=g]");
        for (Element link : links) {
            Elements titles = link.select("h3[class=r]");
            String title = titles.text();

            Elements bodies = link.select("span[class=st]");
            String body = bodies.text();

            System.out.println("Title: "+title);
            System.out.println("Body: "+body+"\n");
        }
    }
    catch (IOException e) {
        e.printStackTrace();
    }
}
}

此外,我建议您自己使用chrome.您只需右键单击要刮取的任何内容,然后检查元素.它将带您到该元素位于html的确切位置.在这种情况下,您首先要找出所有结果列表的根目录在哪里.找到该元素时,您要指定元素,***指定一个唯一的属性进行搜索.在这种情况下,根元素是

Also, to do this yourself I would suggest using chrome. You just right click on whatever you want to scrape and go to inspect element. It will take you to the exact spot in the html where that element is located. In this case you first want to find out where the root of all the result listings are. When you find that, you want to specify the element, and preferably an unique attribute to search it by. In this case the root element is

<ol eid="" id="rso">

下面,您会看到一堆以

<li class="g"> 

这是要放入初始元素数组中的内容,然后对于每个元素,您都希望找到标题和正文所在的位置.在这种情况下,我发现标题位于

This is what you want to put into your initial elements array, then for each element you will want to find the spot where the title and body are. In this case, I found the title to be under the

<h3 class="r" style="white-space: normal;">

元素.因此,您将在每个清单中搜索该元素.身体也一样.我发现主体位于下面,因此我使用.text()方法进行搜索,然后返回该元素下的所有文本.关键是始终尝试查找具有原始属性的元素(使用类名是理想的).如果不这样做,仅搜索"div"之类的内容,它将在整个页面中搜索包含div的ANY元素并返回该元素.因此,您将获得比您想要的更多的结果.我希望这能很好地解释它.让我知道您是否还有其他问题.

element. So you will search for that element in each listing. The same goes for the body. I found the body to be under so I searched for that using the .text() method and it returned all the text under that element. The key is to ALWAYS try and find the element with an original attribute (using a class name is ideal). If you don't and only search for something like "div" it will search the entire page for ANY element containing div and return that. So you will get WAY more results than you want. I hope this explains it well. Let me know if you have any more questions.