Comments on: Google.py

By: pdp

pdp — Tue, 17 Mar 2009 16:08:22 +0000

the code is old! perhaps you need to change the regexes slightly…

By: vatts

vatts — Sun, 15 Mar 2009 13:21:29 +0000

Hey why do i run the code and get just this:

[vatts@VattServ Desktop]$ python google.py 
usage: google.py query
[vatts@VattServ Desktop]$ python google.py linux
# linux
# crawls the first 5 pages only (50 results max)
[vatts@VattServ Desktop]$

By: pdp

pdp — Sat, 09 Feb 2008 19:58:24 +0000

might be a good idea to collaborate.

By: barbsie

barbsie — Sat, 09 Feb 2008 08:36:51 +0000

interesting :) . I’m working on something similar, together with someone else. Basically it combines dns enumeration with some google scraping and whois stuff. It’s a major rewrite of the old dnsenum thingy. If you are interested to exchange ideas, drop me a mail…

By: pdp

pdp — Sat, 09 Feb 2008 08:21:57 +0000

yep, good enough solution to evade Google's bot detection mechanism. I was planning to implement all these stuff into the next release of MET (Massive Enumeration Toolset), but I never find time for it. I hope that the project is not dead just yet.

By: barbsie

barbsie — Fri, 08 Feb 2008 12:58:12 +0000

Hi,

I also wrote a googlescraper in ruby. During my tests I had the impression that google not only checks the agent. Therefore I also added random delay in getting next pages and resetting cookies on top of randomizing agents. Oh…and also the use of a proxy (so privoxy with tor can be used)…

By: pdp

pdp — Wed, 06 Feb 2008 10:58:36 +0000

10x for sharing. I will have a look.

By: GooHackle

GooHackle — Wed, 06 Feb 2008 03:49:30 +0000

First of all, congratulations for this site.

Now about this post, back in the hot google hacking days I wrote a similar script but in Perl. Some months ago I wrote it in PHP and publish a online version of this “google parser” named GooParser to test a couple of things. You can try it here: Google Parser

Cheers,
GooHackle

By: Willy

Willy — Thu, 17 Jan 2008 08:11:48 +0000

I noticed that in the script. I wish I’d known it when I was writing mine. It still seemed to work to just search through someone else’s site that didn’t check the user agent, but knowing that would have saved me about an hour of messing around trying to get past the forbidden error.

By: zaggy

zaggy — Wed, 16 Jan 2008 10:13:07 +0000

C:\shared>google.py define:lol
# define:lol
# crawls the first 5 pages only (50 results max)

Did I miss something?

By: pdp

pdp — Tue, 15 Jan 2008 14:30:50 +0000

Willy, the Google.py code shows how to change the user agent. You can just grab the Get object and use it in your projects.

By: Willy

Willy — Tue, 15 Jan 2008 14:23:17 +0000

If you put this in an infinite loop you can mess with google trends, although the results are only changed for whatever region you’re connected through.

I didn’t know how to change the user agent from python when I wrote mine though, so I did it through another site that returns results from Google, they don’t check.

By: Garrett Gee

Garrett Gee — Sat, 22 Dec 2007 19:58:31 +0000

Many thanks for this python module. I recently started a project that needs to leverage search engine results, and I was very sad to see google removing their soap interface over the ajax one.

By: Script en python para bÃºsquedas en google // menéame

Script en python para bÃºsquedas en google // menéame — Sat, 22 Dec 2007 14:29:25 +0000

[...] Script en python para bÃºsquedas en googlewww.gnucitizen.org/projects/google_py/ por Carme hace pocos segundos [...]