Comments on: Yahoo Site Explorer Spider

By: pdp

pdp — Sat, 01 Mar 2008 07:08:36 +0000

it is not possible to construct true AJAX spider. Same Origin Policies apply.

By: Antonio Vivaldi

Antonio Vivaldi — Fri, 29 Feb 2008 12:05:26 +0000

I only got excited until I read the code. I need a JS spider that actually HITS the site (not fetches pages from a pre-spidered list of yahoo’s).

The Web 2.0 world needs a new JS spider to handle link-checking, perf stats and a host of other things. AJAX components and DHTML screw every spider that is not based on JS and a DOM.

Where can we go? Has anyone written a true JS and DOM based spider (like a FF plug in) that actually spiders???

By: JavaScript XSS Scanner | GNUCITIZEN

JavaScript XSS Scanner | GNUCITIZEN — Mon, 16 Jul 2007 19:13:08 +0000

[...] which I promised to realize after I publish the Yahoo Site Explorer Spider discussed in detail over here. Before going any further, I must say a few words about the legal side of using this tool. [...]

By: Liquidmatrix Security Digest » Yahoo Site Explorer Spider

Liquidmatrix Security Digest » Yahoo Site Explorer Spider — Mon, 16 Jul 2007 13:12:13 +0000

[...] Article Link [...]

By: pdp

pdp — Mon, 16 Jul 2007 13:10:24 +0000

heh, I am working on another one which allows you to perform XSS scans right from the browser.

By: Adrian Pastor

Adrian Pastor — Mon, 16 Jul 2007 12:34:09 +0000

nice one pdp. I’m really impressed by the speed of the results! Couldn’t believe it! damn you, you got me excited about this, even though I have other exciting toys to play with on my todo list :)

Regarding the name I would call it a “passive spider”, since it doesn’t actually visit the target site, which makes it more ideal from the cracker’s perspective.

By: pdp

pdp — Mon, 16 Jul 2007 08:24:52 +0000

You are right. Although I seriously doubt that Yahoo will craw the pinged website right away, more likely it will schedule it for some latter point, attackers that are interested in particular sites can ping for updates constantly thus ensuring that Yahoo has the latest index.

By: David Kierznowski

David Kierznowski — Mon, 16 Jul 2007 06:29:07 +0000

Yahoo calls it “Site Explorer” :) however, its interesting to me that the service supports a ping and update feature, so maybe web spider is just fine.

By: pdp

pdp — Mon, 16 Jul 2007 06:22:23 +0000

Interesting question. I am not quite sure, I must say. From my prospective, it is still a spider. However, what name do you think will suit best for a tool like this one?

By: David Kierznowski

David Kierznowski — Mon, 16 Jul 2007 06:18:22 +0000

Nice proof of concept. I agree, the speed of this spider and the depth option is awesome! Only a technical question, but can we call this a web spider as it doesn’t really spider the site but rather fetches pages already indexed by Yahoo?