Yahoo Site Explorer Spider
Here you go. On this page you will find a small POC (Proof of Concept) of a client-side (only JavaScript) spider that is based on the top of Yahoo Site Explorer PageData service which you can read more about from this page. I’ve being talking about client-side spiders for quite some time now over here and here and I even came up with POC based on Yahoo Pipes for my OWASP presentation on Advanced Web Hacking Reveled
, which you can find over there.
Web spiders in particular are nothing interesting. They have been with us for quite some time now and there is no point of discussing what they can do. Though, spiders are the first step towards a successful web attack. Obviously, in order to find the weaknesses within a web application, first of all we have to enumerate all entry points. This is where we launch spiders. Sometimes spiders are semi-automatic or completely automatic and may contain attack payloads for propagating themselves. These spiders are also know as worms.
But how is that possible? I though JavaScript cannot access pages outside of the current origin. Is that a browser bug?
Nope! This is not a browser bug. It is a feature of the Web. In my case, I am using Yahoo to provide me with an index of resources crowed by theirs and Google’s spiders. This index is provided as a JSON service. Here is a description of what the service does:
The Page Data service allows you to retrieve information about the subpages in a domain or beneath a path that exist within the Yahoo! index. Yahoo Developer
This is great but how can attackers use this service?
Well the most obvious way to make use of Yahoo Site Explorer service is in the situations where attackers want to find bugs in other sites in real time. Billy Hoffman and I have presented with real life scenarios how XSS vulnerabilities can be found almost automatically from withing the client without the support from a server side technology. This is very dangerous because worms can be written entirely with client-side languages such as JavaScript or ActionScrpt. As such, XSS/AJAX worms are very hard to detect - actually almost impossible.
You see, worms are sometimes quite stupid. They propagate either too fast or too slow. Very often, they are static and attack from specific IP ranges. During the first stage, we can definitely see a raise of particular type of traffic that originates from a particular geographical region. In order to stop further propagation, we can simply block the malicious traffic based on the worm signature. Game Over for the worm. The good guys win!
This makes them super dangerous. Having the ability to spider for new targets and identify vulnerabilities within them, makes AJAX worms even more dangerous and I presume highly demanded in the underground circles.
The spider that I wrote is anything by malicious. It just spiders. However, keep in mind that it will take less then 5 minutes to make it equipped with the latest AJAX exploits. Therefore, I am not responsible for your actions. Be responsible. Here is the spider source code:
http://www.gnucitizen.org/projects/yahoo-site-explorer-spider/spider.jsand this is how I use it:
http://www.gnucitizen.org/projects/yahoo-site-explorer-spider/spider.js
comments
Nice proof of concept. I agree, the speed of this spider and the depth option is awesome! Only a technical question, but can we call this a web spider as it doesn’t really spider the site but rather fetches pages already indexed by Yahoo?
Interesting question. I am not quite sure, I must say. From my prospective, it is still a spider. However, what name do you think will suit best for a tool like this one?
Yahoo calls it “Site Explorer” :) however, its interesting to me that the service supports a ping and update feature, so maybe web spider is just fine.
You are right. Although I seriously doubt that Yahoo will craw the pinged website right away, more likely it will schedule it for some latter point, attackers that are interested in particular sites can ping for updates constantly thus ensuring that Yahoo has the latest index.
nice one pdp. I’m really impressed by the speed of the results! Couldn’t believe it! damn you, you got me excited about this, even though I have other exciting toys to play with on my todo list :)
Regarding the name I would call it a “passive spider”, since it doesn’t actually visit the target site, which makes it more ideal from the cracker’s perspective.
heh, I am working on another one which allows you to perform XSS scans right from the browser.
I only got excited until I read the code. I need a JS spider that actually HITS the site (not fetches pages from a pre-spidered list of yahoo’s).
The Web 2.0 world needs a new JS spider to handle link-checking, perf stats and a host of other things. AJAX components and DHTML screw every spider that is not based on JS and a DOM.
Where can we go? Has anyone written a true JS and DOM based spider (like a FF plug in) that actually spiders???
it is not possible to construct true AJAX spider. Same Origin Policies apply.