The spider that is presented here, relays on TidyURL pipe that I've created for this proof of concept. The spider depth is set to 3 (default). This implementation is flexible enought to use any other pipe with similar properties.
code that can traverse the web
The spider that is presented here, relays on TidyURL pipe that I've created for this proof of concept. The spider depth is set to 3 (default). This implementation is flexible enought to use any other pipe with similar properties.