Automated Web Foo or Fud!

Jeremiah is the most outspoken that I have seen regarding the effectives of automated web application tools. His recent post, Are web application scanners ***ing useless?, almost sounds frustrated. While developing the initial version of the Technika Security Framework, I have really had a chance to think about this, which I haven't done since an OWASP presentation I attended 2-3 years ago, anyone have the link for this?

Firstly, as always I love offering life experiences instead of just shooting air. Some time ago, a friend of mine decided to try out an automated web application tool against a client site... it turned into one of those, "unwise in hindsight" experiences! Whilst the tool ran over night, what he would soon find out, is that the tool would submit over 10000 requests to the application, and that this data was being stored on the customers backend database, which was then being replicated and used in other applications. What is worse, the company being tested were a group of corporate lawyers, ouch! In short, it was a painful lesson, and luckily the company got away without a lawsuit!

My second experience was with another good friend of mine. The project was for a large bank, which had already used a popular automated web application tool, and wanted a manual test for benchmarking and future planning. Again, the experience was negative, the automated scanner found some issues, but missed a critical business logic risk which made this particular bank say yes, to us, and limit the tools usage.

The challenges I see with automated web app tools are these:

This list is by know means comprehensive but just some initial thoughts I've had over the past few weeks. I do see alot of limitations with these tools. However, I really think well-thought and developed tools can really aid a tester and make the job a little easier, if you have the money and time. So what's left in my opinion? Two things. Of course, I love the Technika Sec Framework concept, as it utilises the browser, and therefore allows us to focus our code and time on the important stuff rather then worrying whether the tool is RFC, and whether it supports A,B & C. Also, because its JavaScript and open, developers far better than I, can have a platform to design some kick ass tools. Second, I really love the HTTP proxy concept, in the form of tamper data or paros type tools.

We hope to use TSF in the future hijack the DOM to provide these features for us. To me, this is the way forward, but of course I am bias :-)