The Value Of Automated Security Tests

Tue, 04 Dec 2007 21:28:16 GMT
by pdp

I think that I should speak up how I feel about automated security tests. I don't think that this post will bring much value to you but at least you will be able to see what it feels like from the field. I will try to keep my thoughts short and clean and emphasize on the main points without going too much out of scope. I think that this topic has been already widely discussed so there is no need to waste more time on it. Everyone should make up their own mind.

I personally believe that there is a place for automated security tests although I wont recommend them to anyone who is serious about their security. Automated scanners are only good for identifying vulnerabilities in a bulk manner following predefined types of patterns. We already know that unless we come up with an AI (Artificial Intelligence) type of software we will never be able to provide that much of a value here. The problem is not whether scanners can identify vulnerabilities, the problem is whether they are conscious enough (to have the right amount of semantics and pragmatics) to define these vulnerabilities, to zoom OUT and IN and as such provide global view and more fine-grained one depending on the scenario or the test case.

A lot of the scanning vendors reason that scanners can identify all issues of a given type and this type of service usually cost less and it is performed a lot faster when compared to the service a h4kur can provide, which obviously will cost a lot more and will require a lot more time. They are right! There is no doubt about that. Though, we should decide on the motivation of getting a security test on first place. Why do we need it? What are we trying to achieve by getting it?

If you are a software vendor, automated scanners should definitely be part of your products' lifecycle. If you are a system administrator, you depend on automated security tests in order to reduce the burden of managing such a huge work load on your own. However, if you are an organization which is interested in knowing your real security level, you should probably run away from automated security tests and hire some good penetration testers.

I've mentioned earlier that scanners lack the ability to provide sensible picture based on the gathered data. Simply put, the scanner will craw, scan, prob and report but nothing more. On the other hand, a skillful attacker won't be able to provide you with the level of detail a scanner will, but will be able to give you a lot better description about the current security state of your systems. The attacker will be able to identify the weakest points of your network or application and as such give you more value for your money. In comparison to scanners, this approach is a lot more valuable because scanners will only list vulnerabilities based on their severity level. Keep in mind that HIGH risk issues are not often those that needs to be fixed first, not to mention the fact that successful penetration of a given organization often relies on combination of tricks, which is something scanners cannot come up with.

I will stop right here since I find this topic not very much for my taste, though I've been asked so many times so I through that I can easily refer to this post as soon as I need it. The main principles is to follow your needs. I would personally employ people since they can provide me with the intelligence which is a lot more valuable. Scanners will be the last thing on my mind. They may help you with hardening the perimeter (something very important btw) but they will most probably mislead you if you trust them too much. That said, I am done on this topic.

Archived Comments

In In think this subject can be detailed in many ways, and can lead to many different opinions. Hiring a h3ker would be a plus considering the interpretation it might give to his analysis.. But good choice “analysis”, because often that we might admit, it relays (al least in some points) on…”scanners” Of course it relays on scanners, and more often is ..nmap. Why? Cause It’s the nearest tool to see what services are available in a public manner and which of them are not suppose to be. As you mention…”combinations of tricks” is the key. Yes, it is! No scanner I know can do this (at the moment), but I don’t really find it impossible at all..scenarios can be scripted in templates and modulated as an AI…like in the games (I’m no gamer, ..just to mention). So, getting back to “combination of tricks”, a h3ker mind can go much further that considering some port opened, or not, ..or a service available or not. It can do tests based on the vulnerabilities of that version..or missconfigurations. And this is a big +. Of course hiring a h3ker is somehow ..not enough..he need the tools..just his AI is just not enough, and no one has that time to reinvent the wheel…to script new nmap (just an example) to scan a server. Is a conclusion of my thoughts…I think scanners without AI is to few..and a good AI without the tools big time consumption, effort and it came be easily a failure.
flipper, yes of coures, a h4kur without tools is of no use to anyone. However, my point was quite different. All I am trying to say here is that while automated test (automated vulnerability assestment tools, this does not include nmap but mostly Nessus and the clones plus those used in the WebApp world) are of no use if you really want to know how an attacker can penetrate your network or applications. They will show you all your vulnerabilities (as far as they can) but they won't show you how an attacker can gain access without using any vulnerability whatsoever or how an attacker can come up with a combination of attacks. It think that it is really a matter of choice.
Always is a matter of choice..and the choose you make is always a compromise. Companies, often believe in what other companies do..that's why often they copy mistakes from one to each other (way solution are implemented, equipments brands, software, OS..almost everything) unavoidable they believe in others companies testing software. Of course it's only about scanning for some vulnerabilities...but companies risk a lot..very much. They think that covering with patches and updates will solve the problem.Nope. No way u can do it like this. Becoming a h4ker..knowing what can be done is the way..But of use this valuable knowledge ('cause u won't find "Hacking for Idiots" on every corner as being a good material, and it's perfectly understandable) in a good manner, for protecting yourself and upgrade your security level. In this case... definitly u need a h3ker to evaluate, or become one... :)
nice wrap up :)
Thanx, and they call it "audit company". I guess most of us been through such experience.
I do believe that a skilled h4kur will produce much more valuable report than a scanner but: Today big organizations usually hire more than one penetration testing team, since they want different opinion about their security deployment. A mix between a person and an automated script/service will produce a reliable report. 2 days ago in OWASP Israel conference i came across a new Startup company that provide an online automated penetration service, (, they tailor their service to each customer. Combining services such as this PLUS a real manual penetration test twice a year by a skilled individual is the right approach. P.S - pdp i agree with you, that's a dark topic that no-one really wants to talk about, so i'm done here as well :)
People (sysadmin's I worked with) would tell me that you could never automate system builds, but as the tools got better and the faith in the tools got better the reliance on automation increased. Currently some companies rely on automation so much that they do not have the skilled labor to know when their automation has gone to the dogs. Automation should only be used to assist a professional and not an a replacement for the skills that are needed to protect your data/infrastructure/person. However I am repeating what has been said before, it is worth repeating.