Tiger Team Operations vs. Penetration Tests
If you read the Wikipedia’s definition of Tiger Team you get the following: A tiger team is a specialized group tasked with testing the effectiveness of an organization’s ability to protect assets by attempting to circumvent, defeat or otherwise thwart that organization’s internal and external security
. And further down we have In the computer security field, the term is now obsolete, and more common terms are penetration testers or security testers. Security assessment testing of a computer system or network infrastructure is called penetration testing
, which I find very untrue.
There is a significant difference between a tiger team operation and a penetration test. They differentiate largely in terms of quality, pricing and also the time frame which is allocated for each project. Let’s have a look at these differences.
Quality
It is needless to say that the tiger team operations will produce more quality if this is what you are after. Tiger Team operations involve more then one expert in the info security field. Not to mention that each expert specializes or s/he is good at in a different area all together when compared to the rest of the participants. This adds a lot of value and it works a lot better in the long term for companies/organizations who are interested in protecting their digital assets.
When a tiger team operation is established, there is a lot brainstorming involved. This usually leads to greater input and therefore much better job. Simply put, the more heads are thinking on the same problem, the more solutions you will get and much more quality is provided as a result.
Penetration tests, from what I can see from the market today, usually involve only one person. I must admit that I’ve seen penetration tests which consisted of more then one info sec expert but all of them specializing in the same field. As you probably guess, this is not very good from creative input point of view since all experts will tackle the problem from the exact same angle. Therefore, the quality is much lower.
Pricing
Tiger team operations cost a lot more when compared to penetration tests, because they involve several experts for a longer period if time, as you will see in the next section. A single tiger team operation may take a lot of money but at the end of the day you get what you pay for. You can buy jeans from the local market for 5-a but if you want the quality stuff you might want to get the American denim which will cost you a lot more.
In UK for example, anything that is less then £1000 per-day onsite work should tell you that the people who will test you will run Nessus and this is how far their commitment to your situation goes. Still, many companies are doing exactly this. In some very rare situations you get good stuff for not much but this is very, very rare. Probably you’ve hired a good startup company which does not know how much to charge you just yet.
Time frames
Tiger Team operations usually take more time then standard penetration tests. Why? Because they are custom tailored for the specific situation. Strategic planning is the key. But on the good side of the things, you don’t have to attend the team progress on every single step. The quality and professionalism speak for themselves. So, in general you do a better job by not investing your time which usually costs you money.
Penetration tests are very narrowed and can take up to a single day which in some cases is enough in others is just the start but if it is a pentest then what is done is done and this is how much you get otherwise you have to pay more, which may not be enough and which again, takes up of your time. As you can see this is a mess.
Conclusion
I guess I am bias as being the leader of the only tiger team in UK but I wouldn’t have been part of such initiative unless I believe in its values and qualities. There are many differences between both types of services and they all fit different types of clients. Therefore, both of them fit different needs. It is up to the client to decide what they really need.
comments
Thing is though, in the UK most of the serious teams are part of the CHECK scheme run by CESG. Such teams operate exactly as you describe with multidiciplined team members selected for aptitude for the given environment. Exploitation is part of the playbook and such companies have exellent reps globally. Here’s the rub though, such compazanies and there team members are certified by GCHQ ensuring technical quality as well as trustworthiness. That’s why noone here talks of tiger teams. You’re right tho, Nessus doesn’t cut it!
“In UK for example, anything that is less then £1000 per-day onsite work should tell you that the people who will test you will run Nessus and this is how far their commitment to your situation goes. Still, many companies are doing exactly this. In some very rare situations you get good stuff for not much but this is very, very rare. Probably you’ve hired a good startup company which does not know how much to charge you just yet.”
Clearly you never did anything under PITO then :)
You’ll be surprised how cheap good pentesters are these days in the UK. It’s largely a commodity market. The reason pentesting is dying a death is because the market’s flooded with average testers (regardless of price) and things like automated PCI scanners add to the economic pressure. Security consultancies need differentiators to survive, and I agree about the tiger team differential as well as with the point on the number of consultants on a pentest, but network testing cannot compete (economically speaking) with automated scanning in a flooded commoditised market.
The CHECK scheme is nice but still a basic start. It will be soon replaced by CREST which is more relaxed in terms of requirements but harder in terms of getting the actual certification. Although both are popular, they are there for a reason. And the reason is to control the security services market in UK and potentially globally. For example, the price tags for CHECK and CREST are dictated by the government pretty much. And soon you will have a bunch of trained monkeys to fill the gaps in case you feel like complaining regarding the actual value you are providing but this is yet to happen and I guess for now it works well. Just to clarify, I am not saying that CHECK guys are not good, in fact I would say that they are some of the best you can get at the moment.
I don’t know, what is your opinion?
Bob, there are plenty of good pentesters that work on contract basis. You can get even some for a 200 pounds a day but for that money I don’t think that it works for them if it is a once-off thing. If it is between 3 to 6 months contract then it could work I guess. But yes, UK is a weired market and I hope it will do well after all these problems around the constant increases of expanses and the general instability of the market.
@pdp,
A ’small point of order’ if I may?
CREST will not replace CHECK but instead be the scheme of choice for testing providers who perform security tests for non-Government clients. CHECK will remain as it is, but it is CESG’s intention to remove from the CHECK list those companies who do not perform tests for Government clients.
You could argue (though unsuccessfully in my opinion) that this will create a two tier level of service and standards - it won’t and this can be seen by the fact that the CREST Infrastructure Certification Examination has been agreed by CESG to be technically equivalent to the CHECK Assault Course (and as you’ve already stated is harder to pass than the CHECK Assault Course or so I’m led to believe.)
Initiatives like CREST are to be welcomed to provide some much needed clarity to the security testing market, provide an indication of standards which member companies must meet and therefore ensure that end users of their services get quality testing which meets their needs and at a good price.
point taken.
“You’ll be surprised how cheap good pentesters are these days in the UK. It’s largely a commodity market”
OK, we need to differentiate between good pentesters and good *pentests*. If someone is charging you 2K to do a webapp test, trust me, it won’t be very complete even if the guy performing the test is a hell of a webapp hacker. The reason for this is simple: you get what you pay for. For example, if a company got a “good deal” for a pentest, the tester won’t be able to spend a great deal of time doing the assessment due to profit targets set by the company he works for.
Sorry I keep mentioning webapps, I know there is so much more than webapps in pentesting, but this is an area that I deal with a lot. It’s also an area where companies are still willing to pay more because the pentester can find many things that VA scanners like Nessus won’t (NOT a commodity market yet, unlike infrastructure vulnerability assessments).
The security testing industry is like any other. There are no “good deals”, you simply get what you pay for.
And finally, regarding good pentesters, well, there are very few good ones out there. Most “pentesters” are just people with very little experience who simply run a scanner and get paid 25K to do so, but unfortunately they don’t use their brain to discover security issues. I hope I’m not sounding too harsh, but this is the plain truth. A VA analyst is NOT a pentester! Once again, Vulnerability Assessment is NOT Penetration Testing.
My final message for pentesting customers out there is:
1. you WON’T get a good security assessment done by a good pentester for cheap. If you are not paying *at least* above the market average price, it just won’t happen!
2. Vulnerability assessment is NOT penetration testing. VA is mostly about running an automated scanner and getting rid of false positives. Penetration testing is about using automated tools, modifying or creating new tools, exploiting vulnerabilities, and applying the hacker mindset (i.e.: experiment, believe that anything is possible, be creative).
From what I can see from my experience, I very much agree with Pagvac.
Great post and good comments as well. I would say that the penetration test as we have known it will have to evolve into something more like a Tiger Team operation. The threat landscape has changed so much in the last few years where things like breaching physical security and client side attacks are now popular attack vectors…the penetration test has to evolve at some point to address these threats. A company can no longer conduct a network/web application penetration test and think that they are getting a complete picture of their current security posture. This is where a Tiger Team operation has tremendous value…testing all aspects of the security of an organization (people, process, technology).
Also, to add to pdp’s remarks on quality…always remember the “team” in Tiger Team. I have seen the same as pdp with penetration testers in the industry, it’s usually one person conducting the test. While this one penetration tester may be very good at what he or she does, you still need the input and creative thought of others to do a thorough job. With a Tiger Team you need the skill sets of multiple experts…from physical to network security and everything in between. It would be very difficult for one person to conduct a full Tiger Team operation let alone a successful one.
I think the difference between penetration testing and tiger teams should be way more fundamental then just pricing and value for money…
Penetration tests are a dumb idea in general - they are used, in plain view, to reveal weaknesses. This means that there is the assumption of weaknesses and the builders / maintainers will be on edge to prevent the penetration test from succeeding. Put this in conjunction with what pagvac says and you get programmers building against test tools iso real world threats.
I think a tiger team should be invisible or at least covert. A good tiger team is only known to the CEO or the board so that there is no prior knowledge of the attack to those who need to prevent the attack.
A contemplation for dessert?
if “Penetrate and Patch” was effective, we would have run out of security bugs in Internet Explorer by now. (source: Marcus J. Ranum)
absolutely, I’ve been involved in a couple of pentest where the client deliberately sabotages the pentest in order to look good in the report but in general you should try to avoid these types of clients. they are just looking for a quick approve from you that they are OK security-wise, and this kind of thing can really damage your reputation.
Adding onto Pagvac’s differences between Vulnerability Assessments and Penetration Testing, is the targetted goal of a pentest. That is, they are striving to locate and disclose/modify that core piece of data, to penetrate all of the multiple layers of defence from the firewalls down into the database.
Actually CESG have actually advised all CHECK Team Leaders if they take and fail CREST they will be removed as Check Team Leaders.
mort666, I am not a fan of neither of CREST nor CHECK, both certs are profit driven. No to mention that you cannot be a CHECK if your are not working for a CHECK company. That should tell you something. But again, if it suits your needs, why not. Though the post was regarding tiger team vs pentests :) but since there is interest, we may as well start a CREST/CHECK related post.