Good Morning:

Just to give you a general overview of the Incites Redux process, I revisit my 2008 Incites (or projections for those of you not familiar with my lingo). I do this provide some level of accountability, which still seems to be unique in the technology research business. Folks make ridiculous projections, both on market sizing and industry dynamics with impunity. If they are wrong, so what? They still collect their checks and no one is worse for it. Except those poor saps that actually follow their advice.

So hopefully by now you've realized I'm a different kind of analyst and a different type of guy. I not only welcome the scrutiny of my positions, I search it out. So over the next two weeks, I'm going to revisit each of my 2008 Incites and give myself a "grade." Of course, this is self-analysis - but I'm confident that if you strongly disagree with something, you'll let me know. Bashful folks you are not.

Have a great day.

Incite #1: Express Your Inner Bean Counter

Substantiating the value of security continues to plague practitioners, who still can’t specifically answer the question: “Are we secure?” Structured security programs (ISO 27001/2, COBIT, Pragmatic CSO) help align programmatic activities, and look for significant advances in the area of security metrics – where the industry begins to gain consensus about what can and should be tracked.

Read the original Days of Incite post on this topic.

6-month grade: D

OK, this is not an auspicious beginning to my 2008 Incites. Some of my buddies told me I was being a bit optimistic to think that we'd see "significant advances" in the area of security metrics. And they were right. But let's not get the cart ahead of the horse. The first part of the Incite deals with security programs, and if anything the desire for the industry to get a "cookbook" of sorts that provides a set of playbooks to do security remains very high.

I've fielded a lot of inquiries and questions regarding IS 27001/2 and also COBIT. Throw a little of NIST's 800-100 and 800-53 and it remains clear that most practitioners still have no idea where to start when embarking on a security program. I expect these frameworks will accelerate over the short term. I also think that a lot of fairly pragmatic IT professionals will default to following the 12 requirements of the PCI DSS. Notice that I said pragmatic IT professionals, not Pragmatic CSOs.

Most IT professionals facing down the spectre of PCI compliance don't have much of a choice, but to move towards the 12 requirements. Truth be told, that isn't a bad place to start - but it's not a comprehensive security program. For security professionals (yes, Pragmatic CSOs), the program needs to be more holistic and more structured. Remember, security is a journey not a destination and it's not just about passing the assessment and getting the rubber stamp. It's about actually protecting information.

Which brings us to the metrics discussion. Over the past 9 months, I've gotten fairly deep in the metrics community and lent a hand to start up a consensus group to define what can and should be counted. Basically, I'm still looking for my own answer to "Are we secure?" and at this point, the only answer still is a resounding no! One of the books I'm reading during my summer hibernation is called "The Black Swan" and it's really impacting my view on what security really is and how we should be measuring ourselves.

I've got a lot more thinking to do around the topic, but I think the question "Are we secure?" is the wrong question to ask. To me, "how quickly can we recover from a fixed number of attack scenarios?" seems to be a more appropriate question, especially given that we've never been able to predict where the next major attack will come - and I suspect we never will. But we may be able to model the type of damage an attack will cause, and figure out how long it would take to recover. 

I know that the risk counters out there want and like to build big models to assess the risk and quantify it (and hopefully over time reduce the applicable risk that is being measured), but I'm still not sold on that approach.

To their credit the Risk Management Insight guys have been very patient with my constant wingeing about their approach and Jack has put together a number of thoughtful pieces about why actually quantifying your risk is a good thing. Yet, I'm still questioning how that kind of analysis yields any kind of return relative to the time and resources required to build the model.

But even that's not the point. The point is that we need to bifurcate the metrics process into (for lack of a better term), an elementary school track and a PhD track. There are some very very smart people out there that are talking at a PhD level. Their work is impressive, but it's not accessible. The common men and women out there, just trying to get out through the day, do not track (nor could they collect) the metrics the PhD's are talking about.

Yet, the PhD's tend to be very critical of the "good enough" approach of the rest of the world. The folks that count patches and AV updates and spams blocked. And this is the problem with trying to produce a SINGLE set of metrics for the entire industry. Thus, I now realize the idea of gaining true consensus onsecurity metrics is probably a pipe dream.

We should be looking for AT LEAST two different sets of metrics and data sets. The first is really tracking activity and that is for the unsophisticated practitioner that is just trying to get a handle on what they are doing operationally. We have the data and although it's not perfect, at least it's there. 

Then there is the ongoing research that the PhD's are pushing to model and quantify risk and figure out what the knobs are that really impact security outcomes. Optimally, the PhD's find some stuff that everyone else can use over time. But to think we are going to get to a consensus anytime soon is, well, optimistic. And in this business, hope and optimism get a D.  

Photo credit: "censored" originally uploaded by tifotter