EAC Blog: The joke of analysts' vendor rankings

The folks at TechTarget were kind enough to let me republish my posts
at the Expert Answer Center here. This post first appeared on July 12.
Link here.

As I discussed yesterday, I've spend a lot of time helping end users buy security products more effectively. Inevitably the customers want a recommendation on what product they should buy. Most are chagrined when I tell them that I won't do that. I can certainly provide some perspective on who are the market leaders, based on lots of criteria. But I won't recommend a product for them. That's their job, because it's their ass that's on the line if the decision is wrong. You can't outsource the decision, not if you like your job anyway.

But not all of the analysts out there think this way. The business of ranking vendors is a big one. Whether you want to call it a Magic Quadrant, a Wave, market sizing, or anything else -- these are all fundamentally the same. They strive to generate a generic answer to what is the best product, based upon some arbitrary criteria.

Here's the problem. Your environment isn't generic, is it? If you look exactly like companies of similar size in similar businesses, then what is your differentiation? Why are you different? There are definitely similarities between businesses based on size and industry, but each organization has their own strategic imperatives, culture, threshold for pain and budgets. You cannot generically decide what products will potentially be best.

And even worse, as pointed out by looking at James Governor's post on MQ sample sizes and this Forrester post, you see that these opinions are based on a very small sample sizes. Help me understand how Forrester decides market leadership based upon talking to seven vendors and 10 users? Gartner at least "talks" to a hundred or so users. But this is not statistically significant stuff, let's be clear about that.

Sure Gartner and the others field lots of inbound calls, but rankings are based on specific quantitative criteria based upon qualitative conversations. So it all gets back to opinion. They are making it up. Which is OK, but only if you trust the analyst.

As for me, I don't trust anyone. Sure, it's a personality quirk, but if my career was on the line, I'd be real careful about using these tools as key arbiters of the decision.

I recommend you use an MQ or a Wave or any other chart as a guidepost. Do your own research and potentially validate your findings relative to the analyst chart. But don't allow the chart to dictate your short list. There are a lot of vendors that don't even get on the >chart or are poorly ranked because they don't (or can't) play the game. That doesn't mean those vendors wouldn't be the best fit for what you< need to do.

But what annoys me the most is that these analysts agree with me on how and when to use a chart. They don't want the responsibility. I ranted about that on my personal blog. The problem is that customers don't get it and until they do, these
vendor rankings will be much more important then they should be.