Another perspective on vendor rankings

I've written pretty extensively on vendor rankings from analysts and how and why they should be used by end users. Here is a smattering of stuff:

• MQ does matter - even if G doesn't think so (here)
• The joke of analysts' vendor rankings (here)

But something that Chris Harrington of InfoSecPodcast wrote earlier this week fills a hole in my documented position. Basically Chris' post (here) overlays all of that procurement blather that I spout above with the customer size segmentation filter. Huh? OK, I'm talking in tongues again.

Chris' point, and it's a very good one, is that a magic quadrant is built for the LARGE ENTERPRISE. That's who Gartner hangs out with and that's where they get their information (besides vendors that is). The folks that are MQ Leaders have done a good job selling high-ticket items to large companies with big budgets.

What if you are a mid-sized company, that is looking for a small-ticket item, and you have very little budget? Then you'd be like 90% of the world and the MQ would be TOTALLY IRRELEVANT to you. Chris' friend is pretty much like a lot of the folks I run across every day. The conversation goes pretty much like this:

Him: "But they aren't in the Leader Quadrant."
Me: "So what? Why do you care who sells a lot into the enterprise? That's not you."
Him: "Because I care. My CIO used to work for a big company and he believes in the MQ."
Me: "Then he's an idiot."

The conversation usually ends right about there, but the point is the same. When you are buying a security product, the vendor rankings can be a useful guidepost to define a short list, BUT ONLY if you look like the analyst's typical customer. If not, then best case you are wasting your time. Worse case, you are buying something that you don't need and likely spending way more than you need to.

While I'm on the topic, Thomas at Matasano questioned the usefulness of "joke" post (here), given most analyst work is not "statistically relevant." So let me clarify things a bit more. My point is that both Gartner and Forrester make you want to believe that they are talking to thousands of end users and developing these positions. But they aren't. And if they are just answering an inquiry - that is fine.

BUT if they are using that data to place vendors on a chart, it's a problem. The chart by the virtue of it being a chart indicates a QUANTITATIVE analysis. But the underlying information to develop the chart is in fact QUALITATIVE. That's my issue.

If they want to do a ranking of vendors based on what they hear, that is fine. But if they place them on a chart (where vendors will inevitably get out the ruler and measure the distance between the dots), then there needs to be more quantitative rigor to the analysis.

That's my story and I'm sticking to it.