Can You Have a Review if No One Shows Up?

I spoke about maturity in my recent Network World article (here), and I stumbled across another HUGE example of what happens when a market matures. Having spent a while in the anti-spam business, I still follow the space closely. I saw that SC Magazine published a group test on email security services. So I clicked on the link to see what they had to say. It was kind of funny to say the least.

This quote says it all:

We were disappointed by the poor turnout for testing. Managed service companies have always been reluctant to allow testing. Taken to extremes, as in the case of the now-bankrupt Avecho, we received a complete stonewall response to queries about the service risks alienating the market. In this test, over a dozen vendors were approached, and while several indicated interest, only four finally provisioned services.

Non-participants included high-profile players like Postini, Frontbridge (now owned by Microsoft) and MessageLabs.

So basically out of the 15-20 vendors in this space FOUR showed up. And these four are pretty low profile (Black Spider, Mimecast, MIMEsweeper, Softscan).

Why wouldn't the leaders show up? Are they scared? Of course not. THEY HAVE NOTHING TO GAIN.

Remember the role of product reviews in the procurement cycle. I described that here. If I can talk to real reference customers, why do I care about a product review? You don't. Each of the vendors passing on the review have thousands of customers. They've got plenty of references.

The other key issue is THEY CANNOT AFFORD TO LOSE. The last thing you want to do is show up and lose, especially to someone that you have 1000x the number of customers. That would be a bad day.

Do you wonder why Cisco doesn't show up for IDS or firewall reviews? They don't have to. People will look at their stuff because they are Cisco.

If a market is mature, product reviews are useless because the true leaders will not show up. So you can find out who is the strongest of the weak. But you won't learn a thing about who you should pick.