Anti-spam

Given innovation by spammers and fraudsters, keeping content filtering algorithms accurate and timely is proving very difficult for content-focused security vendors. In 2006, heuristics-based detection cocktails fall out of favor, pushing the pendulum back towards signatures that favor entrenched AV vendors. Users increasingly embrace in the cloud content filtering for e-mail, IM, and web traffic because it allows them to get rid of another box in the perimeter and stop worrying about exponentially increasing message volumes.

Grade: B+

Original Days of Incite post: here
Incite Redux post: here

Spam made a comeback in 2006 in a big way. Image-based spam and other nefarious techniques kept most of the anti-spam vendors on their heels all year and also created a lot of swap-outs and turmoil on the email security gateway.

The problem with the Incite was that spam signatures didn’t fare much better than anything else in detecting the new wave of spam. The business of catching spam is a thankless situation and much like the AV battles of a few years ago, tend to leave a few very large players and a lot of carnage. Anti-spam (along with other content security functions) is also increasingly being bundled into the UTM platform.

This is another case in point as to why I won’t be making any more product architecture projections in the future. Customers don’t care whether it’s heuristics, signatures, or black magic. They want the spam to stop and that didn’t happen well enough in 2006. The email security vendors have a lot of work to do.

The part of the Incite that really resonated was the drive towards services. Whether it’s the big couple of services players (Postini, MessageLabs), consolidated challengers (Microsoft/FrontBridge, SurfControl/Black Spider) or Tom, Dick and Harry installing a few Barracuda boxes in their garage and calling themselves an email security service – there are plenty of options for customers.

The large enterprise will still use their dedicated email security appliances, but the mid-market will continue to flock to the services as we move forward.

Following this M&A stuff is becoming a full time job. This morning, SurfControl acquired Black Spider (link here), which is a content security service provider that does email and web filtering. Postini, MessageLabs, and ScanSafe are their top competitors. I had decided not to do a separate piece, but then I heard from a friend of mine who is over in the UK and he mentioned how everyone over there was fired up about it. Sometimes I need to remember that we all live in a global village.

The specifics of the deal are pretty straight forward. SurfControl pays US$36 million and gains entrance into the managed service business. Black Spider gets an exit, and worldwide distribution given their real strength was in EMEA. As an added bonus, Surf Control can probably sell the blackspider.com domain name to Columbia Pictures for a pretty penny (if you've seen the Spider-Man3 trailer, you know what I'm talking about).

To be clear, Black Spider was small with about 1200 customers and doing less than $5 million in revenues, but that doesn't matter. They'll fit into SurfControl like a glove. If a customer wants a service option, SurfControl doesn't have to walk away from the deal (or the customer). It's a pretty compelling way to play into the inevitable trend that most customers will want to filter email and web traffic in the network (see Incite on Content Security here).

And as opposed to other deals announced this week, $36 million is very affordable for SurfControl.

Yet, there are always challenges every deal, and with this one comes the challenge of channel conflict. It needs to be very clear to the SurfControl field force when they should look to sell a service or an appliance. The worst case scenario is that they try to sell an appliance, and only when the client says a resolute NO do they move towards the service. By then, Postini or MessageLabs is in the house and will win the business.

You also will have a potential area of conflict around their VARs trying to get into the MSS business themselves. When I was in the business, I saw a lot of that and it's only been increasing. You know, a VAR buys 3 Barracuda's and bingo, they are in the email security business.

But for the most part, this deal makes perfect sense and is a precursor to maybe some bigger folks that offer appliances moving to take out the leading service providers. McAfee already sells Postini's stuff and IBM is very close to MessageLabs. So it wouldn't surprise me to see more deals in the space sooner rather than later.

I get a few comments and trackbacks to my blog postings every week and realize that many of you either get my stuff through RSS or e-mail and tend not to ever see those thoughts - that do add a bunch to the conversation. So every so often I'll go through and point out interesting discussions.

Not unexpectedly the Blue Security post garnered the most comment activity, with a chap called Spy der Man discussing some open source efforts (dubbed Black Frog) to distribute the Blue Security techniques. I made the analogy between Blue and Black Frog to Napster and Kazaa. Martin McKeay covers the issues with Black Frog in this post (http://www.computerworld.com/blogs/node/2559).

I also heard from Martin and Michael Masnick of TechDirt to clarify their sentiments about Blue Security's demise. They are chagrined about the spammer's victory. I'm much more sanguine and not surprised by the entire episode. Live by the sword, die by the sword - as they say.

In any case, thanks to all that contribute comments to the blog. It's that kind of discourse that makes this a lot of fun. 

Sometimes I can't understand human nature. Two weeks ago, people (especially TypePad bloggers) were cursing Blue Security. Those morons! How could they have been so stupid to take down all of Six Apart? My blog was down for 9 hours, damn them to hell!

Well, now they are gone and people are sad. It kind of feels like a funeral for a real scumbag. High profile bloggers like Techdirt, Dan Gillmor, even my pal Martin McKeay gave similar eulogies - "Well they sucked, but we're sad to see them go." Give me a break. This sentiment just doesn't resonate with me. Though if they have free beer at the wake, I'm in!

To clarify my point, let's draw an analogy to one of my favorite topics - drug dealers. Does anyone care if one drug dealer kills another? Maybe the guy's family, but other than that - probably not. Blue Security's model was to try to take down the drug dealers by giving them more drugs. Didn't think of it that way did you?

Two wrongs don't make a right. My Mom taught me that one pretty early on. Blue Security was spamming the spammers and even if they were successful in taking down a few, 10 others would pop up in their wake.  Just like drug dealers. Take down one and 10 others fight over the vacated street corner.

So what's the answer to stopping spam? The sad truth is that to have ANY impact on the amount of spam out there we need to address the root cause of the issue - the economic incentive to spam. As long as people buy stuff from those unsolicited messages, spam will happen. IT'S AS SIMPLE AS THAT. Vigilantes (like the late Blue Security) are not going to repeal the laws of economics.

Getting back to my drug analogy, Nancy Reagan was right. Just say no! But as long as some folks say yes, we are fighting a losing battle. Blue Security's model was wrong from the start, so them going away was their inevitable end game. Don't kid yourself. Sure the spammers accelerated their demise, but at least it gave them an opportunity to go out in a blaze of glory.

Since I've left the anti-spam business about 9 months ago, it's nice to see that it's still a brutally competitive market where everyone sounds the same. It does seem that innovation on email hygiene (meaning inbound mail) has slowed. I can't recall anyone really introducing a new, important capability over the past 6 months. It's been a lot of point releases aimed at either adding better enterprise management capabilities, scalability or broadening the product scope beyond just inbound email. So now things like IM and compliance/encryption are becoming the battleground - the differentiation so to speak.

At the tail end of my anti-spam tenure, reputation services were all the rage. The concept is that if you know a lot about the sending IP address, you can tell whether they are very likely to be sending spam or good mail. IronPort was the reputation innovator with SenderBase and CipherTrust came later with TrustedSource. Standard disclaimer: I used to work for CipherTrust and am a shareholder (because I can't sell the stock).

Folks like Symantec and Postini always said they had reputation services under the covers, but never really made them visible enough to prove it. Recently (like within the last two weeks), BorderWare (link here) and Habeas (link here) have introduced their own reputation services. Either broader, BorderWare's tracks IP and VoIP data, or larger, Habeas claims 60 million IP addresses in their database - which may or may not be true. I'm sure they have 60 million things in a database. What those things are is subject to interpretation. You have to love marketing.

But if you are a customer looking at these solutions, does it matter? The vendors will try to paint their reputation stuff as broader, more accurate, bigger and will let you drop more bad messages at the gateway. Who do you believe? I say believe none of them. Reputation is now a standard part of the game and its certainly under the covers. You don't buy an anti-spam product because of a reputation service. You buy it because it stops your bad mail.

Content security is a different animal. That is hard for many to believe that have grown up in the network security space, where an attack is an attack is an attack. Maybe 50% of spam is ridiculous. Dealing with nasty inappropriate stuff or prescription drugs, all the products catch that stuff - or they don't get to play.

It's the borderline stuff that is very difficult to categorize. One man's spam is another man's gold. A lot of spam is subjective, so it's very hard to say in absolute terms whether a message is really spam. That's why end user quarantine is so important, then the users at least get to see if there are false positives in the mix. Then you've got the language issue. Non-English spam provides a lot of variability in results. You can't just drop a US anti-spam product into the Far East. It's not a firewall.

But getting back to reputation, your definition of spam may be different and your traffic is going to be different. So you'll need to figure things out for yourself. In the content security space, the eval is everything. You need to test these products out. Maybe the specific vendor's reputation database works great for you. But it may not. And the only way you'll find out is by running the products against actual mail. That's right, run the email gateways against a subset of your live mail flow.

Theoretically, reputation should still be a differentiator. But folks like Proofpoint and MailFrontier/SonicWall continue to stop spam without it. So maybe it doesn't matter. Unfortunately I can't answer the question for you. You'll need to be the judge.

I did a quick interview this morning with Eric Parizo on the Security Wire weekly podcast regarding the Blue Security DDoS attack and who was at fault. It was a pretty far ranging conversation and we also discussed how customers should deal with spam and whether fighting fire with fire is ever the right answer.

Here's their description and the link:

This week News Editor Eric Parizo interviews Security Incite's Mike Rothman about who's to blame for the controversial DOS attack against Blue Security and Six Apart, plus all the top news and the worm that just won't die. Listen on your computer or download to your favorite mobile device.

http://media.techtarget.com/audioCast/SECURITY/SecurityWireWeekly_mp3_5-10-06.mp3

I spoke about maturity in my recent Network World article (here), and I stumbled across another HUGE example of what happens when a market matures. Having spent a while in the anti-spam business, I still follow the space closely. I saw that SC Magazine published a group test on email security services. So I clicked on the link to see what they had to say. It was kind of funny to say the least.

This quote says it all:

We were disappointed by the poor turnout for testing. Managed service companies have always been reluctant to allow testing. Taken to extremes, as in the case of the now-bankrupt Avecho, we received a complete stonewall response to queries about the service risks alienating the market. In this test, over a dozen vendors were approached, and while several indicated interest, only four finally provisioned services.

Non-participants included high-profile players like Postini, Frontbridge (now owned by Microsoft) and MessageLabs.

So basically out of the 15-20 vendors in this space FOUR showed up. And these four are pretty low profile (Black Spider, Mimecast, MIMEsweeper, Softscan).

Why wouldn't the leaders show up? Are they scared? Of course not. THEY HAVE NOTHING TO GAIN.

Remember the role of product reviews in the procurement cycle. I described that here. If I can talk to real reference customers, why do I care about a product review? You don't. Each of the vendors passing on the review have thousands of customers. They've got plenty of references.

The other key issue is THEY CANNOT AFFORD TO LOSE. The last thing you want to do is show up and lose, especially to someone that you have 1000x the number of customers. That would be a bad day.

Do you wonder why Cisco doesn't show up for IDS or firewall reviews? They don't have to. People will look at their stuff because they are Cisco.

If a market is mature, product reviews are useless because the true leaders will not show up. So you can find out who is the strongest of the weak. But you won't learn a thing about who you should pick.

 

 

I know that many of you read Security Incite Rants via your RSS reader or in e-mail, and that means you miss some of the comments. I'll point you to an interesting exchange about yesterday's spam testing post [read it here]. You actually need to hit the link on securityincite.com and scroll down on the page to read the comments.

I appreciate that Gordon Carmack of the University of Waterloo took the time to present a passionate and well thought out argument basically poking holes in many of my contentions about spam testing. Of course, I think he's wrong, and said so in my responses.

I also think he doesn't really understands the role of the analyst. Or maybe he does and just doesn't buy it. I've also laid out my ideas on what analysts do and why in the responses. Having done this for a long time, these objections and questioning of my opinion based on the fact that I don't actually test products is nothing new.

At the end of the day, some folks think analysts add value to what they are doing and others don't. Personally, I don't much care which camp anyone falls into. I have no trouble sleeping at night because I know that I help folks make better decisions. And ultimately that is my acid test.

 

Saw a post yesterday on the Ferris Research blog [read it here] about a publicly available spam corpus called TREC 2005 (for more information about TREC read here). Ferris’ position on the value of TREC 2005 is:

The corpus is primarily intended for academic research and development of anti-spam filters and has significant restrictions on its use. This collection is important as it provides a standardized collection to test and compare spam filters in both academic and commercial contexts.

They are wrong. Using any corpus older than a month and obscuring the mail headers is actually detrimental to testing and comparing spam filters. Why? Because spam is a real time phenomenon and using “stale-mail” to test it is a waste of time. Your results will smell worse than 30 day old Wonderbread.

To be clear, a bulk of spam is no longer sent by that shady character using a spam cannon in his garage to blast out 200 million messages a day. Spam is sent by a worldwide network of zombies that have made it much harder to track and stop the onslaught.

A key technical innovation in defending against these zombies was the reputation system. IronPort’s Senderbase and CipherTrust’s TrustedSource are the two highest profile reputation systems out there. Basically, by tracking the types of messages coming from a specific IP (and using some fancy mathematics), you can get a pretty good feel for whether they are a legitimate sender or not.

Combining reputation with heuristics and signatures creates a cocktail of techniques that can be used to more accurately detect spam. Now anyone that says they can consistently always stop 99% of spam is lying to you. Spamming techniques change fast enough that effectiveness will ebb and flow as the spammers and anti-spammers engage in constant point-counterpoint. But in general, most of the solutions out there do a good enough job.

Now back to TREC 2005. I am a big fan of bake-offs (technical evaluations) during the procurement process (see Buying Security products post). Having users compare spam catch rates using stale-mail is a disservice because real time reputation checks cannot happen on stale mail. Who the message is coming from is a critical part of today’s detection techniques. So, using a pre-baked corpus eliminates that set of tests and will make your results suspect at best.

It is also a very bad idea to just forward the test corpus through a bit blaster. This puts your email security gateway as the second hop in and obscures the true sender’s mail header. This dramatically impacts your ability to accurately detect the spam. I can get into more technical nuances off-line, but take my word for it. Your results will be crap. In fact, a number of well-known publications used this technique in early anti-spam reviews and their results weren’t worth the paper they were printed on. But it took them 18 months (and a lot of my personal blood and sweat) to get them to see the fault in this testing methodology.

So how do you test anti-spam products? Basically you need to use them in real mail flow. I believe that you set up a set of test users (that are a bit more understanding than your CEO) and run their ACTUAL mail through the box for a month. Then you can gauge real time effectiveness and select the best fit for your organization. 

UPDATE: Let me clarify a bit that a corpus like this will be useful to an anti-spam research, who presumably understands how to tune their heuristics and/or signatures. My point is that this kind of corpus will NOT be useful to end users trying to compare anti-spam products.

 

I'll apologize to any of you that have seen some pretty nasty trackbacks posted to my blog over the past day. It seems the trackback spammers have found my site and I'm getting blasted.

Having worked at an anti-spam company and using Yahoo mail for many years, I've sort of forgotten what a nuisance spam can be. I really wonder how many of my web site visitors will be interested in "beach voyeur" or "mature galleries" (those were the ones I could actually mention). I suspect very few, but it's a numbers game for those jokers.

I'm currently working with my service provider and should have a fix in the very near term.

And for you anti-spam vendors out there, this is opportunity knocking. The tools used to stop this trackback and comment spam are analogous to SpamAssassin. Open source and very raw. There is definitely room for a commercial version of a sophisticated tool. Now, I'm not going to pay $100k for it, but if it was bundled into my hosting service for a couple of bucks a month I'd be interested. And it's the same technology used to detect email and IM spam.