Train wreck at Montparnasse 1895

From a software engineer’s point of view, Heartbleed is a complete train wreck.

This week, a major security bug called Heartbleed was discovered. Technology sites for programmers, system administrators, and security experts have been abuzz about it all week, but I haven’t seen much coverage of it in the mainstream press. I was able to find this article at CNN.com.

In a nutshell, Heartbleed affects a large fraction of the Web sites on the Internet. It enables an attacker to observe any supposedly-encrypted information that goes into or out of an affected Web site. You’ll notice that is pretty much the opposite of what encryption is supposed to do. Any data you send to an affected Web site — including your password, your financial information, Social Security number, anything — could have been seen and recorded by the Russian mob or those Nigerian scammers: anyone. There is no way to know who has been eavesdropping for how long, or what they’ve overheard. The bug existed, undetected, for over two years before it was discovered this week.

This is a big deal. It is potentially much more serious than the Target security breach that made front-page headlines last winter. In the case of Target, we know what data was disclosed (credit card information) and we even know the affected account holders, so we can start to repair the damage and move on. With Heartbleed, we don’t know what data was disclosed, so we don’t even know what the damage was, yet. I expect the aftershocks of this crisis will be felt for a long time to come. I suspect the reason mainstream media aren’t running with the story yet is they lack the imagination to immediately see its importance.

Security expert Bruce Schneier said of Heartbleed, “‘”Catastrophic’ is the right word. On the scale of 1 to 10, this is an 11.”

What You Need to Do

On affected Web sites, your secure Internet connection wasn’t secure: any data moving between your computer and that Web site could have been seen by an attacker. Since the Heartbleed bug is now known to approximately every black hat in the world, there’s a good chance someone eavesdropped on any site that had the bug, but didn’t fix it fast enough.

Account passwords are a high-value target and it is likely that many username/password pairs have been intercepted. The only way to rule out yours being among them is if you know how the Web server was configured. Most user’s don’t know that and most companies don’t tell.

This means you need to change all your passwords — but there is no point in doing that till after companies have fixed the Heartbleed bug. A fix is already available so it is just a matter of time until all the company’s servers are upgraded with the fix. According to this article from CNN/Money (posted yesterday, April 11), Yahoo, Amazon, Google, and OKCupid have finished rolling out their fixes and are now secure. You can change your passwords there immediately. For other sites, it’s hard to say: banks and financial companies are probably fixed by now, and the rest will have time to catch up over the weekend. So you should stay off secure Web sites until about Monday, and then change your passwords.

Since you have to change your passwords anyway, now is a good opportunity to improve your password practices.

How Heartbleed Works

This cartoon explains in pretty simple terms how the Hearbleed bug works.

More technically, Web sites use a cryptography protocol called SSL to secure data that travels between your browser and the Web server. (“Protocol” is just jargon for “a precisely defined process for doing something.”) There are several different pieces of software that all enable a Web server to use SSL. Which one the Web maintainer uses generally depends on what company made the computer that actually runs the Web site: SSL software is so widely used it comes pre-installed on servers. One particular piece of SSL software is called OpenSSL. It’s popular and widely used because it’s free. It is also, we found out last week, affected by the Heartbleed bug.

The security researchers who discovered the bug created a Web site that explains it in detail. That site is mostly aimed at computing professionals but it should be mostly comprehensible to a knowledgeable user.

If you are a computing professional (or just want to read some technobabble), check out CVE-2014-0160, the technical bulletin on this bug.

How Could This Happen?

Some bloggers and forum posters have gone off half-cocked about conspiracy theories how the NSA inserted this bug deliberately. I am deeply skeptical about that idea because the NSA’s purposes are better served by a back door only they can exploit. NSA people seem to believe they are doing the right thing with their unconstitutional dragnet surveillance programs. They wouldn’t intentionally hand foreign intelligence agencies a giant windfall like Heartbleed (though we can be sure many countries have done a lot of corporate espionage since the bug was disclosed). Besides, they can get people’s data with a National Security Letter anyway. They don’t need to resort to eavesdropping.

The developer who introduced the bug says it was a simple mistake. I haven’t looked at the source code, but from a description of the error, it’s the sort of mistake every programmer makes on a fairly regular basis.

The trick is first to train one’s self to avoid that kind of mistake, but occasionally one gets rushed, or tired, or distracted, and makes the mistake anyway. The next line of defense is testing, but testing generally can only detect problems that the test designers anticipated. I would not necessarily expect OpenSSL to let just anyone read data from memory so I can see how there was no test for it until Heartbleed was discovered. Finally, and least reliably, there’s peer review. Another developer looked at the code change that introduced Heartbleed, but apparently he or she didn’t spot the problem — which was a proverbial needle in a haystack — and it sailed right past.

Some of my colleagues at work have looked at the OpenSSL source code. One of them described it as a “hot mess.” From what I’ve read and heard, it’s complicated, convoluted, disorganized, and roundabout. It worked, kind of, but is brutally hard to understand and modify. In fact, it’s so complicated that even a leading computerized analysis tools couldn’t spot the Heartbleed bug. Code like that is usually the result of years of just-in-time maintenance without taking the time or money to do a serious overhaul at any point. I don’t necessarily fault the OpenSSL developers for that, but I do question the wisdom of so many Web sites relying so heavily on a software project that appears to be under-funded. Ideally, major tech companies like Google, Red Hat, Facebook, etc. will realize it’s in their best interest to have security software that works, and will contribute some of their developers’ time to clean up OpenSSL — or develop a replacement. OpenSSL is free software so any individual or company can work on improving it.

What Happens Next?

This week, Web site operators were scrambling to upgrade their servers. Frankly, if I ran a secure Web site I would take it offline and leave it offline until Heartbleed was fixed. It’s that serious. Even companies that were unwilling to do that will have made this fix their #1 priority.

I expect a second wave of attacks over the coming weeks and months based on information that was gathered during the window of vulnerability between when the bug was announced and when it was fixed. Attackers will probably use stolen passwords and other sensitive data (specifically, certificates and private keys) to break into major corporations’ computers. It is difficult to predict what form those attacks will take, and I am not enough of an expert to be interested in prognosticating anyway.

Most of the response falls under the responsibility of system administrators. Heartbleed is unusual in that the data it allowed to leak has value beyond the life of the bug itself. Therefore, ordinary users are more affected than usual and need to take more action than usual.

I’m updating my passwords over the next few days. It’s also a good idea to keep careful track of your credit-card statements, in case your card number happened to get intercepted. Unlike past data breaches, Heartbleed would let an attacker get your credit card number and the security code at the same time.