DevOps Zone is brought to you in partnership with:

Troy Hunt is a Software Architect and Microsoft MVP for Developer Security. He blogs regularly about security principles in software development at and is the author of the OWASP Top 10 for .NET developers series and free eBook of the same name. Troy is also the creator of the recently released Automated Security Analyser for ASP.NET Websites at Troy is a DZone MVB and is not an employee of DZone and has posted 68 posts at DZone. You can read more from them at their website. View Full User Profile

Everything You Need to Know About the Heartbleed SSL Bug

  • submit to reddit

Massive. Huge. Catastrophic. These are all headlines I’ve seen today that basically say we’re now well and truly screwed when it comes to security on the internet. Specifically though, it’s this:

The Heartbleed bug allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions of the OpenSSL software.

Every now and then in the world of security, something rather serious and broad-reaching happens and we all run around like headless chicken wondering what on earth it means. Did the NSA finally “get us”? Is SSL dead? Is the sky falling? Well it’s bad, but not for everyone and quite possibly not as bad as many are saying it is. Most of the early news has come via (a site created by Codenomicon) which, if I may say so, has done an excellent job of standing up a very nice little website and branding the bug:

Heartbleed logo

But it’s actually a lot more complex than the shiny logo suggests. It doesn’t help that it’s not an easy concept to grasp and that alone compounds the confusion and speculation about what the bug really is, what the bug is not and perhaps most importantly, what you need to do about it. I’m going to try and distil the issue into a set of common questions people are asking – Heartbleed in a nutshell, if you like.

What’s OpenSSL and what versions are affected?

Let’s start here: the Heartbleed bug is only present in the OpenSSL implementation of SSL and TLS (note that even whilst not always the same thing, these acronyms tend to be used interchangeably and usually refer to their implementation over HTTP or in other words, HTTPS). As the name suggestions, this is an open source software product that facilitates communication over the SSL protocol and it is extremely common. At the time of disclosure, about 17% of the world’s “secure” websites were said to be vulnerable to the bug.

Typically, OpenSSL implementations are present on servers running Apache and nginx. Unfortunately, Apache remains the dominant web server today with more than half of the internet’s active sites running on it and nginx has about another 14%:

Netcraft server stats

The Heartbleed bug itself was introduced in December 2011, in fact it appears to have been committed about an hour before New Year’s Eve (read into that what you will). The bug affects OpenSSL version 1.0.1 which was released in March 2012 through to 1.0.1f which hit on Jan 6 of this year. The unfortunate thing about this timing is that you’re only vulnerable if you’ve been doing “the right thing” and keeping your versions up to date! Then again, for those that believe you need to give new releases a little while to get the bugs out before adopting them, would they really have expected it to take more than two years? Probably not.

Is it only sites on Apache and nginx that are affected?

Not all web servers are dependent on OpenSSL. IIS, for example, uses Microsoft’s SChannel implementation which is not at risk of this bug. Does that mean that sites on IIS are not vulnerable to Heartbleed? For the most part, yes, but don’t get too cocky because OpenSSL may still be present within the server farm. A case in point: Tim Post from Stack Overflow tweeted this earlier today:

#stackoverflow, heartbeat, we know.

But aren’t they running all ASP.NET MVC on IIS? Yep, they sure are, it’s made very clear in Nick Craver’s excellent post last year on the road to SSL, in fact you can see the (IIS) web servers all sitting over there to the right in this image:

Stackoverflow server farm topology

Wait – is that nginx in there too??? Yeah, about that:

HTTPS traffic goes to nginx on the load balancer machines and terminates there.

You see the problem? Yes, they may be using IIS but no, it doesn’t mean that OpenSSL doesn’t feature in their server farm architecture and indeed it’s sitting out there at the front of everything else terminating the SSL. The same issue exists if a machine acting as a reverse proxy is sitting in front of IIS and running Apache or nginx.

Edit: Nick and Ben from Stack Overflow have clarified that they now use HAProxy for SSL termination which uses… OpenSSL.

Here’s another one, in fact I received this myself last night from AppHarbor regarding ASP.NET hosting I have with them:

AppHarbor notice about Heartbleed

The point is that often server infrastructure is much more than just the web server alone and not consciously running on Apache or nginx doesn’t mean it doesn’t feature in your environment.

Who found it and why make it public?

The first public evidence of the bug appeared as an OpenSSL advisory on April 7 and warns of “A missing bounds check in the handling of the TLS heartbeat extension”. The bug was discovered and reported by Neel Mehta of Google Security and simply states the impacted versions and recommends upgrading OpenSSL or if that’s not feasible, recompiling it and disabling the heartbeats.

As for why it was made public, this is the age old debate of whether public disclosure fast-tracks remediation or opens vulnerable systems up to attack. Probably a bit of both, the problem with a risk like this as opposed to a single discrete vulnerability on a website is that in order to fix it you need mass uptake of the revised version and until the risk gets socialised, that’s not going to happen. You then have this problem where it only takes one party with malicious intent to start running rampant against sites that are entirely ignorant of the risk and you’ve got a really serious problem.

Regardless, you can’t argue with the focus it’s getting now or the speed with which many are now remediating vulnerable sites. Of course there’ll be others that lag behind and they’ll be the ones at serious risk, let’s take a look at just what that could mean for them.

What’s an SSL heartbeat?

Let’s first understand the feature that this bug has made its way into. The bug we’re talking about here exists in OpenSSL’s implementation of the heartbeat extension, a feature described by the IETF as follows:

The Heartbeat Extension provides a new protocol for TLS/DTLS allowing the usage of keep-alive functionality without performing a renegotiation and a basis for path MTU (PMTU) discovery for DTLS.

In other words, it’s a heartbeat in the same vein as we commonly know it in other aspects of computer systems, namely it’s a check to see if the other party is still present or if they’ve dropped off. When they’re still there, the heartbeat keeps the context between the peers alive hence the “keep-alive” nomenclature. In the context of SSL, the initial negotiation between the client and the server has a communication overhead that the heartbeat helps avoid repeating by establishing if the peer is still “alive”. Without the heartbeat, the only way to do this is by renegotiation which in relative terms, is costly.

How is the risk exploited? And how was it fixed?

There’s plenty of proof of concept code around already, I particularly like Rahul Sasi’s example in his Heartbleed Attack POC and Mass Scanner as he clearly explains the vulnerable code, the fix and what he’s written to test the bug. In short, the original risk in OpenSSL all boils down to this line of code:

buffer = OPENSSL_malloc(1 + 2 + payload + padding);

As Rahul explains:

In the above code memory is allocated from the payload + padding with a user controlled value. There was no length check for this particular allocation and an attacker could force the Openssl server to read arbitrary memory locations.

In other words, an attacker can control the heartbeat size and structure it to be larger than expected, fire it off to the target server using TCP on port 443 and receive a response that contains up to 64kb data in a memory allocation outside the bounds of what the heartbeat should be able to access. Do it again with a different heartbeat size, get another 64kb response from another memory space. Lather, rinse, repeat. Easy peasy.

You can go and inspect the code and the consequent fix in this commit on GitHub if you want to get down into the nitty gritty but in short, the payload is now “bound checked” (it can’t exceed 16 bytes) and the entire heartbeat is now validated to ensure it does not exceed the maximum allowable length. Ultimately this boiled down to a very simple bug in a very small piece of code that required a very small fix. Now it just needs to be installed on half a million vulnerable websites.

What’s the risk? What goes wrong if it’s exploited?

Let’s being with the techie description courtesy of Wikipedia:

missing bounds check in the handling of the TLS heartbeat extension can be used to reveal up to 64k of memory to a connected client or server

Ok, but what does that actually mean? It means an attacker can do stuff like this:

Do not login to Yahoo! The OpenSSL bug #heartbleed allows extraction of usernames and plain passwords!

In case the gravity of this is not clear, what’s being asserted here is that Mark was able to read the credentials of another user directly from memory in Yahoo’s service and he’s done all that remotely just be issuing an SSL heartbeat. Whoa.

More specifically, we’re looking at what was an encrypted HTTPS request to Yahoo’s website (inevitably a POST request to the login page), and we’re seeing a component of that request that was still resident in memory when Mark exercised the exploit and pulled the data. So what else can an attacker gain access to via this bug? Credentials are one thing, but of course this is just by virtue of them occupying memory space which is inevitably but one piece of data in there. Another is the session ID or auth token; even if you can’t pull a user’s credentials via the bug (it’d have to still be in memory at the time), you can still pull the ID that persists their session across requests and consequently enables the attacker to hijack that session and impersonate the user.

The other really serious piece of data that could be pulled is the private key of the certificate. I’m going to repeat precisely what summarises with regards to leaked primary key material because they summarise it so eloquently:

These are the crown jewels, the encryption keys themselves. Leaked secret keys allows the attacker to decrypt any past and future traffic to the protected services and to impersonate the service at will. Any protection given by the encryption and the signatures in the X.509 certificates can be bypassed. Recovery from this leak requires patching the vulnerability, revocation of the compromised keys and reissuing and redistributing new keys. Even doing all this will still leave any traffic intercepted by the attacker in the past still vulnerable to decryption. All this has to be done by the owners of the services.

Don’t miss the severity of that piece around past interception; the exposure of the private key presents the opportunity to decrypt which data was previously sent over the wire before this vulnerability was even known. If someone was to have been storing these pcaps (*cough* NSA *cough*) and they’ve then been able to pull the keys by exploiting this bug, all your previous data are belong to them.

It can be that bad. In theory – but then there’s this:

Heap allocation patterns make private key exposure unlikely for #heartbleed #dontpanic.

Remember Neel? Yeah, he’s the guy that found the bug to begin with. Is he right and everyone else is wrong? Maybe. Maybe not:

@bengrubb We're very certain. #heartbleed

What about Codenomicon – remember them too? They’re the guys who created the website I keep referring to which has become such a canonical reference for the bug.

Published at DZone with permission of Troy Hunt, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)


Michael Litwak replied on Wed, 2014/04/16 - 11:55am

Under the heading I’m a sys admin – what do I do?, the first step is updating OpenSSL to 1.0.1g, but the second step is missing: identify all the server applications which have their own copy of the dynamic OpenSSL libraries within the application's  folder structure, or apps which statically link to OpenSSL.  For all such applications, if they link to a vulnerable version of OpenSSL, then the dynamic OpenSSL libraries or the application itself needs to be updated.

Example server applications or libraries which fall into this category include FileZilla Server, hMailServer, PHP 5.5.8 for Windows, Tomcat Native (tcnative-1.dll, which is found in Atlassian JIRA 6.0 for Windows and perhaps many other server applications), older versions of CollabNet Subversion EDGE for Windows, and many more.  There is no definitive list of such applications, but a few lists of vendors who have reported Heartbleed vulnerabilities and updates are available, including lists from SANS, and the Finnish National Cyber Security Centre.  New announcements are still coming daily, making it tough for sys admins to know that they have identified all the Heartbleed vulnerabilities on their servers.  I would not be surprised if many companies wind up repeating their key and SSL certificate redeployments because of this.

Apparently, if your company uses the same wildcard SSL certificate on numerous servers, and if any one of those servers is vulnerable to Heartbleed, then the keys and SSL certificate need to be reissued and deployed to all these servers (even ones with no direct Heartbleed vulnerability).  Revoking the old keys and cert, and notifying users of the need to change password follow from there.

Other vulnerabilities not mentioned specifically in the article include SSL VPN hardware running vulnerable firmware, such as FortiGate UTM devices running older 5.0 firmware.

Systems which are safe from Heartbleed because they run older versions of OpenSSL (or have applications which link to older OpenSSL libraries) may be vulnerable to other well known SSL/TLS exploits, such as the BEAST attack.  Updating such systems and reconfiguring/reordering their ciphers may be necessary to improve security.  Disabling the SSL 2.0 protocol, and perhaps TLS 1.0, may also be called for.

The online Heartbleed scanners, such as the one from Qualys SSL LABS (which also checks for BEAST vulnerability), only check on Port 443 (i.e. https).  If your web servers are running https on an alternate port, such as 8443, you need another way to scan them for vulnerabilities.  Ditto for FTPS (FTP via SSL/TLS), SSL VPN's, SMTP email services over SSL/TLS, etc.  I haven't tried this script  yet, but it claims you can scan your servers across  numerous ports/services for Heartbleed vulnerabilities.

In the future, incidents such as the Heartbleed bug should be tracked in a coordinated way, so system administrators can reference a definitive list of vulnerable products and take appropriate action.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.