Cloudbleed: How Cloudflare’s Memory Leak Exposed their Customer Sensitive Data

Posted on February 26, 2017 at 11:29 pm

No Comments

Cloudflare, the popular Content Delivery Network (CDN) trusted by over 5.5 million websites, has warned customers of a recent bug that releases private information to standard search engines. Due to some unusual circumstances, Cloudflare edge servers would run past the end of a buffer and disclose unauthorized data back to users if that data transversed Cloudflare.

While cyber security is always in flux, the most recent bug with Cloudflare, being called Cloudbleed, is one of the worst cases of data breached over the past few years. In fact, many security experts are saying that this bug is as bad as it ever gets because companies using Cloudflare can’t prove to their customer that their private data is secure.

Acting as a proxy, Cloudflare is the middle man between an online user and the actual website that is being visited. This extra level of protection helps optimize and secure websites from malicious attacks because the Cloudflare servers contribute to making the HTTP requests and filter out suspicious activity.

However, the centralized use of Cloudflare servers opens companies to security issues if Cloudflare experiences a bug such as Cloudbleed. Before we continue to understand Cloudbleed, let’s take a quick look at what makes this bug a significant issue:

  • The length of time the bug went unnoticed is substantial. The leakage could be ongoing since September 22, 2017.
  • During the five months of vulnerability, Cloudflare has said the greatest period of impact was February 13 through February 18th. During this period 1 in every 3,300,000 HTTP requests that was processed through the Cloudflare servers potentially resuded in memory leaks. This means that .00003% of HTTP requests could have leaked private information.
  • The bug provided access to highly sensitive information ranging from passwords, cookies, and private user information is among the data that could have been exposed.
  • Cached data on search engines have made sensitive data available to the general public. Since the information was open to public access, private information could have been cached on search engines like Google, Bing and DuckGoGo. This is one of the scariest parts of Cloudbleed because it 's hard to contain the fallout of this bug.

To better understand how much you should be concerned, you can use this website to check if a site utilizes Cloudflare. This information can help you understand the level of danger your private information has been placed in with Cloudbleed and how much proactive action you should take to protect your privacy.

How Could This Happen To Cloudflare Customers?

Cloudbleed is a bug in Cloudflare’s HTML parser, and private data of users on any website using Cloudflare potentially exposed data to anyone making an HTTP request. The reason that this bug is so bad is that the modern web is designed to aggressively cache HTTP responses to help with speed for online content.

The scariest part of this entire situation is how easy it would be to access private data. As information was passed through Cloudflare servers, some data might have been leaked from any website using Cloudflare services. No matter if the information was passed through HTTPS, the private data could have been picked up by third-party scrapers and public search engines to be found by anyone searching.

Many of Cloudflare’s services rely on edge servers that parse and modify HTML pages. This type of operation allows Cloudflare to rewrite http:// pages to https://, insert Google Analytics tags, and enable AMP. The parser allowed for real-time modifications to be made to the HTML pages that Cloudflare performs.

While the parser is not the sole culprit for Cloudbleed, it is one of the ingredients that allowed for this to happen. When combined with several other elements private data would leak from buffers on Cloudflare services.

The root cause of the Cloudflare leak occurred when a pointer was able to step past the end of the buffer. This is called a buffer overrun, and as private information was able to step past the buffer and the HTML parser would leak information if specific conditions were met, including:

  • The final buffer containing data had to finish with a malformed script or img tag.
  • The end user had to either have Email Obfuscation enabled or Automatic HTTPS Rewrites/Server Side Excludes.
  • Another Cloudflare feature that uses an old parser.
  • The buffer had to be less than 4k in length.

Many people are comparing Cloudbleed to the 2014 Heartbleed because data was transferred portions of requests in a similar fashion.

However, Cloudbleed is even scarier because Heartbleed transferred private keys, where\as, the Cloudflare leak transferred private data through means that was exposed to the general public. The Heartbleed bug occurred at the TLS layer, and if you wanted to exploit the Heartbleed bug, someone would have to make a particular TLS request that is not commonly made.

As a result of the Cloudflare bug and how the modern web caches HTTP responses, whatever secrets passed through Cloudflare is available on Google and other search engines.

This means that, unlike Heartbleed, you would not have to be using advanced requests. This means that private data was exposed to the general public without the knowledge of websites storing the information.

Cloudbleed Impacts Mobile Apps Too

The extent of impact on our digital world is only just being realized, as cyber security experts are finding evidence that the memory leak has affected mobile apps. Some mobile apps are designed to make use of some web browsers on the backend for content delivery to users with HTTPS termination.

A recent sampling of approximately 3,500 of the most popular iOS apps showed that about 200 apps use Cloudflare services. Favorite apps like the Bible App, Dropbox, Fitbit, Fiverr, Uber, and Meetup are some of the apps that could be impacted by this bug.

Cloudflare sent and email early February 24, telling their customers if their sites have exposed data in any third party caches. Fortunately, many Cloudflare customers received emails explaining the situation and informing them that their domain was not leaking private information.

If retrieved by search engines, the data will be cached throughout the Internet throughout the lifetime of the bug. This poses similar risks as official websites, but the type of information used in mobile apps tend to me more private than standard use of laptops and tablet use.

If you have downloaded an app that uses Cloudflare, then you should take several proactive steps to protect yourself. First, contact the app’s Customer Service to see if their services were compromised. Second, reset all passwords for active accounts. Finally, monitor your accounts for any suspicious activity.

What Should You Do If Your Business Uses Cloudflare?

Since cached data is at the heart of Cloudbleed, it will be difficult to mitigate the impact of this bug. If you own a website or a service that uses Cloudflare, then you should scour the web to find peaked authentication tokens and user credentials. This will protect your customers and defend your brand.

If you find leaked information, then you will need to terminate related sessions and require password changes for all user accounts that are affected. Also, your business will need to send out emails explaining the actions you are taking and how your business will prevent future issues like this from happening again.

How Will This Problem Get Fixed?

Major search engines like Google and Bing are actively scrubbing cached pages and removing cached information that contains private data. However, this leak has been occurring for at least four months. This means that a massive amount of content has been released across the Internet.

There is no telling who may be running third-party scrapers to scrape and archive private data from affected websites and mobile apps.

Cloudflare tasked a cross-functional team of software engineers, infosec, and operations in two offices in San Francisco and London. This global team has worked in 12-hour intervals so each team can hand off work to each other, allowing for the teams to work on the issue 24 hours a day.

In addition to finding the source of this bug, Cloudflare has also worked with Google, Bing, Yahoo, and other search engines to remove cached pages from their search engines.

Is my site or app affected by Cloudbleed?

There is an unofficial tool where you can check if your site or service/app you're using has been hit by Cloudbleed. Cloudbleed List Checker lets you submit the site you want to check and will show if its possibly affected by the Cloudflare memory leak.