Even the best caching can't escape the fundamental limitations on performance created by the speed of light. Using Load Balancing, Cloudflare's customers can now route requests between multiple origins, allowing them to serve requests from the closest (and fastest) geographic location.

The Cloudflare Load Balancer automatically sends you notifications when things fail, and when they come back up again, so you can sleep well at night knowing we are keeping your website or API running.

If a DDoS attack can bring down your DNS provider or load balancer, it doesn't matter whether your servers are healthy or not. Our load balancing service runs in Cloudflare's 110+ datacenters, and with experience dealing with some of the largest DDoS attacks, we can withstand traffic volumes that smaller providers, virtual machines or hardware appliances can't. This also allows us to help you avoid business-impacting downtime when major cloud compute providers have issues: when we identify a connectivity reaching your application on AWS, we can fail over to your backup infrastructure on Google, a different region, or on-premise servers, and keep your site up and your customers happy.

Further, when proxying traffic through Cloudflare, you immediately benefit from faster failover responsiveness. In building Load Balancing we took a careful look at the existing global load balancing solutions on the market, and found that many rely on DNS to reroute traffic. Cloudflare will work as a DNS load balancer for non-proxied traffic (gray clouded in your DNS control panel) but works as an active proxy with near instant failover for proxied traffic. This means you don’t have to wait for TTLs to expire for traffic to shift.

We know from experience that it's common for a DNS change to take at least 60 seconds to propagate, even when everything is configured perfectly. Sixty seconds can represent thousands of failed requests, frustrated customers and lost sales. Cloudflare’s Load Balancer can reroute proxied traffic almost instantly, giving you the benefits previously reserved for on-premise load balancers but in the cloud.

Even customers with a single origin can benefit from Cloudflare’s Load Balancer. The product includes active and passive monitoring that will now alert you if we have trouble reaching your origin. Cloudflare’s position actively monitoring our customers’ traffic means we can let you know much more quickly than other monitoring services when your site is experiencing a problem

Valérian Saliou, Chief Technology Officer at Crisp, is using Cloudflare to geographically route WebSocket traffic: "When we rolled out Cloudflare Load Balancing to route traffic across our atlas of websocket servers, we immediately got messages from customers in Asia and Oceania thanking us for the improvement."

How Can I Get Started with Load Balancing?

So where do you go from here? You can enable Load Balancing in the Traffic app of your Cloudflare dashboard. We've put together a tutorial on how to get started with failover & health checks between multiple servers.

We want load balancing to be available to everyone. For that reason, Load Balancing starts at $5 per month, and includes 500,000 DNS queries each month (enough for the vast majority of sites).

Setting up a Load Balancer only takes a couple of minutes, and Load Balancers can easily be shared across multiple sites on Cloudflare, avoiding the need for repetitive configuration.

If you want to understand more about Load Balancing on Cloudflare, visit our product page or take a dive through our help articles.

Cloudflare is excited to announce deployments in Detroit (Michigan) and San Diego (California), which are our 114th and 115th data centers respectively. They join Colombo, Sri Lanka and Cape Town, South Africa in the cohort of four new cities added just this week to our growing global network, which spans 57 countries and counting.

For over 6 million Internet properties, we now serve customer traffic from across 26 North American cities, including 22 in the United States alone. We're not going to stop building our network until we're within milliseconds of every Internet user, and to that end, data centers are already in the works in eight additional North American cities (and many others around the world).

Connections

Source: Baja Insider

Detroit and San Diego share something special, as they are immediately adjacent to international borders with Canada and Mexico respectively. Detroit has four border crossings to Windsor, Ontario, including the Ambassador Bridge, which was built in the Roaring Twenties, and accommodates over a quarter of all merchandise trade with Canada.

Founded in 1701, and best known for cars and Motown, Detroit eagerly awaits a 3,000 pound bronze RoboCop statue to watch over Delta City (track progress here). An early member of our technical operations team and fan of Detroit Techno is especially excited that this month's Movement Electronic Music Festival is a Cloudflare customer.

San Diego, the second largest city (by population) in California, is home to deep canyons and its world renowned San Diego Zoo. Later this month, skateboarders across San Diego will celebrate Tony Hawk Day.

Expanding the edge

Projects such as the non-profit Internet exchange DET-IX, enable greater regional interconnection, bringing together hosts, ISPs, and "edge" networks (such as Cloudflare), to exchange traffic locally instead of further away in Chicago. As we turn up peering with additional networks in the coming weeks, we expect to serve a growing share of Internet users in Detroit and San Diego right from their city.

Stay tuned for more data centers. The world's bluest sky will soon welcome the Cloudflare orange cloud.

• The infringement claim is not a close one. The asserted patent, US 6453335 (‘335 patent) was filed in 1998, and describes a system for monitoring an existing data channel and inserting error pages when transmission rates fall below a certain level. Nothing from 1998—including the ’335 patent—comes close to describing our state-of-the-art product that is provisioned via DNS, speeds up internet content delivery, and protects against malicious attackers. Our technology is innovative and different, and Cloudflare’s technology has about 150 patents issued or in process.

• We also expect to show that the patent itself is invalid. For example, if the ’335 patent is read broadly enough to cover our system (which shouldn’t happen), it would also cover any system where electronic communications are examined and redacted or modified. But this is not new. Filtering products performing similar functions were around long before the time of the ‘335 patent. Blackbird’s claims under the ‘335 patent are much, much broader than what is justified in light of the prior art.

This claim is a nuisance for us. The lawsuit will take our time, distract us from our work, and it will cost us money to fight. Patent trolls like Blackbird Technologies (@bbird_tech) exist to create these headaches so companies will play along and give them money to go away.

There’s no social value here. There’s no support for a maligned inventor. There’s no competing business or product. There’s no validation of an incentive structure that supports innovation. This is a shakedown where a patent troll, Blackbird Technologies, creates as much nuisance as it can so its attorney-principals can try to grab some cash.

Indominus rex, Jurassic World

Worse still, Blackbird is a new, especially dangerous breed of patent troll. Like the dinosaur in the latest Jurassic Park movie, a synthetic combination of Tyrannosaurs and Velociraptor, Blackbird combines both a law firm and intellectual property rights holder into a single entity. In doing so, they remove legal fees from their cost structure and can bring lawsuits of potentially dubious merit without having to bear any meaningful cost. In other words, Blackbird’s new breed of entity is specifically designed to add leverage and amplify the already widely maligned problem of patent trolling.

Cloudflare does not intend to play along. As explained later in this blog post, we plan to (i) contest the patent lawsuit vigorously, (ii) fund an award for a crowdsourced search for prior art that can be used to invalidate Blackbird patents, and (iii) ask the relevant bar associations to investigate what we consider to be violations of the rules of professional conduct by Blackbird and its attorneys.

Part I — The Patent Troll Problem and Blackbird Tech

A troll is a creature that lives in a cave, under a bridge, or in the hills and serves no productive purpose—other than to be threatening, ugly, and malodorous. In modern parlance, the term is used for someone who merely seeks to annoy others to get a rise out of them, either in the comments section of an Internet page or by making nuisance legal claims. A patent troll is a company formed for only one purpose—to purchase patents and assert them broadly against real, productive companies that are actively engaged and making and selling products. The point isn’t whether their patents are infringed and/or invalid. A perverse incentive structure encourages trolls to bring lawsuits against numerous companies and demand nuisance settlements—settlements well below the millions of dollars it can take a company to protect themselves and defend a patent infringement case.

"Freemont Troll" by Esther Lee (Flickr)

Although the case against patent trolls is well known, it is worth reviewing.

Patent trolls serve no productive purpose. They don’t make products, conduct research and development, hire employees, or add value to society. This is why they are often called non-practicing entities (NPE). They merely take existing, and largely unexercised patents, and assert them widely against numerous successful companies to try and pry some money for themselves. Often, the trolls lie in wait until after those companies have made irreversible investments in their technology to maximize their leverage. They buy patents solely for the purpose of initiating litigation, and benefit from their non-practicing status to limit their costs in litigation discovery and to limit counterclaims of patent infringement that could be raised if they were actually engaged in productive activity. And increasingly in the world of software or tech patents, the trolls rely on the broadest possible interpretation of vague or generalized patents to sow as much uncertainty as possible.

The social and economic costs of patent trolls are staggering, no matter how you measure them. Estimates in recent years suggest that the number of cases brought by patent trolls have increased more than 500% over the past ten years. And about 70% of all patent infringement claims are filed by patent trolls, a share that has more than doubled in recent years. It is estimated that litigation initiated by patent trolls in U.S. courts cost companies as much as $30 billion in direct costs, a number that has increased more than four-fold over the last ten years. And those direct costs are only the tip of the iceberg, as losses in company value and foregone economic opportunity may be as many as five or ten times that amount.

[See, generally, Chien, Colleen. “Startups and Patent Trolls.” Santa Clara University Legal Studies Research Paper No. 09-12 Working Paper Series, 2012. Bessen, James E. Michael J. Meurer. “The Direct Costs from NPE Disputes.” Boston University School of Law, Law and Economics Research Paper No. 12-34, June 28, 2012. RPX. 2015 Report: NPE Litigation, Patent Marketplace, and NPE Cost.]

Cloudflare’s Support for the Patent System

Make no mistake, Cloudflare is a strong supporter of the patent system. The value and stability of our company is supported by the nearly 150 patents awarded or in process on our technology. We wouldn’t be able to exist, and wouldn’t be able to invest to bring our company to scale, if we didn’t have the security of a well-functioning patent system.

In a fair fight, we’d be able to defend ourselves by pushing back against a practicing entity with the technology set forth in our own patents. If the other entity was actually applying its patent to commercial products and felt hindered in the marketplace because of our operations, we might be able to raise countering arguments that a practicing entity was closer to infringing on our patents than we are to infringing theirs. It would help us to defend the lines around our use of own patented technology and demonstrate whether the infringement was actually happening.

But patent trolls get to live in a fictional world without a business or operations that can be compared or challenged.

And the value of the patent system isn’t one limited to recent developments or to the tech boom. It has supported the innovative spirit of the American people since America’s founding.

It has been reported that during the ransacking of Washington by British soldiers in the War of 1812, then Superintendent of Patents, William Thorton, put himself in front of a British cannon and declared, “Are you Englishmen or only Goths and Vandals? This is the Patent Office, a depository of the ingenuity of the American nation, in which the whole civilized world is interested. Would you destroy it? If so, fire away, and let the charge pass through my body.” Whether or not the result of Superintendent Thorton’s bold stance, the patent building was one of the few government buildings spared by the British army in that attack.

When he redecorated the Oval Office early in his administration, President Barack Obama’s addition of a bust of Martin Luther King, Jr. garnered the most attention. But he also brought in three mechanical devices from the National Museum of American History’s patent collection (models of the telegraph, paddle wheel, and gear-cutter) to reflect the importance of technology and the creative spirit. In that spirit, he later took aggressive action issuing five executive orders to challenge patent trolls (“History Will Remember Barack Obama as the Great Slayer of Patent Trolls,” Wired, 3/20/14).

We are fighting back against patent trolls not because we don’t like the patent system. To the contrary, it’s due to our deep appreciation and need for the patent system that the actions of patent trolls are so offensive to us. Rather than appreciating the economic and social value of an effective patent system, these trolls distort and manipulate the patent system and the court system merely to line their own pockets. Those actions call into question the legitimacy of both systems in a way that is unacceptable to Cloudflare.

Blackbird Technologies

Even though Blackbird Tech is a pure patent troll, which has no operations aside from using patents it purchased to bring litigation against legitimate companies, they have specifically designed their business to create a new breed which poses serious additional concern.

Blackbird was formed three years ago by two attorneys who left law firms where they had been engaged in patent defense work — Wendy Verlander (@bbirdtech_CEO; LinkedIn) at WilmerHale, and Chris Freeman (LinkedIn) at Kirkland & Ellis. Notably, both of those firms promote themselves as ready to protect companies from patent trolls. Kirkland trumpets that its IP practice group scored a victory against the “original patent troll,” while WilmerHale has a Patent Troll Initiative that aims to help businesses deal comprehensively with patent trolls.

Having gained valuable experience and training by working for clients who paid their firms handsomely to fight suits brought by patent trolls, Verlander and Freeman were well aware of the harm done to their clients by patent trolls. Yet, Verlander and Freeman decided to cast their lot with the other side and formed a patent troll for themselves.

Blackbird Technologies has filed 107 cases since September of 2014, making it one of the most prolific trolls in the United States. Its website links to a “News” item titled “4 Frequent Filers of IP Suits to Watch this Year” which highlights Blackbird as “a newer entrant on the list of top patent plaintiffs, coming in at fourth place with 48 suits last year in the District of Delaware spanning a wide range of technologies.” Some of the patents at issue include: Bicycle Pet Carrier, Buttock Lift Support, Sports Bra, and Method for Managing a Parking Lot. A complete list of Blackbird's patents is available here. Although they have been very aggressive about filing such claims, they have still not taken a single case through trial. And only a couple of those cases made it to the claim construction phase, where the Court defines the meaning of the patents at issue. Instead, many of Blackbird’s cases have been resolved shortly after filing, suggesting that these cases were never about legal rights or claims but were instead about creating the impetus for a nuisance settlement in the face of significant litigation costs.

In the Cloudflare case, Blackbird is asserting a 1998 patent filed by Oliver Kaufmann. Based on an assignment agreement filed with the USPTO, Blackbird purchased the ‘335 patent from Mr. Kaufmann in October 2016 for $1 plus “other good and valuable consideration.” Because a mere five months later Blackbird used the ‘335 patent as the sole basis for lawsuits against Cloudflare and Fastly, we think the actual but undisclosed compensation between the parties is something considerably more than $1, and may very well involve a contingency arrangement for Mr. Kaufmann.

Mr. Kaufmann appears to be the head of a group of four small IT companies in Augsburg, Germany that share a website — www.exklusiv.de. The companies provide services such as on premise networking and phone, ISP, database design and management, and 3D printing. If Mr. Kauffman ever tried to commercialize the ‘335 patent (in the U.S. or otherwise), we would expect to see some evidence of that on the website. We don’t. Of course, the patent troll system attempts to shield Mr. Kaufmann and his companies’ operations from the lawsuit by having Blackbird become the owner of the patent — even if Mr. Kaufmann maintains an interest in the litigation.

Part II — Cloudflare’s Response

Cloudflare’s mission has always been to help build a better Internet. So it won’t be surprising to frequent readers of this blog that Cloudflare isn’t interested in a short term and narrow resolution of our own interests. We’re not going to reach a settlement that would pay tens of thousands of dollars to Blackbird to avoid millions in legal fees. That would only allow patent trolls to keep playing their game and preying upon other innovative companies that share our interest in making the Internet work better, especially newer and more vulnerable companies.

We are pursuing a four-pronged approach to challenge Blackbird’s actions in this case.

Step 1 — Cloudflare will fight this case in the courts

Cloudflare will not settle this case, and doesn’t plan to settle any patent troll case, ever.

We will litigate this claim as vigorously and expeditiously as possible. Because it is clear to us after even minimal search that there is likely considerable evidence of prior art, we expect to file an Inter Parties Review of the ‘335 patent with the USPTO. If appropriate, we will seek sanctions and fee awards against Blackbird for bringing this case.

Step 2 — Cloudflare will fund a crowdsourced effort to find evidence to invalidate Blackbird’s patents… all of them

In a separate blog post today, Cloudflare is announcing that it will award up to $50,000 to support a search for prior art that can be used to invalidate Blackbird’s patents. Patent trolls like Blackbird have demonstrated that they can’t be trusted to act responsibly or take the public good into account when asserting patents against practicing companies.

Crowd by Hamaz Butt (Flickr)

Patent trolls take advantage of a system they assume is tilted in their favor, where they can take vague technology patents issued years ago and apply them as broadly as imaginable to the latest technology. Patent trolls think they can sit back and pick off settlements from companies because their lawsuits are a nuisance and the costs of defending those suits are considerable.

Changing this dynamic and leveling the playing field is going to require an entirely new way of approaching these challenges. Fighting such strong, though perverse, economic incentives is going to require a groundswell. It calls for a broad and active group of innovative folks who are willing to join the cause of helping expose the weaknesses in the patents held by these patent trolls, most of which were issued so long ago and are so broad that they never should have been issued in the first place.

Exposing the weakness of these patents means finding prior art that demonstrates the patents were issued improperly and working to have the USPTO invalidate those patents before they can be weaponized against legitimate companies that are working to innovate, to provide a livelihood to their employees and their families, and to make good products. We aim to invalidate every one of Blackbird’s patents and need your help. Come join us in this effort!

Prior art is any evidence that a patented invention was already known at the time the patent application was filed. Prior art does not need to be a competing patent, it does not necessarily need to exist physically or be commercially available. It is enough that there is evidence that the innovation was publicly known at any point when or before the patent application was filed. Though certainly the more formal and specific the prior art, the better.

As described in more detail in the other blog post, Cloudflare will award up to $20,000 to be shared among the participants who provide us the most useful prior art that can be used in our invalidity challenge to the ‘335 patent in the pending litigation. We will use our outside patent counsel to evaluate the responses and divide the award based on relevance and usefulness. We will also accept information and arguments which invalidate Blackbird patents such as a showing of obviousness or material defects in the patent application.

In addition, we are seeking prior art information on any of the 37 other patents or patent applications owned by Blackbird.

We have a pool of up to $30,000 that we will distribute to people who submit information on any Blackbird patent. Again, the money will be awarded based on relevance and usefulness of the prior art. We will publish all the information that we receive on these patents, so that others may benefit from the information and file challenges on Blackbird’s patents. And where justified, Cloudflare may institute patent office proceedings of any Blackbird patent when there is sufficient prior art to call into question its validity.

These opportunities remain open as long as Blackbird’s case against Cloudflare is still active. If interested, you should go over to the other blog post here, join the cause and be one of a group of talented and motivated experts who will push back against lazy trolls who try to take advantage of innovators by exploiting bugs in the judicial system.

Step 3 — Cloudflare will investigate Blackbird’s operations to develop facts that support our arguments in the litigation and expose how patent trolls really operate

Because patent trolls do not have operations beyond their ownership and litigation of patents for inventions they did not create, it can often be difficult to litigate against them. In addition, because of the unique organization of Blackbird, which appears to actually be a law firm that brings lawsuits on its own behalf and without a real client, there are important facts about this dispute that are difficult or impossible for us to determine at present.

As a result, we expect to investigate and to use the discovery process in this case and to retain a group of private investigators to look into Blackbird’s operations and attorneys to determine what counterclaims and arguments may be available to us in defense of the case. At a minimum, Blackbird suggests that it is engaged in a new and even more ingenious way of operating as a patent troll with maximum leverage and minimal inefficiencies in their operation. At a time when the economic evidence and public sentiment against the work of patent trolls is strong, it is important to bring their operations to the light of day and understand exactly how they work.

Step 4 — File complaints against Blackbird attorneys by bar association disciplinary counsel in Massachusetts and Illinois

The practice of law is supposed to be different from business, with lawyers held to a higher ethical standard. In most states, it is illegal for those who have not been admitted to the bar to provide legal services. This often gives lawyers a privileged position when it comes to handling things like real estate transactions, tax matters, and estate planning that might also be handled by other professionals, and it gives them a near monopoly on access to and use of the court system. In exchange, the legal profession is strictly governed by a series of ethical rules and other rules of professional conduct to make sure the profession sufficiently self-regulates to maintain public confidence and therefore secure the advantages of its monopoly. That’s why admission of new lawyers to the bar is generally premised not only on proof of competence (the bar exam), but also an investigation of character and fitness, and an oath committing to the principles of the profession. Importantly, the rules support the notion that the practice of law is a profession and not a mere business endeavor, this distinction is seen as essential to distinguishing the legal profession and maintaining the monopoly. See, generally Richard L. Abel, American Lawyers (1989).

Specifically, the American Bar Association established its Model Rules of Professional Conduct in 1983, which have subsequently been adopted by bar associations in almost every state, including the two states where Blackbird has its offices, Massachusetts and Illinois. The rules set forth a series of responsibilities that each lawyer must undertake, including duties to the profession generally and duties of candor to the court, which may require actions that run counter to the attorney’s self-interest.

For example, attorneys have an affirmative obligation to disclose when they have knowledge of adverse controlling legal authority (a rule or controlling court opinion that invalidates their proposed outcome), even if such disclosure runs counter to the interests of the attorney’s client. This disclosure is required even if both opposing counsel and the court are otherwise unaware of that authority (Model Rule 3.3(a)(2)). So attorneys are not allowed to take advantage of the judicial system just because they have an opening to do so.

Most of the rules address the extensive obligations owed to a client, to make sure lawyers don’t leverage their monopoly position to take advantage of the client. Among other things, the special nature of the Attorney-Client relationship involves a number of controls on the business relationship between the two — the attorney is to strictly avoid a conflict of interest with her work for the client (Rules 1.6-1.10), and should avoid self-dealing or commingling of funds (Rules 1.8 and 1.15).

As made clear in this blog post, Blackbird’s founders made the decision to leave law firms that were engaged in the defense of clients who were faced with patent lawsuits, and formed a new law firm focused on bringing law suits as a patent troll. The only services promoted on its website (http://www.blackbird-tech.com/) are legal services; the website notes that Blackbird represents a “new model” which provides the benefits of “top law firm experience” offering clients the ability to “litigate at reduced costs.”

Of 12 total employees listed on the Blackbird website, 7 are attorneys. The remaining 5 are very junior employees described as “analysts” (3 are current undergraduate students and 2 received Bachelor's degrees last May). As far as we can determine, Blackbird produces no products or services which it makes available to the public. Rather, it offers litigation services and is in the business of filing lawsuits. And its output in that regard is prolific, as it has filed a total of 107 lawsuits since September 2014.

As final confirmation that Blackbird is a law firm marketing legal services, its own website includes a disclaimer about “Attorney Advertising,” which states explicitly “[p]lease note that this website may contain attorney advertising.”

Blackbird’s “new model” seems to be only that its operations set out to distort the traditional Attorney-Client relationship. Blackbird’s website makes a direct pitch of its legal services to recruit clients with potential claims and then, instead of taking them on as a client, purchases their claims and provides additional consideration that likely gives the client an ongoing interest in the resulting litigation. In doing so, Blackbird is flouting its ethical obligations meant to protect clients and distorting the judicial process by obfuscating and limiting potential counterclaims against the real party in interest.

As a result, we think Blackbird may be engaged in several activities that violate the rules of professional responsibility in the jurisdictions where they have been admitted as members of the bar. So, Cloudflare is submitting today letters with the bar disciplinary committees in Massachusetts and Illinois asking them to look into the following practices of Blackbird Tech:

1. Blackbird may have acquired a proprietary interest in the subject matter of the litigation in violation of Rule 1.8(i) — Attorneys have a near monopoly of representing clients in the judicial system. Rule of Professional Conduct 1.8(i) explicitly prohibits an attorney from “acquir[ing] a proprietary interest in a cause of action or subject matter of litigation.” But that is exactly what Blackbird does. Blackbird’s website contains a pitch to recruit clients with potential legal claims under their patents, but then buys those claims and brings them on their own behalf. Wouldn’t that be a violation of Rule 1.8(i)? Doesn’t Blackbird’s attempt to pitch this as a “new model” of being a patent troll ignore the fact that the only non-law firm activity in which they are engaged (buying patents to bring lawsuits) is the exact thing prohibited by Rule 1.8(i)? They shouldn’t be able to use creative contractual or corporate structures to avoid its responsibility under the rules.

2. Blackbird may be sharing fees or firm equity with non-lawyers in violation of Rule 5.4(a) or 5.4(d) — In order to preserve the integrity of the Attorney-Client relationship, Rule of Professional Conduct 5.4(a) prohibits attorneys from splitting legal fees in individual matters with non-lawyers, and Rule 5.4(d) prohibits providing an equity interest in a firm to non-lawyers. We think Blackbird may be violating both provisions. Although he no longer owns the patent and is not a party to the case, the assignment agreement’s terms (specifying payment of only $1) makes it possible that Mr. Kaufman has a contingency interest in the lawsuit. If that is the case, wouldn’t Blackbird be in violation of Rule 5.4(a)? Similarly, Blackbird has moved very quickly since its founding to file lawsuits against a great number of companies — 107 complaints since September 2014. So far, none of those cases have gone to trial. We intend to examine whether they have used financial support from non-lawyers to fund the very fast start to their operations in exchange for an impermissible equity interest, or have shared an equity interest with patent holders like Mr. Kaufmann, either of which would be in violation of Rule 5.4(d).

In Closing

We've always advised our customers not to pay threatened DDoS attackers' ransom requests but, instead, to mount a real defense. There are a lot of analogies here and we believe it's important to stand up for what is right and against this new breed of patent troll.

If you’ve read this far, then you must share Cloudflare’s passion on this issue. So join us! Write about and talk about these issues. Contact your political representatives and ask them to support all the legislative reform efforts that are out there. And most importantly, take some time and join our crowdsourced effort to identify prior art that will invalidate all of the Blackbird patents.

Watch this space. We’re just getting started.

As readers of this blog likely know, especially if you read this post, Cloudflare has been sued by a dangerous new breed of patent troll, Blackbird Technologies, asserting a very old and very vague patent. And we know we are not alone in being frustrated about the way that such patent trolls inhibit the growth of innovative companies. Cloudflare is asking for your help in this effort, and we’re putting our money where our mouth is.

Patent trolls take advantage of a system they assume is tilted in their favor, where they can take vague technology patents issued years ago and apply them as broadly as imaginable to the latest technology. And they do this without the limitations of having to show the original patent holder would have actually exercised the patent, because most of them don’t, at all. Patent trolls think they can sit back and pick off settlements from companies because their lawsuits are a nuisance and the costs of defending those suits are considerable.

Changing this dynamic and leveling the playing field is going to require an entirely new approach. Fighting such strong, though perverse, economic incentives is going to require a groundswell. It calls for a large and active group of innovative folks who are willing to join the cause of helping expose the weakness in the patents held by these patent trolls. Exposing the weakness of these patents means finding prior art that demonstrates the patents were issued improperly and working to have the USPTO invalidate those patents before they can be weaponized against legitimate companies that are working to innovate, to provide a livelihood to their employees and their families, and to make good products.

At Cloudflare, we’re lucky to have the best and most loyal customers on the Internet, a whole bunch of fans who are ridiculously smart and don’t take kindly to lazy trolls who try to take advantage by exploiting bugs in the judicial system. And of course, we’re more than happy to give them a bit of a nudge.

Indiana Jones: Raiders of the Lost Ark

Awards for Prior Art

Cloudflare is committing up to $50,000 to support a search for prior art that can be used to invalidate patents held by Blackbird Tech…all of them. By filing 107 lawsuits against companies since September 2014, Blackbird has demonstrated that it is going to use its patents to sue productive companies. And patent trolls often target newer and more innovative companies, so it is important to investigate and research to discover whether prior art existed on the Blackbird patents and to keep them from filing additional cases against us or against other companies.

By “prior art,” we mean evidence that the patented technology was in use or known before the inventor conceived of or filed for his patent. Often, prior art consists of academic papers, technical websites describing current technology, other patents/patent applications—or, really, any other evidence showing that the technology was in use or known. For example, for the ‘335 patent, we are only asking for evidence that pre-dates the filing date of the patent—July 21, 1998. We will also accept information and arguments which invalidate Blackbird patents such as a showing of obviousness or material defects in the patent application. And, for our purposes, the “patented technology” is defined by the claims that are listed at the end of the ’355 patent.

The $50,000 will fund two separate awards.

The first bounty (up to $20,000) is for prior art which reads on the patent Blackbird is using to sue Cloudflare, the ‘335 patent. $10,000 is guaranteed and will be divided among prior art submissions that raise substantive questions on the ‘335 patent. The remaining $10,000 will be used to compensate prior art submissions that Cloudflare uses as evidence in an invalidation procedure at the USPTO or invalidation at trial. The latest date of prior art on the ‘335 patent would be July 20, 1998.

The larger bounty (up to $30,000) will be spread among those submitting substantial prior art which reads on any of the 34 other outstanding Blackbird patents or their 3 in-flight patent applications and could lead to the invalidation of these dubious patents. Cloudflare will pay the second bounty to people who submit relevant and substantive prior art which, in Cloudflare’s opinion, reads on any other Blackbird patent. The money will be distributed based on the quality of the prior art, the perceived value of the patent, and the extent to which the evidence is used in a proceeding to invalidate one of the Blackbird patents.

We will maintain a list of all the Blackbird patents at cloudflare.com/blackbirdpatents/. The list will provide the number of each patent, the relevant latest date of prior art, and will list germane already-identified prior art. We will update the list periodically as we get new information submitted.

How to participate

You may submit your suggestion for prior art either through cloudflare.com/priorartsearch/. Cloudflare’s experts and attorneys will review each submission for its value in invalidating each patent. Again, the money will be awarded based on relevance and usefulness. Cloudflare will keep the award will be open until the end of our litigation, but we may start making initial partial awards as early as within 90 days.

In order to be eligible for any bounty you need to be the first person to submit a particular piece of prior art and the prior art must not be known to Cloudflare (e.g., the prior art must not be listed on the face of the Blackbird patent or in the patent prosecution’s history). For each submission of prior art, you will need to give us your name and contact information. In your submission, you will need to document:

• Your name and email address
• The number of the Blackbird patent at issue
• What the prior art is.
• Where you found it.
• When it was first publicly available.
• Where it was first publicly available.
• Why you think it reads on a Blackbird’s patent.
• Any other information which you think is helpful to our review.

Big hint: don’t go to Google’s patent page and click on the prior art button, we’ve already done that. We will then use our outside patent counsel to evaluate the responses and divide the award based on relevance and usefulness. See here cloudflare.com/priorartrules/ for official rules and details.

Patent trolls like Blackbird have demonstrated that they can’t be trusted to act responsibly or take the public good into account when asserting patents against practicing companies. Making clear that its patents are invalid, or significantly narrower than Blackbird is likely to assert, is another way we can limit Blackbird’s ability to create economic harm.

How logs come in from the edge

Cloudflare already has a data pipeline for HTTP logs. We wanted to utilize what we could of that system for the new DNS analytics. Every time one of our edge services gets an HTTP request, it generates a structured log message in the Cap’n Proto format and sends it to a local multiplexer service. Given the volume of the data, we chose not to record the full DNS message payload, only telemetry data we are interested in such as response code, size, or query name, which has allowed us to keep only ~150 bytes on average per message. It is then fused with processing metadata such as timing information and exceptions triggered during query processing. The benefit of fusing data and metadata at the edge is that we can spread the computational cost across our thousands of edge servers, and log only what we absolutely need.

The multiplexer service (known as “log forwarder”) is running on each edge node, assembling log messages from multiple services and transporting them to our warehouse for processing over a TLS secured channel. A counterpart service running in the warehouse receives and demultiplexes the logs into several Apache Kafka clusters. Over the years, Apache Kafka has proven to be an invaluable service for buffering between producers and downstream consumers, preventing data loss when consumers fail over or require maintenance. Since version 0.10, Kafka allows rack-aware allocation of replicas, which improves resilience against rack or site failure, giving us fault tolerant storage of unprocessed messages.

Having a queue with structured logs has allowed us to investigate issues retrospectively without requiring access to production nodes, but it has proven to be quite laborious at times. In the early days of the project we would skim the queue to find offsets for the rough timespan we needed, and then extract the data into HDFS in Parquet format for offline analysis.

About aggregations

The HTTP analytics service was built around stream processors generating aggregations, so we planned to leverage Apache Spark to stream the logs to HDFS automatically. As Parquet doesn’t natively support indexes or arranging the data in a way that avoids full table scans, it’s impractical for on-line analysis or serving reports over an API. There are extensions like parquet-index that create indexes over the data, but not on-the-fly. Given this, the initial plan was to only show aggregated reports to customers, and keep the raw data for internal troubleshooting.

The problem with aggregated summaries is that they only work on columns with low cardinality (a number of unique values). With aggregation, each column in a given time frame explodes to the number of rows equal to the number of unique entries, so it’s viable to aggregate on something like response code which only has 12 possible values, but not a query name for example. Domain names are subject to popularity, so if, for example, a popular domain name gets asked 1,000 times a minute, one could expect to achieve 1000x row reduction for per-minute aggregation, however in practice it is not so.

Due to how DNS caching works, resolvers will answer identical queries from cache without going to the authoritative server for the duration of the TTL. The TTL tends to be longer than a minute. So, while authoritative servers see the same request many times, our data is skewed towards non-cacheable queries such as typos or random prefix subdomain attacks. In practice, we see anywhere between 0 - 60x row reduction when aggregating by query names, so storing aggregations in multiple resolutions almost negates the row reduction. Aggregations are also done with multiple resolution and key combinations, so aggregating on a high cardinality column can even result in more rows than original.

For these reasons we started by only aggregating logs at the zone level, which was useful enough for trends, but too coarse for root cause analysis. For example, in one case we were investigating short bursts of unavailability in one of our data centers. Having unaggregated data allowed us to narrow the issue down to the specific DNS queries experiencing latency spikes, and then correlated the queries with a misconfigured firewall rule. Cases like these would be much harder to investigate with only aggregated data as it only affected a tiny percentage of requests that would be lost in the aggregations.

So we started looking into several OLAP (Online Analytical Processing) systems. The first system we looked into was Druid. We were really impressed with the capabilities and how the front-end (Pivot and formerly Caravel) is able to slice and dice the data, allowing us to generate reports with arbitrary dimensions. Druid has already been deployed in similar environments with over 100B events/day, so we were confident it could work, but after testing on sampled data we couldn’t justify the hardware costs of hundreds of nodes. Around the same time Yandex open-sourced their OLAP system, ClickHouse.

And then it Clicked

ClickHouse has a much simpler system design - all the nodes in a cluster have equal functionality and use only ZooKeeper for coordination. We built a small cluster of several nodes to start kicking the tires, and found the performance to be quite impressive and true to the results advertised in the performance comparisons of analytical DBMS, so we proceeded with building a proof of concept. The first obstacle was a lack of tooling and the small size of the community, so we delved into the ClickHouse design to understand how it works.

ClickHouse doesn’t support ingestion from Kafka directly, as it’s only a database, so we wrote an adapter service in Go. It read Cap’n Proto encoded messages from Kafka, converted them into TSV, and inserted into ClickHouse over the HTTP interface in batches. Later, we rewrote the service to use a Go library using the native ClickHouse interface to boost performance. Since then, we’ve contributed some performance improvements back to the project. One thing we learned during the ingestion performance evaluation is that ClickHouse ingestion performance highly depends on batch size - the number of rows you insert at once. To understand why, we looked further into how ClickHouse stores data.

The most common table engine ClickHouse uses for storage is the MergeTree family. It is conceptually similar to LSM algorithm used in Google’s BigTable or Apache Cassandra, however it avoids an intermediate memory table, and writes directly to disk. This gives it excellent write throughput, as each inserted batch is only sorted by “primary key”, compressed, and written to disk to form a segment. The absence of a memory table or any notion of “freshness” of the data also means that it is append-only and data modification or deletion isn’t supported. The only way to delete data currently is to remove data by calendar months, as segments never overlap a month boundary. The ClickHouse team is actively working on making this configurable. On the other hand, this makes writing and segment merging conflict-free, so the ingestion throughput scales linearly with the number of parallel inserts until the I/O or cores are saturated. This, however, also means it is not fit for tiny batches, which is why we rely on Kafka and inserter services for buffering. ClickHouse then keeps constantly merging segments in the background, so many small parts will be merged and written more times (thus increasing write amplification) and too many unmerged parts will trigger aggressive throttling of insertions until the merging progresses. We have found that several insertions per table per second work best as a tradeoff between real-time ingestion and ingestion performance.

The key to table read performance is indexing and the arrangement of the data on disk. No matter how fast processing is, when the engine needs to scan terabytes of data from disk and use only a fraction of it, it’s going to take time. ClickHouse is a columnar store, so each segment contains a file for each column, with sorted values for each row. This way whole columns not present in the query can be skipped, and then multiple cells can be processed in parallel with vectorized execution. In order to avoid full scans, each segment also has a sparse index file. Given that all columns are sorted by the “primary key”, the index file only contains marks (captured rows) of every Nth row in order to be able to keep it in memory even for very large tables. For example the default settings is to make a mark of every 8,192th row. This way only 122,070 marks are required to sparsely index a table with 1 trillion rows, which easily fits in memory. See primary keys in ClickHouse for a deeper dive into how it works.

When querying the table using primary key columns, the index returns approximate ranges of rows considered. Ideally the ranges should be wide and contiguous. For example, when the typical usage is to generate reports for individual zones, placing the zone on the first position of the primary key will result in rows sorted by zone in each column, making the disk reads for individual zones contiguous, whereas sorting primarily by timestamp would not. The rows can be sorted in only one way, so the primary key must be chosen carefully with the typical query load in mind. In our case, we optimized read queries for individual zones and have a separate table with sampled data for exploratory queries. The lesson learned is that instead of trying to optimize the index for all purposes and splitting the difference, we have made several tables instead.

One such specialisation are tables with aggregations over zones. Queries across all rows are significantly more expensive, as there is no opportunity to exclude data from scanning. This makes it less practical for analysts to compute basic aggregations on long periods of time, so we decided to use materialized views to incrementally compute predefined aggregations, such as counters, uniques, and quantiles. The materialized views leverage the sort phase on batch insertion to do productive work - computing aggregations. So after the newly inserted segment is sorted, it also produces a table with rows representing dimensions, and columns representing aggregation function state. The difference between aggregation state and final result is that we can generate reports using an arbitrary time resolution without actually storing the precomputed data in multiple resolutions. In some cases the state and result can be the same - for example basic counters, where hourly counts can be produced by summing per-minute counts, however it doesn’t make sense to sum unique visitors or latency quantiles. This is when aggregation state is much more useful, as it allows meaningful merging of more complicated states, such as HyperLogLog (HLL) bitmap to produce hourly unique visitors estimate from minutely aggregations. The downside is that storing state can be much more expensive than final values - the aforementioned HLL state tends to be 20-100 bytes / row when compressed, while a counter is only 8 bytes (1 byte compressed on average). These tables are then used to quickly visualise general trends across zones or sites, and also by our API service that uses them opportunistically for simple queries. Having both incrementally aggregated and unaggregated data in the same place allowed us simplify the architecture by deprecating stream processing altogether.

Infrastructure and data integrity

We started with RAID-10 composed of 12 6TB spinning disks on each node, but reevaluated it after the first inevitable disk failures. In the second iteration we migrated to RAID-0, for two reasons. First, it wasn’t possible to hot-swap just the faulty disks, and second the array rebuild took tens of hours which degraded I/O performance. It was significantly faster to replace a faulty node and use internal replication to populate it with data over the network (2x10GbE), than to wait for an array to finish rebuilding. To compensate for higher probability of node failure, we switched to 3-way replication and allocated replicas of each shard to different racks, and started planning for replication to a separate data warehouse.

Another disk failure highlighted a problem with the filesystem we used. Initially we used XFS, but it started to lock up during replication from 2 peers at the same time, thus breaking replication of segments before it completed. This issue has manifested itself with a lot of I/O activity with little disk usage increase as broken parts were deleted, so we gradually migrated to ext4 that didn’t have the same issue.

Visualizing Data

At the time we relied solely on Pandas and ClickHouse’s HTTP interface for ad-hoc analyses, but we wanted to make it more accessible for both analysis and monitoring. Since we knew Caravel - now renamed to Superset - from the experiments with Druid, we started working on an integration with ClickHouse.

Superset is a data visualisation platform designed to be intuitive, and allows analysts to interactively slice and dice the data without writing a single line of SQL. It was initially built and open sourced by AirBnB for Druid, but over time it has gained support for SQL-based backends using SQLAlchemy, an abstraction and ORM for tens of different database dialects. So we wrote and open-sourced a ClickHouse dialect and finally a native Superset integration that has been merged a few days ago.

Superset has served us well for ad-hoc visualisations, but it is still not polished enough for our monitoring use case. At Cloudflare we’re heavy users of Grafana for visualisation of all our metrics, so we wrote and open-sourced a Grafana integration as well.

It has allowed us to seamlessly extend our existing monitoring dashboards with the new analytical data. We liked it so much that we wanted to give the same ability to look at the analytics data to you, our users. So we built a Grafana app to visualise data from Cloudflare DNS Analytics. Finally, we made it available in your Cloudflare dashboard analytics. Over time we’re going to add new data sources, dimensions, and other useful ways how to visualise your data from Cloudflare. Let us know what you’d like to see next.

Does solving these kinds of technical and operational challenges excite you? Cloudflare is always hiring for talented specialists and generalists within our Engineering, Technical Operations and other teams.

I appreciate the feedback we received. How we handle abuse reports has evolved over the last six and a half years of Cloudflare's history. I wanted to take this opportunity to walk through some of the rationale that got us to this point and caused us to have a blindspot to the case that was highlighted in the article.

What Is Cloudflare?

Cloudflare is not a hosting provider. We do not store the definitive copy of any of the content that someone may want to file an abuse claim about. If we terminate a customer it doesn’t make the content go away. Instead, we are more akin to a specialized network. One of the functions of the network that we provide is to add security to the content providers that use us. Part of doing that inherently involves hiding the location of the actual hosting provider. If we didn't do this, a malicious attacker could simply bypass Cloudflare by attacking the host directly.

That created an early question on what we should do when someone reported abusive content that was passing through our network. The first principle was we believed it was important for us to not stand in the way of valid abuse reports being submitted. The litmus test that we came up with was that the existence of Cloudflare ideally shouldn't make it any harder, or any easier, to report and address abuse.

Mistakes of Early Abuse Reporting

The majority (83% over the last week) of the abuse reports that we get involve allegedly copyrighted material transiting our network. Our early abuse policy specified that if we received an abuse report alleging copyrighted material we'd turn over the IP address of the hosting provider so the person filing the abuse report could report the abuse directly.

It didn't take long for malicious attackers to realize this provided an effective way to bypass our protections. They would submit a fake report alleging some image on a legitimate site had been illegally copied, we'd turn over the IP address of our customer, and they'd attack it directly. Clearly that wasn't a workable model.

As a result, we revised our policy to instead act as a proxy for abuse reports that were submitted to us. If a report was submitted then we'd proxy the report through and forward it to the site owner as well as the site's host. We provided the contact information so the parties could address the issue between themselves.

While we have a Trust & Safety team that is staffed around the clock, for the most part abuse handling is automated. Various firms that specialize in finding and taking down copyrighted material generate such a flood, often submitting hundreds of reports for the same allegedly copyrighted item, that manual review of every report would be infeasible.

Violent Threats and Child Sexual Abuse

We've always treated reports of violent threats and child sexual abuse material with additional care. Understandably, from the perspective of the individuals in the ProPublica article, it seems callous and absurd that we would ever forward these reports to the site owner. However, we had a different perspective.

The vast majority of times that violent threats or child sexual abuse material were reported to us occurred on sites that were not dedicated to those topics. Imagine a social network like Facebook was a Cloudflare customer. Somewhere on the site something was posted that included a violent threat. That post was then reported to Cloudflare as the network that sits in front of the Facebook-like site.

In our early days, it seemed reasonable and responsible to pass the complaint on to the Facebook-like customer who could then follow up directly. That also met the litmus test of being what would happen if Cloudflare didn't exist. What the policy didn't account for was site owners who could not be trusted to act responsibly with abuse reports including contact information.

Anonymous Reporting

Beginning in 2014, we saw limited, but very concerning, reports of retaliation based on submitted abuse reports. As a result, we adjusted our process to make it so complaints about violent threats and child sexual abuse material would be sent only to the host, not to the site owner.

We’ve confirmed that in the cases reported to the site mentioned in the ProPublica article we followed this procedure. That change largely addressed the problem of people reporting abuse getting harassed. What we didn’t anticipate is that some hosts would themselves pass the full complaint, including the reporter’s contact information, on to the site owner. We assume this is what happened in the ProPublica cases.

Another change we made in 2015 was to clarify exactly what would happen when someone submitted a report by adding disclaimers to our abuse form. These disclaimers appear in multiple places throughout the abuse submission flow:

In a world without Cloudflare, if you wanted to anonymously report something, you would use a disposable email and a fake name and submit a report to the site's hosting provider or the site itself. We didn't do anything to check that the contact information used in reports was valid so we assumed, with the disclaimer in place, if people wanted to submit reports anonymously they'd do the same thing as they would have if Cloudflare didn't exist.

That was a bad assumption. As the ProPublica article made clear, many people did not read or understand the disclaimer and were surprised that we forwarded their full abuse report to the host who then, in some cases, could forward it to the site owner.

Determining Bad Actors

In reevaluating our policy a key question was when it is appropriate to pass along the full report and when it is not. Again, from the perspective of the author of the ProPublica article, that may seem like an easy distinction. The reality is that requiring an individual working on our Trust & Safety team understand the nature of every site that is on Cloudflare is untenable. Moreover, adding more human intervention that slows down the process of reporting abuse, especially in cases of violent threats and child sexual abuse material, where time may be of the essence, strikes us as a step backward.

Instead, we took the suggestions of many of the comments we received and are implementing a policy where reporters of these types of abuse can choose to submit them and not have their contact information included in what we forward. The person making the abuse report seems in the best position to judge whether or not they want their information to be relayed. Making this change requires some engineering work on our part, but we have prioritized it. By the end of this week, someone submitting an abuse report for one of these categories will have the choice of whether to do so anonymously.

Ongoing Improvements

We are under no illusion that this latest iteration of our abuse process is perfect. In fact, we already have concerns about challenges the new system will create. Anonymous reporting opens a new vector for malicious actors to submit false reports and harass Cloudflare customers. In addition, for responsible Cloudflare customers who want to act on reports, anonymous reports may make it more difficult for them to gather more information from the reporter which may make it more difficult for well-informed action to be taken to address the issue.

We appreciate the feedback on where our previous process broke down. As new problems arise, we anticipate that we'll continue to need to make changes to how we handle abuse reports.

Final Thoughts on Censoring the Internet

While we clearly had a significant blindspot in how we handled one type of abuse reports, we remain committed to our belief that it is not Cloudflare's role to make determinations on what content should and should not be online. That belief comes from a number of principles.

Cloudflare is more akin to a network than a hosting provider. I'd be deeply troubled if my ISP started restricting what types of content I can access. As a network, we don't think it's appropriate for Cloudflare to be making those restrictions either.

That is not to say we support all the content that passes through Cloudflare's network. We, both as an organization and as individuals, have political beliefs and views of what is right and wrong. There are institutions — law enforcement, legislatures, and courts — that have a social and political legitimacy to determine what content is legal and illegal. We follow the lead of those organizations in all the jurisdictions we operate. But, as more and more of the Internet sits behind fewer and fewer private companies, we're concerned that the political beliefs and biases of those organizations will determine what can and cannot be online.

If you're interested, I gave a talk a few years ago about how we think about our role in policing online content. It's about an hour long, but if you're interested in the topic, I encourage you to watch it in order to better understand our perspective.

From time to time an organization will sign up for Cloudflare that we find revolting because they stand for something that is the opposite of what we think is right. Usually, those organizations don't pay us. Every once in awhile one of them does. When that happens it's one of the greatest pleasures of my job to quietly write the check for 100% of what they pay us to an organization that opposes them. The best way to fight hateful speech is with more speech.

I appreciate the feedback on how we can improve our abuse process. We are implementing the changes that were recommended. They take engineering, so they aren't available immediately, but will be live by the end of this week. We continue to iterate and improve on our mission of helping build a better Internet.

Have you noticed something new in your Cloudflare analytics dashboard this morning? You can now see detailed DNS analytics for your domains on Cloudflare.

If you want to skip to the punch and start exploring, go check it out here. Otherwise, hop on the DNS magic school bus - and let us show you all the neat stats in your now-available DNS analytics.

DNS analytics dashboard: What does it know? Does it know things? Let’s find out.

At the top of the DNS analytics dashboard you can see your DNS traffic health. This “Queries by Response Codes” graph breaks down queries by what response code Cloudflare DNS answered to the visitor. Like HTTP response codes, DNS response codes give an indication of what is happening behind the scenes. Mostly you will just see NOERROR, the HTTP 200 of DNS response codes, and NXDOMAIN, the HTTP 404 of DNS response codes. NXDOMAIN is particularly interesting - what are people querying for that doesn’t exist?

If you are an enterprise customer and you want to know what all the NXDOMAIN queries are, just scroll down a little bit where we show you the top queries for your domain and top queries for your domain for DNS records that don’t exist (aka top NXDOMAIN queries).

If you are curious then where all of these NXDOMAIN queries are coming from, all Pro, Business and Enterprise plan customers can scroll down a little bit further to the geography section where we show you the breakdown of where your queries come from and also where your queries returning NXDOMAIN come from.

If you are an Enterprise customer and want to dive in deeper just for one or a few names, you can filter down the entire dashboard by hostname. You can even filter the dashboard by names that don’t exist in your DNS records so you can explore traffic for misconfigured records resolvers are looking for that don’t exist.

Lastly, back up at the top, business and enterprise customers can see a breakdown of traffic by record type. If your domain has DNSSEC enabled, you’ll even see queries for DNSKEY records, meaning DNS resolvers that are DNSSEC-aware are asking Cloudflare for your DNSSEC keys to verify DNS records are untampered with.

If you like what you’re seeing and wish you could use the data to piece together your own dashboards? You can. We’ve developed a Grafana plugin you can use to build your own dashboards and an API you can use to create your own tools. Plus the API gives you some added neat information like the distribution of queries over IPv6 vs IPv4 and UDP vs TCP, as well as the distribution of answers by query and response size. (DNS people rejoice -- now you can finally see if there’s a difference in query size between IPv6 and IPv4, and what is the breaking point where Cloudflare DNS answers switch from UDP to TCP.)

What about DNS Firewall and CNAME setup?

DNS Firewall customers, you’re next! Stay tuned! We have a really cool dashboard coming up for you that shows latency and errors on a per origin basis. We can’t wait for you to see it. In the meantime, you can see all your data through the Grafana plugin and API.

If you use Cloudflare but don’t see traffic in the DNS analytics dashboard, it’s because you use Cloudflare through a CNAME setup where your DNS is hosted outside of Cloudflare. Your DNS isn’t on us, so we can’t give you cool DNS analytics. If you’re interested in moving your DNS over to Cloudflare to try out DNS analytics and the newly announced Cloudflare Load Balancing, give our team a holler, they will swap your setup for you.

Let’s get started!

We are excited for you to get started digging into your DNS traffic. Let us know what you think by sending your feedback to dns-analytics-feedback@cloudflare.com. Cloudflare DNS team eagerly awaits your feedback :). Now go on vacation, we’ll look after your DNS!

Relying on a single wireless router to provide internet in every room of the home is like expecting a single light bulb to illuminate the entire house. It’s physics - WiFi radio waves don’t travel through walls or objects easily. The eero Home WiFi System is a new take on home connectivity, bucking the trend of one high-powered device in the center of the home. Instead, eero uses multiple access points that talk to each other via our proprietary mesh technology -- TrueMesh -- to spread coverage and a high throughput connection throughout a home.

eero’s hardware - its distributed access point system - solves the problem of spreading a consistent, stable WiFi signal in a home. But hardware is only part of the puzzle. On the backend of eero’s technology, we face different challenges: how do we build a highly available, high performance infrastructure that’s able to communicate with each eero device? We’ve discussed parts of our architecture previously, but we haven’t yet explored into how we use Cloudflare to eliminate one “single-point-of-failure” in our architecture.

How eeros interact with the cloud

eero devices are in near constant communication with our cloud. They send device diagnostic data to the cloud as well as data to support certain features in our mobile application. An example of this is when we added the ability for eero users to see real time data usage by device. While most products stream this data from a local server, both our user experience and security models rely on relaying that data through our cloud application to aggregate, analyze, and prevent direct local misuse from an unsecured web server.

In addition to streaming data to our cloud, eero devices need to continuously understand whether they have access to the internet. This specific architecture resulted in some special thinking.

First, we needed to ensure eero devices had a secondary source of truth (other than our cloud) to check if they could reach the internet. We originally thought about checking common resources like google.com and twitter.com, however we felt that had negative privacy implications since we didn’t own both sides of that request and connection. We wanted to prevent third-party sites from counting or analyzing eero installations to preserve privacy.

Second, we wanted to make sure that this secondary check was truly “out of band” of our current cloud infrastructure. Since the internet itself is highly distributed, we needed a service that mimicked that and was resilient to swaths of the internet being unreachable.

What Cloudflare allows us to do

Enter Cloudflare. Cloudflare provides CDN, security, and high availability, all in one tool. By storing a small file in Amazon S3, with Cloudflare at that domain, we’re able to serve a secondary internet check to hundreds of thousands of devices. Regardless of whether that connection can reach Amazon S3 and regardless of the health of the internet at-large, we have confidence Cloudflare will deliver the file.

We’re essentially using Cloudflare as a cloud canary. Since Cloudflare exposes a robust API, we’re able to track eero requests to this secondary internet check to create alerts when eeros are not able to resolve our cloud. And, as a bonus, since the file is 99.97% of the time served from Cloudflare’s edge, we can serve hundreds of gigabytes of the same tiny file and not spend anything with S3 to do it.

Cloudflare enables eero to provide an “out of band,” high availability, and low cost method to test internet availability from every eero device. We’re excited to help Cloudflare think about the specific needs of Internet of Things products, and continue to build out features that support scale across millions of devices. To learn more about how eero is solving connectivity in the home visit us at www.eero.com.

In a traditional TLS handshake, the client authenticates the server, and the server doesn’t know too much about the client. However, starting now, Cloudflare is offering enterprise customers TLS with client authentication, meaning that the server additionally authenticates that the client connecting to it is authorized to connect.

TLS Client Authentication is useful in cases where a server is keeping track of hundreds of thousands or millions of clients, as in IoT, or in a mobile app with millions of installs exchanging secure information. For example, an IoT company can issue a unique client certificate per device, and then limit connections to their IoT infrastructure to only their devices by blocking connections where the client doesn’t present a certificate signed by the company’s certificate authority.

Or in the case of a mobile banking app, where the bank wants to ensure customers’ secure financial data doesn’t get stolen by bots spoofing their mobile app, they can issue a unique certificate to every app install and in the TLS handshake validate requests are coming from their mobile app. Client authentication is also useful for VPNs, enterprise networks or staging sites, where corporations and developers need to lock down connections to only laptops and phones owned by their employees and teammates.

You may be thinking - don’t we have API keys for that? But client certificates offer a layer of security that API keys cannot provide. If an API key gets compromised mid-connection, it can be reused to fire its own valid, trusted requests to the backend infrastructure. However, the private key of the client certificate is used to create a digital signature in every TLS connection, and so even if the certificate is sniffed mid-connection, new requests can’t be instantiated with it.

Handshakes With TLS Client Auth

In a handshake with TLS Client Authentication, the server expects the client to present a certificate, and sends the client a client certificate request with the server hello. Then in the key exchange in the next trip to the server, the client also sends its client certificate. The client certificate is then used to sign the TLS handshake and the digital signature is sent to the server for verification. You can see the whole handshake here:

TLS Client Authentication On The Edge

TLS Client Authentication can be CPU intensive to implement - it’s an additional cryptographic operation on every request. And if there’s a flood of invalid traffic, each request in that traffic flood kicks off a verification step. Companies can move the TLS client authentication to Cloudflare’s edge to offload the expensive verification.

If we are performing TLS Client Authentication for a company, the company sends us the root certificate(s) we should validate the client certificates against. Then the company can set TLS Client Authentication to one of two modes: enforce mode returns a 403 and optional custom JSON or HTML when the client certificate is invalid, and report mode forwards all requests to the origin, even if the certificate is invalid. Cloudflare will send a header including the status of the certificate (none, valid, invalid) and the certificate Subject Key Identifier (SKI) to the origin. For companies that use the client certificate for identification, Cloudflare can also forward any field of the client certificate as a custom header.

We have TLS Client Auth set up on a test domain for you to try, at auth.pizza.

If you curl auth.pizza, you’ll get back a 403 and a custom JSON error, telling you that you are not authorized.

curl https://auth.pizza -H 'Accept: application/json'  

However, if you download this certificate: pizza.pem and curl the domain again using that certificate, you will be authenticated.

curl https://auth.pizza -H 'Accept: application/json' --cert pizza.pem  

Note that this demo is on the full domain, auth.pizza, but you can also set client auth on a subdomain, such as client.auth.pizza.

Get Started

To use TLS client authentication, you must first set up PKI (Public Key Infrastructure) infrastructure to issue client certificates. If you are interested in running TLS client authentication but don’t have PKI infrastructure set up to issue client certificates, we have open sourced our PKI for you to use. Here is great documentation by our friends at CoreOS on how to use cfssl to issue client certificates. If you prefer not to run your own CA and rely on an established certificate authority, we have partnered with a few certificate authorities who can provide the client certificates for you.

If you are an enterprise customer and would like to get started using TLS client authentication with Cloudflare, reach out to your account team and we’ll help you get setup. If you are not yet an enterprise customer but are interested in trying out TLS client authentication, get in touch.

Within the next year, we’ll be adding TLS client authentication support for all Cloudflare plans. After all, using encryption to make the web more trusted is what we’re about. Stay tuned.

In October, we wrote about a 1.75M rps DDoS attack we mitigated on our network, launched by 52,467 unique IP’s, mostly hacked CCTV cameras.

We continued to see more IoT devices in DDoS attacks, and so we started to put together a security solution to protect the devices from becoming part of the botnet in the first place. Today we’re announcing it: Cloudflare Orbit.

PC-era security doesn’t work in IoT-era computing

As we talked to IoT companies, over and over again we heard the same thing. In the consumer electronics space, IoT manufacturers were telling us that they were shipping patches to their devices, but their end users didn’t always download and install them. (Reserve your judgment, how many times have you pressed ignore when your phone asked you to update its operating system?) In the industrial control, medical and automotive spaces, where devices are used in life-critical functions, we heard a different story. Even if someone wanted to apply a patch, it just wasn’t that easy. For example, even if the manager of a nuclear power plant wants to update software on their thermostats, shutting down operations long enough to do that means the update has to be scheduled.

This is if the device can be patched at all - most devices are patchable, but up to a point. When Jeep was shown to be vulnerable to remote code execution, Chrysler had to recall 1.4 million vehicles.

This model where the end user is responsible for the security of the overall device is a relic of the PC age, where users knew and understood that their computers could have vulnerabilities, and as their software vendors released patches ––on so called “Patch Tuesdays,” for example––users knew to go and download them.

PC security does not work for IoT. Consumers do not share that similar understanding that they need to update their toasters, lightbulbs and cars, because they’ve never needed to in the past. And since we will never write perfect code, we need a better way of securing devices without waiting for users to do it for us.

Introducing Orbit

Thinking about this challenge, we were reminded of when the ShellShock vulnerability was discovered, Cloudflare was able to automatically keep the requests that would trigger the vulnerability from reaching our customers. With Orbit, Cloudflare can do the same thing, only for devices. For example, when Jeep was shown to be vulnerable, instead of recalling 1.4 million vehicles, Fiat Chrysler could have patched the bug in all the vehicles with just a simple rule in Cloudflare's firewall restricting access to the vulnerable DBUS service listening on port 6667 of every Jeep.

Orbit sits one layer before the device and provides a shield of security, so even if the device is running past its operating system’s expiration date, Cloudflare protects it from exploits. And while devices may be seldom patched, the Cloudflare security team is shipping code every day, adding new firewall rules to Cloudflare’s edge. Think of it like changing IoT to I*oT — devices can still access the Internet, but only after passing through Cloudflare where malicious requests can be filtered.

For the last year, Cloudflare has been working with a number of IoT vendors to develop Orbit. Already more than 120 million IoT devices are safer behind Cloudflare’s network. Lockitron is one of the IoT companies using Cloudflare. “Keeping our products and customers secure is our primary concern,” says Paul Gerhardt, co-founder of Lockitron. “Cloudflare provides an extra layer of security that allows us to keep our devices continually updated and ahead of any vulnerabilities.”

Instead of writing and shipping a patch, IoT companies can write logic on Cloudflare’s edge, and write their own firewall rules to run on Cloudflare, and it updates the Cloudflare Orbit layer immediately, for all of their devices, without their users ever being so much as nudged to install something.

Plus, with requests going through Cloudflare, Cloudflare can compress transmitted data and speed up traffic, meaning less time is spent waiting on open connections and more time left in battery.

An Extra Layer of Authentication

A common challenge we heard from IoT device manufacturers was how to authenticate devices and know which connecting clients were authorized company devices, and which were bots only pretending to be. Starting today, Cloudflare now offers Enterprise domains TLS Client Authentication, a TLS handshake where the client authenticates the server’s certificate (as with any TLS handshake) and also the client has a certificate that the server authenticates.

Some IoT vendors already implement their own Client Authentication, but do so at the same origin servers that handle the rest of their IoT infrastructure. Not only is this computationally expensive, but any invalid traffic flood causes a burden on the whole server.

With Client Authentication on Cloudflare, Cloudflare’s edge handles the load of the TLS handshakes, validating the device client certificates and only sending the IoT infrastructure traffic from authorized devices.

What People Are Saying

“This approach of adding security to the network is extremely important for industrial manufacturers. Being able to patch vulnerabilities from the network rather than at the device level is a major shift in the way we secure IoT devices, and one that is completely necessary.” — Sam Cece, CEO of Swift Sensors, an industrial IoT company.

“Car controllers are IoT devices. Karamba Security hardens these devices and prevents cyberattacks with zero false positives to maintain driver and passenger safety. We view Cloudflare’s Orbit as a complementary solution that enables secure connectivity between the cars’ hardened controllers and the car company’s data center for trusted, over-the-air updates.” — Ami Dotan, CEO of Karamba Security, which is building secure platforms for smart automobiles.

“We are at the beginning of a new era in which a vast number of devices will be connecting to the Internet and security will play a critical role in the successful roll-out and adoption of IoT devices. Cloudflare’s Orbit adds another layer of defense that compliments other security measures such as strong hardware-based device security and helps ensure a safer Internet of Things." — Quinn Li, VP and global head of Qualcomm Ventures, the investment arm of Qualcomm Incorporated, the leading supplier of components for IoT devices.

"IoT devices create a distinct security challenge both because of the inability of most end users to update their software, as well as the cost that manufacturers bear if they release an update that bricks devices. This is even worse for legacy devices, many of which are effectively unpatchable. Cloudflare's Orbit provides a unique approach to help with these challenges, by deploying a defensive layer in the network where security updates can be safely made without end-user intervention or on-device changes." — Michael Freedman, professor of computer science at Princeton University and CTO of Timescale.

Get Started

Orbit is available now to all IoT device companies. To learn more or get started, visit the site, or get in touch. We’re excited to hear from you.