We just sent notice we are terminating service for 8chan. There comes a time when enough is enough. But this isn't the end. We need to have a broader conversation about addressing the root causes of hate online.
We just blocked Kiwifarms. The threats on the site escalated enough in the last 48 hours that, in spite of proactively working with law enforcement, it became enough of an imminent emergency we could no longer wait for them to act. Details of our decision:
There’s a lot of buzz right now about a “massive DDoS attack” targeting the US, complete with scary-looking graphs (see Tweet below). While it makes for a good headline in these already dramatic times, it’s not accurate. The reality is far more boring. 1/X
One of the smartest decisions we made at
@Cloudflare
was recognizing that the primary purpose of our blog was attracting employees, not attracting customers.
I’m told mine is a contrarian view on the events of the last few days, so here goes…
Contrary to what
@kevinroose
and others have written, Microsoft was not a winner of the events of the last few days around
#OpenAI
. They were in a much better place on Friday morning last week…
@paulg
Oh Paul, you know better. I had to borrow money from my mom to pay my taxes when we were starting Cloudflare. But I certainly came from a relatively privileged background, and so did the AirBnB founders. It’s hard to take risks if you don’t have a safety net.
#bereal
Trying to come up with analogy to explain
#xzbackdoor
to my mom. Best I’ve come up with so far:
Imagine a grizzled, tired, old oil pipeline maintainer was approached by a plucky young kid who was new to the small town the oil pipeline passed through. Slowly, over years, the kid…
@Austen
We fired ~40 sales people out of over 1,500 in our go to market org. That’s a normal quarter. When we’re doing performance management right, we can often tell within 3 months or less of a sales hire, even during the holidays, whether they’re going to be successful or not. Sadly,…
@parastang95
Mistakes happen. The root problem was we didn’t have systems in place to keep them from causing a widespread issue. That’s a problem of leadership that I am more responsible for than the engineer who made the typo.
Details on how we caused an 23 min outage for~50% of
@Cloudflare
's network today. The root cause was a typo in a router configuration on our private backbone. We've applied safeguards to ensure a mistake like this will not cause problems in the future.
Just sent the last
@Cloudflare
employment offer of 2021. I still personally send all the offers out because nothing is more important than hiring. Some numbers: We received more than 200,000 applications. We extended 1,455 offers. And we had a 92% offer acceptance rate.
Nothing we're seeing related to the Facebook services outage suggests it was an attack. Most likely explanation is that the company's Internet routes (BGP) were withdrawn by mistake during maintenance.
#hugops
We’ve made the determination that
#Log4J
is so bad we’re going to try and roll out at least some protection for all
@Cloudflare
customers by default, even free customers who do not have our WAF. Working on how to do that safely now.
Except T-Mobile, which is having a bad day almost certainly entirely of their own team’s making. So, please,
#hugops
. And don’t worry, this is one thing that does not need to get added to the list of craziness that has been 2020. 8/8
Proud of our whole team for creating 1.1.1.1, the Internet’s fastest, privacy-first DNS resolver. It’s
@Cloudflare
’s first consumer product. And, if you’re wondering whose dopey idea it was to launch on April Fools (and Easter), look no further than me.
When I read this story by
@troyhunt
about how
@Cloudflare
and
@Azure
misfired to cause a huge bill I felt terrible. I reached out to
@scottgu
and proposed we split Troy's cost. Scott immediately agreed. Great to support our mutual customers!
It starts with T-Mobile. They were making some changes to their network configurations today. Unfortunately, it went badly. The result has been for around the last 6 hours a series of cascading failures for their users, impacting both their voice and data networks. 2/X
Shhh… 🤫
@Cloudflare
Registrar just quietly rolled out support for the following TLDs:
.app
.boo
.channel
.dad
.day
.dev
.esq
.foo
.how
.mov
.new
.nexus
.page
.phd
.prof
.rsvp
.soy
At
@Cloudflare
, we understand the Russian cyber attack capabilities and stand prepared to defend our clients against any cyber retaliation that results from global sanctions.
The collective hallucination that this is about “taxes” rather than finding somewhere talented junior engineers can afford their own apartment and senior engineering managers can afford a nice house—because sensible housing policies—will be what kills SF.
So now people are looking around for an explanation and they stumble across sites like the Arbor Networks attack map. It looks terrifying today! Thing is, it always looks terrifying. It’s a marketing gimmick put up to sell DDoS mitigation services so that’s not surprising. 4/X
Earliest evidence we’ve found so far of
#Log4J
exploit is 2021-12-01 04:36:50 UTC. That suggests it was in the wild at least 9 days before publicly disclosed. However, don’t see evidence of mass exploitation until after public disclosure.
We are aware that
@Okta
may have been compromised. There is no evidence that Cloudflare has been compromised. Okta is merely an identity provider for Cloudflare. Thankfully, we have multiple layers of security beyond Okta, and would never consider them to be a standalone option.
Okta compromised… again. Here’s how
@Cloudflare
, even though we were (again) targeted, was able to mitigate the attack. And some best security practice suggestions for
@okta
and their customers.
Just sent out the last
@Cloudflare
employment offers of 2022. Received roughly 400,000 applications, up 49% over 2021. Of those, we interviewed 15,805 candidates to ultimately hire 1,418. 37% of the hires were designated fully remote, up from 14% in 2021. (1/2)
Dear
@antoniocostapm
, when your team promises to work with companies in exchange for moving significant jobs to Portugal and shows laws to support those promises; then your bureaucrats refuse to follow those laws and promises — is that ok? I feel lied to. Cc:
@paddycosgrave
Just published an initial post mortem on the incident that impacted many
@Cloudflare
customers use of our dashboard and APIs. Lots of lessons.
#CodeOrange
Wow!
@Cloudflare
's 1.1.1.1 () has now passed handling more than 1 trillion requests per day. Not bad for a project we launched on April Fools Day a little over four years ago.
Lots of reports of Russian censorship of Western media. We are seeing evidence of that. But, generally, consumption of Western media in Russia is up more than 3x in the last month — in spite of censorship.
#truthfindsaway
Seeing a marked increase in cyberattacks this evening. Combined with the deeply disturbing headlines, fear the world just turned up the crazy dial another notch. We’re ready online at
@Cloudflare
. But… worried for the world.
From
@Cloudflare
’s vantage point, we can see a number of things that show there is no massive DDoS attack. First, traffic from WARP to supposedly impacted services is normal and has no increase in errors. 5/X
When
@Cloudflare
started we didn't want to build a DNS service, so we reached out to Dyn & UltraDNS about becoming a customer. Both blew us off because we were "too small." So we built our own. Good lesson about serving all customers, no matter how small.
We are resetting the
@Okta
credentials of any employees who’ve changed their passwords in the last 4 months, out of abundance of caution. We’ve confirmed no compromise. Okta is one layer of security. Given they may have an issue we’re evaluating alternatives for that layer.
That caused a lot of T-Mobile users to complain on Twitter and other forums that they weren’t able to reach popular services. Then services like Down Detector scraps Twitter and report those services as being offline. 3/X
This is the sad story about what happened to my friend and
@Cloudflare
’s brilliant third cofounder Lee Holloway (
@icqheretic
). We’d never have been able to pull off what we did without him on the team. I miss him every day.
Just sent the last
@Cloudflare
offer of 2023. What an incredible year!
* 1,162,526 applicants
* Offer extended to less than 0.1%
* 90%+ offer acceptance rate
Companies are just collections of people; our team is incredible and continues to get better and better! Happy New Year!
Second, there is no spike in traffic to any of the major Internet Exchanges, which you do see during actual DDoS attacks and definitely would during one allegedly this disruptive. 6/X
Remember reading when Google's 8.8.8.8 crossed through over 1.2 trillion requests per day back in 2018 and thinking: that's unfathomable. Well, 1.1.1.1 is now handling more than 1.3 trillion requests per day. Still feels unfathomable.
As a precaution, we’ve removed all
@Cloudflare
customer cryptographic material from servers in Ukraine. We continue to serve traffic there for Ukrainian users, for now, via our
#Keyless
technology.
Finally, our team know the network operators at nearly all the other major Internet services and platforms and none of them are reporting anything anomalous. 7/X
In other words: sometimes owning a call option on an asset is better for multiple reasons than owning the asset itself. Last week Microsoft roughly owned a call option on OpenAI. Today, at best, they own some fraction of the asset itself.
We had an issue that impacted some portions of the
@Cloudflare
network. It appears that a router in Atlanta had an error that caused bad routes across our backbone. That resulted in misrouted traffic to PoPs that connect to our backbone. 1/2
The Juniper Networks news reminded me that when
@Zatlyn
and I were first starting
@Cloudflare
my initial prediction on our “exit” was selling to them for $250M. Michelle disagreed. She turned out to be a much better predictor of the future.
Here's what went wrong on the Internet earlier today causing
@Cloudflare
and several other networks to be unreachable for many users. It's time for providers like
@Verizon
to be held responsible for not filtering BGP routes and implementing RPKI.
The teams at
@verizon
and
@noction
should be incredibly embarrassed at their failings this morning which impacted
@Cloudflare
and other large chunks of the Internet. It’s absurd BGP is so fragile. It’s more absurd Verizon would blindly accept routes without basic filters.
We’ve seen reports of service outages across the Internet. Confirmed
@Cloudflare
’s services all operational. No uptick in attacks. We are seeing local drops in traffic from some upstream providers. Not yet clear if they’re related or not. All indications: not a Cloudflare issue.
The first five
@Cloudflare
employees —
@zatlyn
,
@icqheretic
,
@IanPye
,
@mtourne
, and I — officially started on January 4, 2010. We were above a nail salon in Palo Alto at 542 Emerson. Our first task was to assemble the BBQ for the deck. It's been quite a decade.
Every ~8 years there’s a 10x improvement in how computing is provisioned. Bare metal > VMs > Managed VMs/containers. We’re due for the next step-function improvement and at
@Cloudflare
we’re convinced it’ll be built around a technology called Isolates:
Wow!
@Cloudflare
's 1.1.1.1 just crossed handling 300 billion queries per day. Still well shy of the trillion+ that Google's 8.8.8.8 handles, but growing fast!
@_FiveM
@Cloudflare
Completely unacceptable. I’ve ordered the account be restored. Called on team to investigate why an automated system took such draconian action without any warning. It may be you’re doing something that breaks plan limits, but we need to have more nuanced solutions. Apologies.
AWS’s bandwidth charges are egregious. Their wholesale cost in AWS-East is likely less than $200/Gbps/mo. That equates to a 10,000%+ markup.
#nevertrustamazon
My take on this is a bit different than others: it’s really hard to make money as a dev tools company unless you find a way to sell storage, compute, and bandwidth. So clearly they needed to build these hooks.
Proud of the
@Cloudflare
team’s work defeating yet another patent troll. As part of
#ProjectJengo
we took this one all the way to trial and not only prevailed for ourselves but got the troll’s patent cancelled. This is a win not just for us but for the industry as a whole.…
The entire problem with Carta is they’re the Facebook of B2B. From day one the whole premise was “if we hoover up private company’s cap table information we can eventually build the world’s biggest secondary market.” That’s the only way to justify their multiple & valuation. 1/2
Incredible that
@Cloudflare
stopped the largest DDoS attack in history and it was just another day at the office. I wasn't even aware of the scale until I read this post.
@Sativa888
Pretty sure a major DDoS attack would be amazing for us given we’re one of the only companies that could stop it.
Good lesson: when you hear hoof beats, think 🐎 not 🦓.
Years ago, I remember reading that Google's 8.8.8.8 handled 2 trillion queries per day and being blown away. Today, just 3 1/2 years after launch, 1.1.1.1 is a quarter of the way to that same milestone. And >15% of its queries are encrypted!
#progress
All threat research groups have cool names. Most of them are full of BS.
@Cloudflare
is planning to launch a threat research group that's no full of BS. But still need a cool name. Any suggestions? If we choose your suggestion I'll make sure you get lots of branded swag.
Proud of the role
@Cloudflare
played in ensuring the Internet stayed online in 2020. We stand ready for whatever 2021 brings (but, for the record, will be totally cool if it’s less eventful).
Thrilled to partner with NVIDIA to bring AI to the edge!!
@Cloudflare
Workers is the largest, fastest, most used edge computing working. With NVIDIA's hardware running at our edge we open a whole new class of applications for developers.
#DeveloperWeek
Excited to work with
@PalantirTech
to help customers understand what they're spending on the cloud and how they can optimize those costs using
@Cloudflare
Workers.
AI inference will run on end devices or at the edge of the network as close as possible to where end users are. Sending inference tasks back to a centralized cloud will look increasingly anachronistic.
Many people know about the wall of lava lamps in San Francisco
@Cloudflare
uses to generate random numbers. Fewer know about the wall of double pendulums in London we use here.
#entropy
We isolated the Atlanta router and shut down our backbone, routing traffic across transit providers instead. There was some congestion that caused slow performance on some links as the logging caught up. Everything is restored now and we're looking into the root cause. 2/2
Writing the
@Cloudflare
S-1 was a fun process for me, perhaps a less fun for the (terrific) bankers we worked with. The number of times I said “Maybe should just start over with a blank page” inspired the celebratory cake they made for us.
Early on, I remember other startup founders being so puzzled why
@Cloudflare
didn’t have perfect tooling and other processes fully figured out like they did. I was embarrassed; we just hadn’t had time. Funny thing: none of those perfectly tooled startups are around anymore. 🤔
Tomorrow begins Platform Week at
@Cloudflare
. We'll feature all the ways we're extending Cloudflare Workers to be the best serverless development platform. R2's beta will open up. But… that's not even close to the only exciting announcement for the week!
#staytuned
Sometimes you and your partner’s timing doesn’t line up precisely and you compromise by sending a press release out in the middle of the night. All good. Suffice it to say: this is just an early taste of much, much more to come tomorrow.
At
@Cloudflare
, we need to manage a lot of services running on a lot of servers distributed across more than 200 cities worldwide. Here's an inside view on one of the tools we use to do that:
@HashiCorp
Nomad.
Just finished sending the final batch of
@Cloudflare
job offers for 2020. It's been incredible to see the caliber of the people applying to join our team over the last year. The seeds of what we will become tomorrow are being planted today. (And they're mighty impressive seeds.)
So far the Ukrainian Internet continues to function. Seeing a 50% increase in traffic, day-over-day, likely as people are scouring the news online as they wake up.
Ha! AWS required us to remove the term “multi-cloud” from our materials at one of their conferences. Someone hasn’t learned the anti-trust lesson from Microsoft yet. They will.