27 ms·

Cloudflare's AI Psychosis

120 points by gregzeng95 2mo ago

MarkWayneNewton 2mo ago
[flagged]
orf 2mo ago
> First of all tons of more outages than ever, remember that React useEffect fkup [0]? Complete insane that this would happen at an infra company that runs a third or so of the web. Bugs happen all the time. They roughly increase with scale, not decrease. There’s an argument to be made about better testing, but this specific bug seems like a perfect one to slip through: multiple services, hard to spot at code review, involves JS/frontend, invisible at low traffic (test/UT envs). So IMO it’s not completely insane. Is the implication that Cloudflare should have no bugs whatsoever?
- minraws 2mo ago
  Bugs happen all the time yes, outages don't. If outages are increasing with scale you get 1 or maybe 2 free passes. After that you either have in-ept Engineering or just in-ept leadership. I used to be all in on CF a while ago, now I am moving off them almost entirely. Same issue with GitHub, I can understand if you can't build for the scale when you couldn't predict it but if after over 12-18 months things don't seem to be improving what are you even doing? I honestly think all of these companies are deluded if they think people will stick around with all these weekly outage events. I have a homelab server I have had 2 outages in 1 year because my shitty ISP went down. Still at 99.9% uptime, I have done nothing special. I now have backup internet as well. Is it big? Nope but it doesn't need to be cf scale. And scale is the reason to use these services why would I use cloudflare if a homelab would have been enough? If they aren't designing and scaling their systems to handle this scale they might as well close shop, someone else might do it better. As a infra/dev person who does his own thing on the side, I might be the most impacted by these outages, so I might be coming off as harsh. But they cost me both time/money and headache in extra development work. Imagine prod deploys are down for 2 days why? Because GitHub actions keep failing... Oh serving new OTA updates broke? Why? dig into the code.. go oncall with users instead of doing work, realize it's a CF outage and the writes failed. (Feel the tears streaming down your face). If I have to waste dev time, with AI and me together we could self host it with higher reliability with significantly cheaper costs at this point even at fairly decent scale. I think any infra company that has more than 1 outage a year is already not worth investing in. But more than 3 and you might be better off self hosting, even in this ram apocalypse. If all people in SF are this unserious about reliability (which hasn't been my experience but HN seems especially open to break the prod if you have to) Then well software companies really do deserve to be replaced by AI.
  - orf 2mo ago
    To put it bluntly: that is complete nonsense. AWS has had more than 1 outage a year - is it not worth investing in? Are they not serious? Outages are just a specific kind of bug, often surfaced by the interactions of several discrete bugs. Saying “you’re not serious if you have more than 1 bug a year” is silly.
    - minraws 2mo ago
      This might be rude but this is the reason we have shitty software everywhere. The "there can always be a bug, get over it mindset" is the reason software sucks, everything needs a dozen patches to even work and no one can have any peace in this business. I honestly don't even want to debate this because I see no point, as honestly the side that would have said outages shouldn't happen is dead buried and out of a job at this point. But, > Saying “you’re not serious if you have more than 1 bug a year” is silly. > Outages are just a specific kind of bug, often surfaced by the interactions of several discrete bugs. Several bugs or 1 bug? you decide and tell me, when you have made up your mind. Outages happen as a result of the system's design being shoddy in the first place, and an attitude where bugs shipped are acceptable because it's only a single component, rest of the system shouldn't fail. It won't go down was the entire thing with microservices and "the cloud" I could link some blogs and brochures if you want me to. I am honestly over the moon with these conversations on HN, really proves the point why AI is rightly replacing engineers in software, because there wasn't any engineering to begin with. Imagine if a civil engineer said bridges fall it's fine for a few bridges to fall because bugs happen, and honestly bridge falling is just a kind of bug where several discrete bugs happened at once. I am certain there is some room to argue about it, but what is clear is that if it was happening every single month you would have stopping using bridges unless absolutely necessary. I am not sure if there is ever a reply for this honestly, I honestly don't even blame the people just the culture at this point. If that's what software is to some, "something that can fail at any time" and cloud's selling point is just fewer "switches to flip" I am sure I can't change that. But I won't be deluded into thinking that software outages are just an everyday "bug" and it's fine to have several outages a year, it is the normal mode of operation in-fact no harm done.
spicyusername 2mo ago
The fact is that every major company is AI accelerating their development, and soon, it will just be every company. We're already past the point of getting our hackles raised when we suspect something is AI written. Everything is or soon will be AI written. So I don't think it's that useful to blame bad product design on AI. Bad product design is just bad product design. Bad writing is just bad writing. Bad graphic design is just bad graphic design. The industry needs to get over this hump of pretending we're not all going to adopt AI for every imaginable thing. The cat's out of the bag, for better or for worse.
- decide1000 2mo ago
  It's not just the AI. The post mentions poor choices because of a product management mindset instead of engineers leading the PMs.
  - kylecazar 2mo ago
    Only part of the article I disagree with a bit. I'm willing to bet most infrastructure PM's at CF are technical people with engineering backgrounds. I think what's more likely the culprit is the desperation to claim ground. They (and many others) want to rush out an answer to all of AI's new challenges out of fear of being beaten to the punch. It usually leads to a complex, almost incoherent product suite.
- GPerson 2mo ago
  I believe protest and criticism are corrective processes, and we shouldn’t give up.
  - orliesaurus 2mo ago
    Thank you! I am a CF customer, I pay for the service, I just want it to get better and better!
- xpct 2mo ago
  Well, let's not normalize bad product design. Worse, let's not normalize pushing out broken projects that never get maintained. It's noise and breaks trust. I'd say the maintenance part is especially bad now, I hold very little trust that people will maintain the projects they share.
decide1000 2mo ago
Great read. Exact pinpointing of my feelings about CF lately.
jbrooks84 2mo ago
Accurate
rfgplk 2mo ago
[dead]
daishi55 2mo ago
I don’t understand the impulse to whine and complain about the aesthetics of a big company. > a little more cringey and clique Why on earth do I care if someone thinks Cloudflare is cringe? What is interesting about that? They are following the market like everyone else. > Too many ways to do the same thing, none of them great…They got D1 (SQLite serverless), Durable Objects with their own SQLite, KV, R2, Queues, and Hyperdrive to speed up external Postgres or MySQL. What does this have to do with AI psychosis? I thought that was the thesis of this article? As I said, it’s just directionless complaining.
- gegtik 2mo ago
  i read the article as caring about the values, principles, and goals of the human beings driving it. if you are looking exclusively at compiled bits and bytes and/or quarterly statements then I agree, human/societal judgment seems irrelevant.
  - daishi55 2mo ago
    What values and principles are supposed to be preventing cloudflare, the internet and technology company, from exploring hot new internet markets and technologies?
    - orliesaurus 2mo ago
      the constant "fights" with Vercel instead of delivering shareholder value by refining the product
      - esseph 2mo ago
        I'm not even sure if I would know who Vercel is if I were at Cloudflare. The scale between the two is staggeringly different.
        Plont 2mo ago
        That makes it worse, not better.
jeresuikkila 2mo ago
I'm also curious about the question posted at the end: where did the infrastructure nerds go?
- dgellow 2mo ago
  Likely to the AI labs and hyperscalers? Given the insane growth and level of expenditure I assume it’s a fun place where to work (assuming you’re ok contributing actively to the AI industry)
reisse 2mo ago
> There was a time Cloudflare just made the internet better There wasn't. Cloudflare is a cancer grown too big. And it was always positioned to become one, the middleman between users and the Internet. It is already painful to browse web sometimes using the non-"standard" tools (that is, not a Chrome with Google account signed in, not an EU/US residential IP). What if tomorrow Cloudflare checks will require attested and signed browser binaries?
- gruez 2mo ago
  >It is already painful to browse web sometimes using the non-"standard" tools (that is, not a Chrome with Google account signed in, not an EU/US residential IP). What if tomorrow Cloudflare checks will require attested and signed browser binaries? If it's painful, that's not because of cloudflare. You can get past turnstile challenges using tor browser in a VM. You still might be blocked because of policies set by site owners, but that can hardly be blamed on cloudflare.
  - dingaling 2mo ago
    > but that can hardly be blamed on cloudflare They are installing the gates with configurable locks that the site owners use. They are absolutely to blame.
    - gruez 2mo ago
      Should nginx be blamed too, because that also allows site owners to ban tor/VPN users?
      - 398642258909 2mo ago
        Never had the issue of nginx gatekeeping, but Cloudflare's shitty captchas constantly block the access, since my browser settings don't adhere to their MitM-mandated rules.
  - anon5868 2mo ago
    >in a VM Are you saying it's possible to detect VMs in the browser without using the WebGL vendor? Mind sharing some resources?
hypfer 2mo ago
Why would they do anything else? They own a huge chunk of the internet now - with many, many companies having deeply integrated CF into their own stuff. Switching costs are so high, from a business standpoint, investing in technical excellence would be wasted money. They're just a corp like everyone else. Gravity also applies to them. __ Don't take this as "they're right to do so". Take this as "You should never have started integrating their products this deeply. This was inevitable and obvious." Using CF always was trading short-term wins for long-term losses. We're now entering said long-term.
alberth 2mo ago
What’s missed from the post is that Cloudflare for many years now has felt that they are a “Cloud 2.0” company. And AWS/GCP/etc are “Cloud 1.0”. They believe that AI Agent will fundamentally change the internet, and it’s hard to fault them for seeing it that way when their own stats shows that bot/agents account for the majority of Internet traffic. As such, going all in on AI fits into their thesis of being “Cloud 2.0”.
latchkey 2mo ago
on the bright side, this rant doesn't sound ai generated.
- orliesaurus 2mo ago
  thank you, I actually only used it to find the specific sources that I linked at the bototm (through Google Search AI: i.e. "what was the URL of the postmortem for that useEffect bug")
- dude250711 2mo ago
  An organic rant, for some locally-produced even.
  - orliesaurus 2mo ago
    > dude250711 I feel like I know you :)
tosh 2mo ago
I think like with most large companies that still ship new stuff: it gets a bit more complex for customers to figure out what the good nuggets are and what to ignore for now same @ google, aws and so on (I usually rely on opinions of people i trust to find the 'javascript — the good parts' version of large offerings) the other extreme would be not to try and ship new stuff which is also risky difficult to find a balance and probably a good idea to over-index on momentum and new stuff while investing enough in hardening and improving the stuff that sticks
- tosh 2mo ago
  I also noticed the ui/ux of cloudflare got way better in the last weeks. That's a good sign. Usually ui/ux in large companies gets worse not better!
piterrro 2mo ago
The problem is, beside their size, they have not moat to compete in AI space. Their data centers are spread across the world in other companys DCs (btw do they own any DC actually from top to bottom?).
- dgellow 2mo ago
  Side question: does anyone have a moat in the AI industry? AI vendors for sure do not. Hyperscalers don’t seem to have one. Datacenter companies are in an industry that is more difficult to enter but don’t have a moat per se. Same for manufacturers of HBM and other hardware. NVIDIA is maybe the only player with an actual moat thanks to CUDA & co
  - piterrro 2mo ago
    Companies specialized in building DCs and AI labs specialized in niche models?
    - dgellow 2mo ago
      Actually, not DC builders themselves, but companies that manufacture fairly specific parts used by DCs, like the massive gas turbines they’ve been using. I’m not familiar with that part of the industry but would assume that has to be a business with a pretty big moat
gedy 2mo ago
A good point made in the post (that's not really specific to CF) is the problem with multiplying features. It's interesting that most of the extremely-hot-on-AI companies and leaders I know are hyped about 100x product dev, etc and want to throw tons of new assorted features out. This is not a good thing! I've been at companies with lots of staffing and velocity (pre-AI), and it is a huge foot-gun to think pumping out features is automatically a good thing. Most paying customers do not want this from SaaS type companies. Smaller companies especially should calm down and do one thing well, and unfortunately AI dev does not encourage that.
- esseph 2mo ago
  > Smaller companies especially should calm down and do one thing well They're going to get absolutely steamrolled unless it is a high risk critical industry.
  - gedy 2mo ago
    Maybe, but pretending to be Atlassian or whomever doesn't help either afaics
tokioyoyo 2mo ago
It sounds like a blog post in support of a slowdown for product releases. But the reality is there are multiple companies trying to capture the attention of the same market. Maybe 2 years ago it would’ve been acceptable to release the product internally, slowly add features, dogfood it, then open to public a year later. But now someone else will release the same product within a month, get sticky customers, and most likely won’t switch. It’s much easier and faster to release features, so once anyone sees a competition gaining some attention, they just copy the same feature. Which, i think, is fair. It’s easy to blame the org, but the market is extremely competitive right now. Kind of race to the bottom, except the hardware parts, which have different constraints right now.
- orliesaurus 2mo ago
  Hi, I wrote the post, I think blaming the org for releasing things that aren't engineered/created with "love" (including docs, examples, long-term planning and a roadmap of improvement that will follow-on), and suitable for production, without labeling them as beta, is in fact detrimental to the status of a company's image.
  - esseph 2mo ago
    So where do you stand on AWS that may release more than 25 new services a year?
    - orliesaurus 2mo ago
      I wouldnt be surprised if AWS buys Vercel (an AWS wrapper btw) and puts all the "cool new things" under the vercel cli.
  - tokioyoyo 2mo ago
    AWS/GCP/Azure are full of abandonware. I’m more familiar with the AWS-side of things, and most of them were/are built on top of existing core services. AWS’s/GCP’s image is fine. I’d say as long as CF’s core (CDN/DDoS mitigation/etc.) is stable, they should experiment. To put it this way, it took GH almost a year to release Stacked PRs to public. And it’s already being questioned if it’ll stand the test of time. Again, insanely competitive market, as everyone wants to sell shovels.
yodon 2mo ago
Cloudflare's UI is basically the web equivalent of the git CLI. Powerful, but the densest collection of UI anti-patterns known to man. Their "designers" obviously walked up hill both ways in the snow to school every day while being eaten by a walrus and feel anyone who doesn't enjoy a suitably opaque navigation scheme isn't worth acknowledging.
orliesaurus 2mo ago
Hey everyone, author here, I am very frustrated with the state of Cloudflare and I posted this blog the other day on my blog and here on HN (but generated no discussion), thanks for re-posting it, I am gonna start reading the comments and start replying
sarreph 2mo ago
I have to say: Yes, there is a lot of velocity from Cloudflare pushing out AI-adjacent features as of late. But also: What they've doing with Workers et al. has made them a platform for fully-fledged apps and deployments. I personally love the ecosystem and use it for all of my projects now. It's like the perfect blend of ease-of-use and DX of Heroku, and breadth of services of AWS / GCP. Well, maybe not quite AWS or GCP, but that's kind of the point -- they've created an opinionated system of "objects" that are all highly extensible to the point where in my opinion you can pretty much deploy anything you want on it. It took me a bit of time to learn the "Cloudflare way" of doing things, but once I started making on it I saw just how flexible (and low cost!) everything is.
aleda145 2mo ago
Hard disagree. Sure their docs are bad, and it's sometimes buggy. But to me cloudflare is actually innovating and trying to offer "new" infrastructure. Durable objects and workers are super cool. R2 has free egress, isn't it insane that they could pull that off? If what you want is a VM to run postgres then there are other offerings that would be much better. Not saying they aren't evil though, they probably are, but the infra stuff is cool.
- orliesaurus 2mo ago
  I disagree with your disagreement. I wrote the post and I am not blaming the innovation. I am blaming the chaos of innovation. Everything feels DISJOINTED. The innovation is useful, the delivery of the innovation is the pitfall here.
  - bestham 2mo ago
    Innovation is only linear in hindsight . Pure speculation on my part but once CF made some leaps in a particular area this enabled them to gather more data and shift direction to reach their over all goals by stifling the causes and the innovation that lead them there. New management comes in, requirements change and this is evident to us on the outside as disjointed behaviour. Companies need to make money from the internet for it to exist. Companies also need to behave and refrain from tragedy of the commons for the internet to keep existing as we know it.
- jokethrowaway 2mo ago
  They have cool tech but they can't do product, like most of big tech. Have you tried wrangler? holy Hire enough product owners and that's what happens. That's why they need to buy startups every once and then, to bring some good bacteria in their messed up corporate gut.
  - orliesaurus 2mo ago
    yep, wrangler is a monster, the idea is solid - it's like the gcloud or heroku toolbet - the execution isn't nearly as neat.
themgt 2mo ago
I view a lot of this as downstream from sort of the IaaS version of the "DRM problem" in cryptography. In the DRM problem you want to sell someone a Blu-Ray with a movie they can play, but also not allow them to copy the movie. So they get the data but can't have the data. In the IaaS problem you want to sell someone a piece of server(s) in a datacenter they can play with, but also not allow them to just directly use the hardware (because then they could have open standards and portability). So the product teams are required to work backwards from the necessity of wrapping compute in a thing you can sell as a product. This also explains nonsense like "durable objects", which may have some purpose but become the weekly podcast fad after getting hyped by cloud providers trying to find new ways to sell compute without selling compute.
nzeid 2mo ago
This is definitely as opinionated as an opinion piece can go. Worth mentioning I don't use CF anywhere or care about it... but my employer does. I can really only remember one catastrophic outage. It involved some part of CF that had no redundancy, and the incident was followed by a pretty whiny blog about a data center. That's pretty much it. There were other minor outages but I'm not aware that it affected business. Despite the disclaimer I think the author should explain precisely how "AI product mindset" or whatever is actually translating to bad outcomes.
- gexla 2mo ago
  Just skimmed the article, but the CF your employer used was probably around a lot longer than AI? There's parts of it that can bring everything down. Then there's parts of it that will bring down just what your team(s) put in a load of hard work to build on top of it.
  - nzeid 2mo ago
    The blog implies that CF's core business is suffering because of added investment in what the author describes as AI. See their useEffect bug as an example. EDIT: To be clear, my employer still uses CF and I don't see any evidence that it's gotten worse.
NicoJuicy 2mo ago
For an article that claims to know Cloudflare, I don't think he does. Cloudflare build from the bottom up. First the SDN ( Software Defined Network), then the products on top. The first one was logical: CDN + DDOS. All the rest continued to build on top of the SDN - Workers, D1, ... But once they have the full stack ( which happened arround D1), it was time to let others build on top of Cloudflare. That's the current stage ( Cloud 2.0), that was always the goal. It's even in their ticker ( NET ). In practise, AI could be a godsend. A total new foundation for software where the Cloudflare stack is a perfect product fit( disposable compute, where others will protect their compute with long running compute contracts). I can't blame them to try to seize the opportunity. Tbh. I feels like the author hasn't actually adjusted to how Cloudflare is building it's cloud ( eg. how powerfull Durable Objects is). PS. A big outage is a long time ago... ( we use them, I remember the issues those days and the post mortem, a lot became more stable as far as I noticed).
thecatapps 2mo ago
I remember being so excited about Cloudflare's releases that I'd check their blog every couple of weeks for what new thing they were cooking up. This was before I actually had to use them to serve production clients. When it came time to launch (2023?), the initial setup was using DNS/Cache, Workers, Workers KV, R2, D1, Durable Objects, Access, and Queues. My first hint was that the Typescript library to access their API was simply just wrong. API requests through the library would fail, complaining about missing fields or invalid types, even though the types said my construction was correct, and I received dismissive replies when raising it in their Discord. Then there were the D1 issues with random requests failing. Then there were the Durable Objects issues with syncing clients in our collaborative editor. Then there were the KV and (by extension) Access outages. Eventually it made sense to switch it all over to AWS. Today, I trust Cloudflare for DNS, Cache, DDoS protection, and not much else. The "AI psychosis" that may or may not have infected Cloudflare and caused this reminds me of Github's, except Cloudflare has a much much bigger moat.
sssilver 2mo ago
> There was a time Cloudflare just made the internet better This is the kind of naive thinking that enabled what's going on today. Sadly, the decentralized Internet is now destroyed and will never exist again.
- SpicyLemonZest 2mo ago
  What decentralized solution ever existed for preventing DDoS attacks while allowing a spike of traffic when, say, your blog post gets a highly upvoted HN submission? I am not really convinced this is a problem that can be solved without centralization. It'd be like trying to decentralize airports or communications cables.
  - esseph 2mo ago
    > What decentralized solution ever existed for preventing DDoS attacks while allowing a spike of traffic when, say, your blog post gets a highly upvoted HN submission? BGP FlowSpec https://github.com/exa-networks/exabgp/wiki/FlowSpec-Overview https://github.com/exa-networks/exabgp/wiki/FlowSpec-Overvie...
- orliesaurus 2mo ago
  Yeah but you can't blame 1 company for destroying the whole thing
vb-8448 2mo ago
> Too many ways to do the same thing, none of them great They want to eat at the AI table, given that they are not producing models or GPUs and not building datacentres, they try with the product. But we are still in the early stage so it's not clear what will work and what not, so they have to try as much as possible.
hasyimibhar 2mo ago
> Examples? Ok -> Let us look at data storage. > They got D1 (SQLite serverless), Durable Objects with their own SQLite, KV, R2, Queues, and Hyperdrive to speed up external Postgres or MySQL. Maybe it's just me, but I don't see this as confusing: - D1 is SQLite on cloud, their go-to relational database - Durable object is the cousin of durable execution (via actor model instead of saga) - KV is for caching like Redis (sub-ms read latency) - R2 is S3, R2 SQL is Iceberg + Athena (for OLAP queries) - Queue is SQS - Hyperdrive is bridge to allow CF worker to use native Postgres/MySQL driver (since v8 isolate cannot maintain persistent connection) I've built some stuff on top of CF, and the data ecosystem is actually useful.
0xbadcafebee 2mo ago
I don't know where this person got the idea that a company "run by engineers" would result in good products. Engineers don't always care about the end result. I've seen engineers completely compromise the utility and experience of the product to make their own job easier or more fun. If it "sounds logical", they assume it's good and don't ask more questions about how it will impact users. Would you hire a mechanical engineer to run a car company? The mechanical engineer cares if the transmission is highly efficient; they don't care if the seats are comfortable, or if the car can fit standard wheels/tires. Those are the things the car's user cares about. They expect the transmission to be efficient, but they equally expect it to be comfortable and compatible. "Infrastructure" isn't a raw mechanical component. It's a product with an entire "life" outside the technical. How it's controlled by a user, its responsiveness and intuitiveness, how they (and 3rd parties) interface with it, its operational and failure modes, its outputs and inputs... all of that's separate from the internal workings. It's the difference between a car having a door, and having a door that fits perfectly, opens with ease, and closes with no effort, gaps or seams. A ton of extra work is required to make that happen. Making the door is easy; making it fit well is much harder. Putting a nerd in charge will not fix the focus on the end result. You need to put people in charge who are obsessed with the customer's experience. Say what you will about Jeff Bezos and Steve Jobs, they at least got that right.
- orliesaurus 2mo ago
  take astral, planetscale.... Pi even - so many good orgs where the product is the result of engineering passion
  - esseph 2mo ago
    Never heard of any of those. Pi the AI harness, sure. Not sure those are good examples if people can't relate to them. Note: Astral was acquired by OpenAI, so I'd assume they will no longer meet your criteria soon.
    - orliesaurus 2mo ago
      ok those are probably my examples from my memory, feel free to replace those names with the ones you love the most.
JSR_FDED 2mo ago
I know CF has a good reputation, but at the same time when I first approached their website I have to agree with the poster, it’s just incoherent.
password4321 2mo ago
Cloudflare failed pre-AI with Cloudbleed, which could have been alright except for how they initially handled it. In addition, a few checks & balances have retired or stepped back so there's nothing stopping AI amplification of that company culture. They may succeed or not, but right now it's hard to stomach as a former fan. I see where a blog post like this could be coming from.
fragmede 2mo ago
Meh. I was trying to add additional domain names to an api key but it didn't work. Assuming that it's not operator error, it's a pretty annoying bug but its not the only one I've hit this week.
JoeDohn 2mo ago
What the author describes is the standard now at all "too big to fail" companies. These companies works in a way where you need to distinguish yourself or you die (even if you're competent), which tends to empower the extrovert with ego/cheek. The slop potion is only a catalyst for this kinda of culture. There is still hope, but in some cases we are on the real edge of an industrial catastrophe. And by "real" I mean physical (i.e : trains / cars / factories / banks that could go sideways cause of this BS wave).

120 points by gregzeng95 2mo ago

[flagged]

orf 2mo ago

> First of all tons of more outages than ever, remember that React useEffect fkup [0]? Complete insane that this would happen at an infra company that runs a third or so of the web. Bugs happen all the time. They roughly increase with scale, not decrease. There’s an argument to be made about better testing, but this specific bug seems like a perfect one to slip through: multiple services, hard to spot at code review, involves JS/frontend, invisible at low traffic (test/UT envs). So IMO it’s not completely insane. Is the implication that Cloudflare should have no bugs whatsoever?

minraws 2mo ago
Bugs happen all the time yes, outages don't. If outages are increasing with scale you get 1 or maybe 2 free passes. After that you either have in-ept Engineering or just in-ept leadership. I used to be all in on CF a while ago, now I am moving off them almost entirely. Same issue with GitHub, I can understand if you can't build for the scale when you couldn't predict it but if after over 12-18 months things don't seem to be improving what are you even doing? I honestly think all of these companies are deluded if they think people will stick around with all these weekly outage events. I have a homelab server I have had 2 outages in 1 year because my shitty ISP went down. Still at 99.9% uptime, I have done nothing special. I now have backup internet as well. Is it big? Nope but it doesn't need to be cf scale. And scale is the reason to use these services why would I use cloudflare if a homelab would have been enough? If they aren't designing and scaling their systems to handle this scale they might as well close shop, someone else might do it better. As a infra/dev person who does his own thing on the side, I might be the most impacted by these outages, so I might be coming off as harsh. But they cost me both time/money and headache in extra development work. Imagine prod deploys are down for 2 days why? Because GitHub actions keep failing... Oh serving new OTA updates broke? Why? dig into the code.. go oncall with users instead of doing work, realize it's a CF outage and the writes failed. (Feel the tears streaming down your face). If I have to waste dev time, with AI and me together we could self host it with higher reliability with significantly cheaper costs at this point even at fairly decent scale. I think any infra company that has more than 1 outage a year is already not worth investing in. But more than 3 and you might be better off self hosting, even in this ram apocalypse. If all people in SF are this unserious about reliability (which hasn't been my experience but HN seems especially open to break the prod if you have to) Then well software companies really do deserve to be replaced by AI.
- orf 2mo ago
  To put it bluntly: that is complete nonsense. AWS has had more than 1 outage a year - is it not worth investing in? Are they not serious? Outages are just a specific kind of bug, often surfaced by the interactions of several discrete bugs. Saying “you’re not serious if you have more than 1 bug a year” is silly.
  - minraws 2mo ago
    This might be rude but this is the reason we have shitty software everywhere. The "there can always be a bug, get over it mindset" is the reason software sucks, everything needs a dozen patches to even work and no one can have any peace in this business. I honestly don't even want to debate this because I see no point, as honestly the side that would have said outages shouldn't happen is dead buried and out of a job at this point. But, > Saying “you’re not serious if you have more than 1 bug a year” is silly. > Outages are just a specific kind of bug, often surfaced by the interactions of several discrete bugs. Several bugs or 1 bug? you decide and tell me, when you have made up your mind. Outages happen as a result of the system's design being shoddy in the first place, and an attitude where bugs shipped are acceptable because it's only a single component, rest of the system shouldn't fail. It won't go down was the entire thing with microservices and "the cloud" I could link some blogs and brochures if you want me to. I am honestly over the moon with these conversations on HN, really proves the point why AI is rightly replacing engineers in software, because there wasn't any engineering to begin with. Imagine if a civil engineer said bridges fall it's fine for a few bridges to fall because bugs happen, and honestly bridge falling is just a kind of bug where several discrete bugs happened at once. I am certain there is some room to argue about it, but what is clear is that if it was happening every single month you would have stopping using bridges unless absolutely necessary. I am not sure if there is ever a reply for this honestly, I honestly don't even blame the people just the culture at this point. If that's what software is to some, "something that can fail at any time" and cloud's selling point is just fewer "switches to flip" I am sure I can't change that. But I won't be deluded into thinking that software outages are just an everyday "bug" and it's fine to have several outages a year, it is the normal mode of operation in-fact no harm done.

spicyusername 2mo ago

The fact is that every major company is AI accelerating their development, and soon, it will just be every company. We're already past the point of getting our hackles raised when we suspect something is AI written. Everything is or soon will be AI written. So I don't think it's that useful to blame bad product design on AI. Bad product design is just bad product design. Bad writing is just bad writing. Bad graphic design is just bad graphic design. The industry needs to get over this hump of pretending we're not all going to adopt AI for every imaginable thing. The cat's out of the bag, for better or for worse.

decide1000 2mo ago
It's not just the AI. The post mentions poor choices because of a product management mindset instead of engineers leading the PMs.
- kylecazar 2mo ago
  Only part of the article I disagree with a bit. I'm willing to bet most infrastructure PM's at CF are technical people with engineering backgrounds. I think what's more likely the culprit is the desperation to claim ground. They (and many others) want to rush out an answer to all of AI's new challenges out of fear of being beaten to the punch. It usually leads to a complex, almost incoherent product suite.
GPerson 2mo ago
I believe protest and criticism are corrective processes, and we shouldn’t give up.
- orliesaurus 2mo ago
  Thank you! I am a CF customer, I pay for the service, I just want it to get better and better!
xpct 2mo ago
Well, let's not normalize bad product design. Worse, let's not normalize pushing out broken projects that never get maintained. It's noise and breaks trust. I'd say the maintenance part is especially bad now, I hold very little trust that people will maintain the projects they share.

decide1000 2mo ago

Great read. Exact pinpointing of my feelings about CF lately.

jbrooks84 2mo ago

Accurate

rfgplk 2mo ago

[dead]

daishi55 2mo ago

I don’t understand the impulse to whine and complain about the aesthetics of a big company. > a little more cringey and clique Why on earth do I care if someone thinks Cloudflare is cringe? What is interesting about that? They are following the market like everyone else. > Too many ways to do the same thing, none of them great…They got D1 (SQLite serverless), Durable Objects with their own SQLite, KV, R2, Queues, and Hyperdrive to speed up external Postgres or MySQL. What does this have to do with AI psychosis? I thought that was the thesis of this article? As I said, it’s just directionless complaining.

gegtik 2mo ago
i read the article as caring about the values, principles, and goals of the human beings driving it. if you are looking exclusively at compiled bits and bytes and/or quarterly statements then I agree, human/societal judgment seems irrelevant.
- daishi55 2mo ago
  What values and principles are supposed to be preventing cloudflare, the internet and technology company, from exploring hot new internet markets and technologies?
  - orliesaurus 2mo ago
    the constant "fights" with Vercel instead of delivering shareholder value by refining the product
    - esseph 2mo ago
      I'm not even sure if I would know who Vercel is if I were at Cloudflare. The scale between the two is staggeringly different.
      - Plont 2mo ago
        That makes it worse, not better.

jeresuikkila 2mo ago

I'm also curious about the question posted at the end: where did the infrastructure nerds go?

dgellow 2mo ago
Likely to the AI labs and hyperscalers? Given the insane growth and level of expenditure I assume it’s a fun place where to work (assuming you’re ok contributing actively to the AI industry)

reisse 2mo ago

> There was a time Cloudflare just made the internet better There wasn't. Cloudflare is a cancer grown too big. And it was always positioned to become one, the middleman between users and the Internet. It is already painful to browse web sometimes using the non-"standard" tools (that is, not a Chrome with Google account signed in, not an EU/US residential IP). What if tomorrow Cloudflare checks will require attested and signed browser binaries?

gruez 2mo ago
>It is already painful to browse web sometimes using the non-"standard" tools (that is, not a Chrome with Google account signed in, not an EU/US residential IP). What if tomorrow Cloudflare checks will require attested and signed browser binaries? If it's painful, that's not because of cloudflare. You can get past turnstile challenges using tor browser in a VM. You still might be blocked because of policies set by site owners, but that can hardly be blamed on cloudflare.
- dingaling 2mo ago
  > but that can hardly be blamed on cloudflare They are installing the gates with configurable locks that the site owners use. They are absolutely to blame.
  - gruez 2mo ago
    Should nginx be blamed too, because that also allows site owners to ban tor/VPN users?
    - 398642258909 2mo ago
      Never had the issue of nginx gatekeeping, but Cloudflare's shitty captchas constantly block the access, since my browser settings don't adhere to their MitM-mandated rules.
- anon5868 2mo ago
  >in a VM Are you saying it's possible to detect VMs in the browser without using the WebGL vendor? Mind sharing some resources?

hypfer 2mo ago

Why would they do anything else? They own a huge chunk of the internet now - with many, many companies having deeply integrated CF into their own stuff. Switching costs are so high, from a business standpoint, investing in technical excellence would be wasted money. They're just a corp like everyone else. Gravity also applies to them. __ Don't take this as "they're right to do so". Take this as "You should never have started integrating their products this deeply. This was inevitable and obvious." Using CF always was trading short-term wins for long-term losses. We're now entering said long-term.

alberth 2mo ago

What’s missed from the post is that Cloudflare for many years now has felt that they are a “Cloud 2.0” company. And AWS/GCP/etc are “Cloud 1.0”. They believe that AI Agent will fundamentally change the internet, and it’s hard to fault them for seeing it that way when their own stats shows that bot/agents account for the majority of Internet traffic. As such, going all in on AI fits into their thesis of being “Cloud 2.0”.

latchkey 2mo ago

on the bright side, this rant doesn't sound ai generated.

orliesaurus 2mo ago
thank you, I actually only used it to find the specific sources that I linked at the bototm (through Google Search AI: i.e. "what was the URL of the postmortem for that useEffect bug")
dude250711 2mo ago
An organic rant, for some locally-produced even.
- orliesaurus 2mo ago
  > dude250711 I feel like I know you :)

tosh 2mo ago

I think like with most large companies that still ship new stuff: it gets a bit more complex for customers to figure out what the good nuggets are and what to ignore for now same @ google, aws and so on (I usually rely on opinions of people i trust to find the 'javascript — the good parts' version of large offerings) the other extreme would be not to try and ship new stuff which is also risky difficult to find a balance and probably a good idea to over-index on momentum and new stuff while investing enough in hardening and improving the stuff that sticks

tosh 2mo ago
I also noticed the ui/ux of cloudflare got way better in the last weeks. That's a good sign. Usually ui/ux in large companies gets worse not better!

piterrro 2mo ago

The problem is, beside their size, they have not moat to compete in AI space. Their data centers are spread across the world in other companys DCs (btw do they own any DC actually from top to bottom?).

dgellow 2mo ago
Side question: does anyone have a moat in the AI industry? AI vendors for sure do not. Hyperscalers don’t seem to have one. Datacenter companies are in an industry that is more difficult to enter but don’t have a moat per se. Same for manufacturers of HBM and other hardware. NVIDIA is maybe the only player with an actual moat thanks to CUDA & co
- piterrro 2mo ago
  Companies specialized in building DCs and AI labs specialized in niche models?
  - dgellow 2mo ago
    Actually, not DC builders themselves, but companies that manufacture fairly specific parts used by DCs, like the massive gas turbines they’ve been using. I’m not familiar with that part of the industry but would assume that has to be a business with a pretty big moat

gedy 2mo ago

A good point made in the post (that's not really specific to CF) is the problem with multiplying features. It's interesting that most of the extremely-hot-on-AI companies and leaders I know are hyped about 100x product dev, etc and want to throw tons of new assorted features out. This is not a good thing! I've been at companies with lots of staffing and velocity (pre-AI), and it is a huge foot-gun to think pumping out features is automatically a good thing. Most paying customers do not want this from SaaS type companies. Smaller companies especially should calm down and do one thing well, and unfortunately AI dev does not encourage that.

esseph 2mo ago
> Smaller companies especially should calm down and do one thing well They're going to get absolutely steamrolled unless it is a high risk critical industry.
- gedy 2mo ago
  Maybe, but pretending to be Atlassian or whomever doesn't help either afaics

tokioyoyo 2mo ago

It sounds like a blog post in support of a slowdown for product releases. But the reality is there are multiple companies trying to capture the attention of the same market. Maybe 2 years ago it would’ve been acceptable to release the product internally, slowly add features, dogfood it, then open to public a year later. But now someone else will release the same product within a month, get sticky customers, and most likely won’t switch. It’s much easier and faster to release features, so once anyone sees a competition gaining some attention, they just copy the same feature. Which, i think, is fair. It’s easy to blame the org, but the market is extremely competitive right now. Kind of race to the bottom, except the hardware parts, which have different constraints right now.

orliesaurus 2mo ago
Hi, I wrote the post, I think blaming the org for releasing things that aren't engineered/created with "love" (including docs, examples, long-term planning and a roadmap of improvement that will follow-on), and suitable for production, without labeling them as beta, is in fact detrimental to the status of a company's image.
- esseph 2mo ago
  So where do you stand on AWS that may release more than 25 new services a year?
  - orliesaurus 2mo ago
    I wouldnt be surprised if AWS buys Vercel (an AWS wrapper btw) and puts all the "cool new things" under the vercel cli.
- tokioyoyo 2mo ago
  AWS/GCP/Azure are full of abandonware. I’m more familiar with the AWS-side of things, and most of them were/are built on top of existing core services. AWS’s/GCP’s image is fine. I’d say as long as CF’s core (CDN/DDoS mitigation/etc.) is stable, they should experiment. To put it this way, it took GH almost a year to release Stacked PRs to public. And it’s already being questioned if it’ll stand the test of time. Again, insanely competitive market, as everyone wants to sell shovels.

yodon 2mo ago

Cloudflare's UI is basically the web equivalent of the git CLI. Powerful, but the densest collection of UI anti-patterns known to man. Their "designers" obviously walked up hill both ways in the snow to school every day while being eaten by a walrus and feel anyone who doesn't enjoy a suitably opaque navigation scheme isn't worth acknowledging.

orliesaurus 2mo ago

Hey everyone, author here, I am very frustrated with the state of Cloudflare and I posted this blog the other day on my blog and here on HN (but generated no discussion), thanks for re-posting it, I am gonna start reading the comments and start replying

sarreph 2mo ago

I have to say: Yes, there is a lot of velocity from Cloudflare pushing out AI-adjacent features as of late. But also: What they've doing with Workers et al. has made them a platform for fully-fledged apps and deployments. I personally love the ecosystem and use it for all of my projects now. It's like the perfect blend of ease-of-use and DX of Heroku, and breadth of services of AWS / GCP. Well, maybe not quite AWS or GCP, but that's kind of the point -- they've created an opinionated system of "objects" that are all highly extensible to the point where in my opinion you can pretty much deploy anything you want on it. It took me a bit of time to learn the "Cloudflare way" of doing things, but once I started making on it I saw just how flexible (and low cost!) everything is.

aleda145 2mo ago

Hard disagree. Sure their docs are bad, and it's sometimes buggy. But to me cloudflare is actually innovating and trying to offer "new" infrastructure. Durable objects and workers are super cool. R2 has free egress, isn't it insane that they could pull that off? If what you want is a VM to run postgres then there are other offerings that would be much better. Not saying they aren't evil though, they probably are, but the infra stuff is cool.

orliesaurus 2mo ago
I disagree with your disagreement. I wrote the post and I am not blaming the innovation. I am blaming the chaos of innovation. Everything feels DISJOINTED. The innovation is useful, the delivery of the innovation is the pitfall here.
- bestham 2mo ago
  Innovation is only linear in hindsight . Pure speculation on my part but once CF made some leaps in a particular area this enabled them to gather more data and shift direction to reach their over all goals by stifling the causes and the innovation that lead them there. New management comes in, requirements change and this is evident to us on the outside as disjointed behaviour. Companies need to make money from the internet for it to exist. Companies also need to behave and refrain from tragedy of the commons for the internet to keep existing as we know it.
jokethrowaway 2mo ago
They have cool tech but they can't do product, like most of big tech. Have you tried wrangler? holy Hire enough product owners and that's what happens. That's why they need to buy startups every once and then, to bring some good bacteria in their messed up corporate gut.
- orliesaurus 2mo ago
  yep, wrangler is a monster, the idea is solid - it's like the gcloud or heroku toolbet - the execution isn't nearly as neat.

themgt 2mo ago

I view a lot of this as downstream from sort of the IaaS version of the "DRM problem" in cryptography. In the DRM problem you want to sell someone a Blu-Ray with a movie they can play, but also not allow them to copy the movie. So they get the data but can't have the data. In the IaaS problem you want to sell someone a piece of server(s) in a datacenter they can play with, but also not allow them to just directly use the hardware (because then they could have open standards and portability). So the product teams are required to work backwards from the necessity of wrapping compute in a thing you can sell as a product. This also explains nonsense like "durable objects", which may have some purpose but become the weekly podcast fad after getting hyped by cloud providers trying to find new ways to sell compute without selling compute.

nzeid 2mo ago

This is definitely as opinionated as an opinion piece can go. Worth mentioning I don't use CF anywhere or care about it... but my employer does. I can really only remember one catastrophic outage. It involved some part of CF that had no redundancy, and the incident was followed by a pretty whiny blog about a data center. That's pretty much it. There were other minor outages but I'm not aware that it affected business. Despite the disclaimer I think the author should explain precisely how "AI product mindset" or whatever is actually translating to bad outcomes.

gexla 2mo ago
Just skimmed the article, but the CF your employer used was probably around a lot longer than AI? There's parts of it that can bring everything down. Then there's parts of it that will bring down just what your team(s) put in a load of hard work to build on top of it.
- nzeid 2mo ago
  The blog implies that CF's core business is suffering because of added investment in what the author describes as AI. See their useEffect bug as an example. EDIT: To be clear, my employer still uses CF and I don't see any evidence that it's gotten worse.

NicoJuicy 2mo ago

For an article that claims to know Cloudflare, I don't think he does. Cloudflare build from the bottom up. First the SDN ( Software Defined Network), then the products on top. The first one was logical: CDN + DDOS. All the rest continued to build on top of the SDN - Workers, D1, ... But once they have the full stack ( which happened arround D1), it was time to let others build on top of Cloudflare. That's the current stage ( Cloud 2.0), that was always the goal. It's even in their ticker ( NET ). In practise, AI could be a godsend. A total new foundation for software where the Cloudflare stack is a perfect product fit( disposable compute, where others will protect their compute with long running compute contracts). I can't blame them to try to seize the opportunity. Tbh. I feels like the author hasn't actually adjusted to how Cloudflare is building it's cloud ( eg. how powerfull Durable Objects is). PS. A big outage is a long time ago... ( we use them, I remember the issues those days and the post mortem, a lot became more stable as far as I noticed).

thecatapps 2mo ago

I remember being so excited about Cloudflare's releases that I'd check their blog every couple of weeks for what new thing they were cooking up. This was before I actually had to use them to serve production clients. When it came time to launch (2023?), the initial setup was using DNS/Cache, Workers, Workers KV, R2, D1, Durable Objects, Access, and Queues. My first hint was that the Typescript library to access their API was simply just wrong. API requests through the library would fail, complaining about missing fields or invalid types, even though the types said my construction was correct, and I received dismissive replies when raising it in their Discord. Then there were the D1 issues with random requests failing. Then there were the Durable Objects issues with syncing clients in our collaborative editor. Then there were the KV and (by extension) Access outages. Eventually it made sense to switch it all over to AWS. Today, I trust Cloudflare for DNS, Cache, DDoS protection, and not much else. The "AI psychosis" that may or may not have infected Cloudflare and caused this reminds me of Github's, except Cloudflare has a much much bigger moat.

sssilver 2mo ago

> There was a time Cloudflare just made the internet better This is the kind of naive thinking that enabled what's going on today. Sadly, the decentralized Internet is now destroyed and will never exist again.

SpicyLemonZest 2mo ago
What decentralized solution ever existed for preventing DDoS attacks while allowing a spike of traffic when, say, your blog post gets a highly upvoted HN submission? I am not really convinced this is a problem that can be solved without centralization. It'd be like trying to decentralize airports or communications cables.
- esseph 2mo ago
  > What decentralized solution ever existed for preventing DDoS attacks while allowing a spike of traffic when, say, your blog post gets a highly upvoted HN submission? BGP FlowSpec https://github.com/exa-networks/exabgp/wiki/FlowSpec-Overview https://github.com/exa-networks/exabgp/wiki/FlowSpec-Overvie...
orliesaurus 2mo ago
Yeah but you can't blame 1 company for destroying the whole thing

vb-8448 2mo ago

> Too many ways to do the same thing, none of them great They want to eat at the AI table, given that they are not producing models or GPUs and not building datacentres, they try with the product. But we are still in the early stage so it's not clear what will work and what not, so they have to try as much as possible.

hasyimibhar 2mo ago

> Examples? Ok -> Let us look at data storage. > They got D1 (SQLite serverless), Durable Objects with their own SQLite, KV, R2, Queues, and Hyperdrive to speed up external Postgres or MySQL. Maybe it's just me, but I don't see this as confusing: - D1 is SQLite on cloud, their go-to relational database - Durable object is the cousin of durable execution (via actor model instead of saga) - KV is for caching like Redis (sub-ms read latency) - R2 is S3, R2 SQL is Iceberg + Athena (for OLAP queries) - Queue is SQS - Hyperdrive is bridge to allow CF worker to use native Postgres/MySQL driver (since v8 isolate cannot maintain persistent connection) I've built some stuff on top of CF, and the data ecosystem is actually useful.

0xbadcafebee 2mo ago

I don't know where this person got the idea that a company "run by engineers" would result in good products. Engineers don't always care about the end result. I've seen engineers completely compromise the utility and experience of the product to make their own job easier or more fun. If it "sounds logical", they assume it's good and don't ask more questions about how it will impact users. Would you hire a mechanical engineer to run a car company? The mechanical engineer cares if the transmission is highly efficient; they don't care if the seats are comfortable, or if the car can fit standard wheels/tires. Those are the things the car's user cares about. They expect the transmission to be efficient, but they equally expect it to be comfortable and compatible. "Infrastructure" isn't a raw mechanical component. It's a product with an entire "life" outside the technical. How it's controlled by a user, its responsiveness and intuitiveness, how they (and 3rd parties) interface with it, its operational and failure modes, its outputs and inputs... all of that's separate from the internal workings. It's the difference between a car having a door, and having a door that fits perfectly, opens with ease, and closes with no effort, gaps or seams. A ton of extra work is required to make that happen. Making the door is easy; making it fit well is much harder. Putting a nerd in charge will not fix the focus on the end result. You need to put people in charge who are obsessed with the customer's experience. Say what you will about Jeff Bezos and Steve Jobs, they at least got that right.

orliesaurus 2mo ago
take astral, planetscale.... Pi even - so many good orgs where the product is the result of engineering passion
- esseph 2mo ago
  Never heard of any of those. Pi the AI harness, sure. Not sure those are good examples if people can't relate to them. Note: Astral was acquired by OpenAI, so I'd assume they will no longer meet your criteria soon.
  - orliesaurus 2mo ago
    ok those are probably my examples from my memory, feel free to replace those names with the ones you love the most.

JSR_FDED 2mo ago

I know CF has a good reputation, but at the same time when I first approached their website I have to agree with the poster, it’s just incoherent.

password4321 2mo ago

Cloudflare failed pre-AI with Cloudbleed, which could have been alright except for how they initially handled it. In addition, a few checks & balances have retired or stepped back so there's nothing stopping AI amplification of that company culture. They may succeed or not, but right now it's hard to stomach as a former fan. I see where a blog post like this could be coming from.

fragmede 2mo ago

Meh. I was trying to add additional domain names to an api key but it didn't work. Assuming that it's not operator error, it's a pretty annoying bug but its not the only one I've hit this week.

JoeDohn 2mo ago

What the author describes is the standard now at all "too big to fail" companies. These companies works in a way where you need to distinguish yourself or you die (even if you're competent), which tends to empower the extrovert with ego/cheek. The slop potion is only a catalyst for this kinda of culture. There is still hope, but in some cases we are on the real edge of an industrial catastrophe. And by "real" I mean physical (i.e : trains / cars / factories / banks that could go sideways cause of this BS wave).