5 ms·

I do not work in the space at all, but it seems like Cloudflare has been having more network disruptions lately than they used to. To anyone who deals with this

by atty 8mo ago

I do not work in the space at all, but it seems like Cloudflare has been having more network disruptions lately than they used to. To anyone who deals with this sort of thing, is that just recency bias?

Icathian 8mo ago
It is not. They went about 5 years without one of these, and had a handful over the last 6 months. They're really going to need to figure out what's going wrong and clean up shop.
- NinjaTrance 8mo ago
  Engineers have been vibe coding a lot recently...
  - jsheard 8mo ago
    The featured blog post where one of their senior engineering PMs presented an allegedly "production grade" Matrix implementation, in which authentication was stubbed out as a TODO, says it all really. I'm glad a quarter of the internet is in such responsible hands.
    - dana321 8mo ago
      Thats a classic claude move, even the new sonnet 4.6 still does this.
      - brutalc 8mo ago
        [dead]
      - bonesss 8mo ago
        It’s almost as classic as just short circuiting tests in lightly obfuscated ways. I could be quite the kernel developer if making the test green was the only criteria.
      - allovertheworld 7mo ago
        Wait till you get AI to write unit tests and tell it the test must pass. After a few rounds it will make the test “assert(true)” when the code cant get the test to pass
    - blibble 8mo ago
      there was also a post here where an engineer was parading around a vibe-coded oauth library he'd made as a demonstration of how great LLMs were at which point the CVEs started to fly in
    - gtowey 8mo ago
      It's spreading and only going to get worse. Management thinks AI tools should make everyone 10x as productive, so they're all trying to run lean teams and load up the remaining engineers with all the work. This will end about as well as the great offshoring of the early 2000s.
    - ranger_danger 8mo ago
      Matrix doesn't actually define how one should do authentication though... every homeserver software is free to implement it however they want.
      - Arathorn 8mo ago
        the main bit of auth which was left unimplemented on matrix-workers was the critical logic which authorizes traffic over federation: https://spec.matrix.org/latest/server-server-api/#authorization-rules https://spec.matrix.org/latest/server-server-api/#authorizat... Auth for clients is also specified in the spec - there is some scope for homeservers to freestyle, but nowadays they have to implement OIDC: https://spec.matrix.org/latest/client-server-api/#client-authentication https://spec.matrix.org/latest/client-server-api/#client-aut...
  - dakiol 8mo ago
    No joke. In my company we "sabotaged" the AI initiative led by the CTO. We used LLMs to deliver features as requested by the CTO, but we introduced a couple of bugs here and there (intentionally). As a result, the quarter ended up with more time allocated to fix bugs and tons of customer claims. The CTO is now undoing his initiative. We all have now some time more to keep our jobs.
    - logicchains 8mo ago
      That's not "sabotaged", that's sabotaged, if you intentionally introduced the bugs. Be very careful admitting something like that publicly unless you're absolutely completely sure nobody could map your HN username to your real identity.
    - samrus 8mo ago
      Thats actively malicious. I understand not going out of your way to catch the LLMs' bugs so as to show the folly of the initiative, but actively sabotaging it is legitimately dangerous behavior. Its acting in bad faith. And i say this as someone who would mostly oppose such an initiative myself I would go so far as to say that you shouldnt be employed in the industry. Malicious actors like you will contribute to an erosion of trust thatll make everything worse
      - sp00chy 8mo ago
        Might be but sometimes you don’t have another choice when employers are enforcing AIs which have no „feeling“ for context of all business processes involved created by human workers in the years before. Those who spent a lot of love and energy for them mostly. And who are now forced to work against an inferior but overpowered workforce. Don’t stop sabotaging AI efforts.
        samrus 8mo ago
        Honestly i kinda like the aesthetic of cyberanarchism, but its not for me. It erodes trust
      - 8mo ago
- Ylpertnodi 8mo ago
  Typo: "shop", should have been with an 'el'. (: phonetically, because 'l's are hard to read.
Betelbuddy 8mo ago
Cloudflare Outages are as predictable, as the Sun coming up tomorrow. Its their engineering culture. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=%09Cloudflare%20outage&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
lysace 8mo ago
It has been roughly speaking five and a half years since the IPO. The original CTO (John Graham-Cumming) left about a year ago.
- jacquesm 8mo ago
  They coasted on momentum for half a year. I don't even think it says anything negative about the current CTO, but more of what an exception JGC is relative to what is normal. A CTO leaving would never show up the next day in the stats, the position is strategic after all. But you'd expect to see the effect after a while, 6 months is longer than I would have expected, but short enough that cause and effect are undeniable. Even so, it is a strong reminder not to rely on any one vendor for critical stuff, in case that wasn't clear enough yet.
  - lysace 8mo ago
    You can coast for quite some time (5-10 years?) if you really lean into it (95% of the knowledge of maintaining and scaling the stack is there in the minds of hundreds of developers). Seems like Matthew Prince didn't choose that route.
    - jacquesm 8mo ago
      The problem is that CF operates in a highly dynamic environment and you can't really do that if the minds of those hundreds of developers relied for the major decision making on a key individual. This is the key individual paradox: they can be a massive asset and make the impossible happen but if and when they leave you've got a real problem unless you can find another individual that is just as capable. Now, I do trust JGC to have created an organization that is as mature as it could be, but at the same time it is next to impossible to quantify your own effect on the whole because you lack objectivity and your underlings may not always tell you the cold hard truth for reasons all their own. And in this case the problem is even larger: the experience collected by the previous guru does not transfer cleanly to the new one, simply because the new one lacks the experience of seeing the company go from being a tiny player to being a behemoth, and that's something you can do only once. I've always been of the opinion that without JGC Cloudflare did not stand a chance, irrespective of those hundreds of developers. And that's before we get into things like goodwill. And of those hundreds of developers you have to wonder how many see the writing on the wall and are thinking of jumping ship. The best ones always leave first. I would not be surprised at all if this whole saga ends with Google, Microsoft or Amazon absorbing CF at a fraction of its current value.
dazc 8mo ago
Launching a new service every 5 minutes is obviously stretching their resources.
candiddevmike 8mo ago
Wait till you see the drama around their horrible terraform provider update/rewrite: https://github.com/cloudflare/terraform-provider-cloudflare/issues/5573#issuecomment-3071740053 https://github.com/cloudflare/terraform-provider-cloudflare/...
slophater 8mo ago
been at cf for 7 yrs but thinking of gtfo soon. the ceo is a manchild, new cto is an idiot, rest of leadership was replaced by yes-men, and the push for AI-first is being a disaster. c levels pretend they care about reliability but pressure teams to constantly ship, cto vibe codes terraform changes without warning anyone, and it's overall a bigger and bigger mess even the blog, that used to be a respected source of technical content, has morphed into a garbage fire of slop and vaporware announcements since jgc left.
- slophater 8mo ago
  amazing how my comment was flagged in 30 seconds... keep bootlicking
- a24446ff87 8mo ago
  GSD! GSD!! ship! ship! ship! **everything breaks** ... **everything breaks again** oh fuck! Code Orange! I repeat, Code Orange! we need to rebuild trust(R)(TM)! we've let our customers down! ... **everything breaks again** Code Orangier! I repeat, Code Orangier!
  - slophater 8mo ago
    exactly. recently "if the cto is shipping more than you, you're doing something wrong" cto can't even articulate a sentence without passing it through an LLM, and instead of doing his job he's posting the stupidest shit to his personal bootlicking chat channel. I cringe every time at the brown-nosers that inhabit that hovel. no words for what the product org is becoming too. they should take their own advice a bit further and just replace all the leadership with an LLM, it would be cheaper and it's the same shit in practice
    - ifwinterco 7mo ago
      I have worked in some dysfunctional places but nothing like that, does sound bad. Just got to keep your head, remember it’s just a job and you get paid regardless. Clock in, clock out, do the work assigned to you but mentally just check out while you look for a new role

by atty 8mo ago

Icathian 8mo ago

It is not. They went about 5 years without one of these, and had a handful over the last 6 months. They're really going to need to figure out what's going wrong and clean up shop.

NinjaTrance 8mo ago
Engineers have been vibe coding a lot recently...
- jsheard 8mo ago
  The featured blog post where one of their senior engineering PMs presented an allegedly "production grade" Matrix implementation, in which authentication was stubbed out as a TODO, says it all really. I'm glad a quarter of the internet is in such responsible hands.
  - dana321 8mo ago
    Thats a classic claude move, even the new sonnet 4.6 still does this.
    - brutalc 8mo ago
      [dead]
    - bonesss 8mo ago
      It’s almost as classic as just short circuiting tests in lightly obfuscated ways. I could be quite the kernel developer if making the test green was the only criteria.
    - allovertheworld 7mo ago
      Wait till you get AI to write unit tests and tell it the test must pass. After a few rounds it will make the test “assert(true)” when the code cant get the test to pass
  - blibble 8mo ago
    there was also a post here where an engineer was parading around a vibe-coded oauth library he'd made as a demonstration of how great LLMs were at which point the CVEs started to fly in
  - gtowey 8mo ago
    It's spreading and only going to get worse. Management thinks AI tools should make everyone 10x as productive, so they're all trying to run lean teams and load up the remaining engineers with all the work. This will end about as well as the great offshoring of the early 2000s.
  - ranger_danger 8mo ago
    Matrix doesn't actually define how one should do authentication though... every homeserver software is free to implement it however they want.
    - Arathorn 8mo ago
      the main bit of auth which was left unimplemented on matrix-workers was the critical logic which authorizes traffic over federation: https://spec.matrix.org/latest/server-server-api/#authorization-rules https://spec.matrix.org/latest/server-server-api/#authorizat... Auth for clients is also specified in the spec - there is some scope for homeservers to freestyle, but nowadays they have to implement OIDC: https://spec.matrix.org/latest/client-server-api/#client-authentication https://spec.matrix.org/latest/client-server-api/#client-aut...
- dakiol 8mo ago
  No joke. In my company we "sabotaged" the AI initiative led by the CTO. We used LLMs to deliver features as requested by the CTO, but we introduced a couple of bugs here and there (intentionally). As a result, the quarter ended up with more time allocated to fix bugs and tons of customer claims. The CTO is now undoing his initiative. We all have now some time more to keep our jobs.
  - logicchains 8mo ago
    That's not "sabotaged", that's sabotaged, if you intentionally introduced the bugs. Be very careful admitting something like that publicly unless you're absolutely completely sure nobody could map your HN username to your real identity.
  - samrus 8mo ago
    Thats actively malicious. I understand not going out of your way to catch the LLMs' bugs so as to show the folly of the initiative, but actively sabotaging it is legitimately dangerous behavior. Its acting in bad faith. And i say this as someone who would mostly oppose such an initiative myself I would go so far as to say that you shouldnt be employed in the industry. Malicious actors like you will contribute to an erosion of trust thatll make everything worse
    - sp00chy 8mo ago
      Might be but sometimes you don’t have another choice when employers are enforcing AIs which have no „feeling“ for context of all business processes involved created by human workers in the years before. Those who spent a lot of love and energy for them mostly. And who are now forced to work against an inferior but overpowered workforce. Don’t stop sabotaging AI efforts.
      - samrus 8mo ago
        Honestly i kinda like the aesthetic of cyberanarchism, but its not for me. It erodes trust
    - 8mo ago
Ylpertnodi 8mo ago
Typo: "shop", should have been with an 'el'. (: phonetically, because 'l's are hard to read.

Betelbuddy 8mo ago

Cloudflare Outages are as predictable, as the Sun coming up tomorrow. Its their engineering culture. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=%09Cloudflare%20outage&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

lysace 8mo ago

It has been roughly speaking five and a half years since the IPO. The original CTO (John Graham-Cumming) left about a year ago.

jacquesm 8mo ago
They coasted on momentum for half a year. I don't even think it says anything negative about the current CTO, but more of what an exception JGC is relative to what is normal. A CTO leaving would never show up the next day in the stats, the position is strategic after all. But you'd expect to see the effect after a while, 6 months is longer than I would have expected, but short enough that cause and effect are undeniable. Even so, it is a strong reminder not to rely on any one vendor for critical stuff, in case that wasn't clear enough yet.
- lysace 8mo ago
  You can coast for quite some time (5-10 years?) if you really lean into it (95% of the knowledge of maintaining and scaling the stack is there in the minds of hundreds of developers). Seems like Matthew Prince didn't choose that route.
  - jacquesm 8mo ago
    The problem is that CF operates in a highly dynamic environment and you can't really do that if the minds of those hundreds of developers relied for the major decision making on a key individual. This is the key individual paradox: they can be a massive asset and make the impossible happen but if and when they leave you've got a real problem unless you can find another individual that is just as capable. Now, I do trust JGC to have created an organization that is as mature as it could be, but at the same time it is next to impossible to quantify your own effect on the whole because you lack objectivity and your underlings may not always tell you the cold hard truth for reasons all their own. And in this case the problem is even larger: the experience collected by the previous guru does not transfer cleanly to the new one, simply because the new one lacks the experience of seeing the company go from being a tiny player to being a behemoth, and that's something you can do only once. I've always been of the opinion that without JGC Cloudflare did not stand a chance, irrespective of those hundreds of developers. And that's before we get into things like goodwill. And of those hundreds of developers you have to wonder how many see the writing on the wall and are thinking of jumping ship. The best ones always leave first. I would not be surprised at all if this whole saga ends with Google, Microsoft or Amazon absorbing CF at a fraction of its current value.

dazc 8mo ago

Launching a new service every 5 minutes is obviously stretching their resources.

candiddevmike 8mo ago

Wait till you see the drama around their horrible terraform provider update/rewrite: https://github.com/cloudflare/terraform-provider-cloudflare/issues/5573#issuecomment-3071740053 https://github.com/cloudflare/terraform-provider-cloudflare/...

slophater 8mo ago

been at cf for 7 yrs but thinking of gtfo soon. the ceo is a manchild, new cto is an idiot, rest of leadership was replaced by yes-men, and the push for AI-first is being a disaster. c levels pretend they care about reliability but pressure teams to constantly ship, cto vibe codes terraform changes without warning anyone, and it's overall a bigger and bigger mess even the blog, that used to be a respected source of technical content, has morphed into a garbage fire of slop and vaporware announcements since jgc left.

slophater 8mo ago
amazing how my comment was flagged in 30 seconds... keep bootlicking
a24446ff87 8mo ago
GSD! GSD!! ship! ship! ship! **everything breaks** ... **everything breaks again** oh fuck! Code Orange! I repeat, Code Orange! we need to rebuild trust(R)(TM)! we've let our customers down! ... **everything breaks again** Code Orangier! I repeat, Code Orangier!
- slophater 8mo ago
  exactly. recently "if the cto is shipping more than you, you're doing something wrong" cto can't even articulate a sentence without passing it through an LLM, and instead of doing his job he's posting the stupidest shit to his personal bootlicking chat channel. I cringe every time at the brown-nosers that inhabit that hovel. no words for what the product org is becoming too. they should take their own advice a bit further and just replace all the leadership with an LLM, it would be cheaper and it's the same shit in practice
  - ifwinterco 7mo ago
    I have worked in some dysfunctional places but nothing like that, does sound bad. Just got to keep your head, remember it’s just a job and you get paid regardless. Clock in, clock out, do the work assigned to you but mentally just check out while you look for a new role