6 ms·

Internet, out. Very big day for an engineering team indeed. Can't vibe code your way out of this issue...

rvz 1y ago
> Can't vibe code your way out of this issue... Exactly. This time, some LLM providers are also down and can't help vibe coders on this issue.
- fragmede 1y ago
  Qwen3 on lm-studio running fine on my work Mac M3, what's wrong with yours?
LostMyLogin 1y ago
Pour one out for everyone on-call right now.
- xvector 1y ago
  After some thankless years preventing outages for a big tech company, I will never take an oncall position again in my life. Most miserable working years I have had. It's wild how normalized working on weekends and evenings becomes in teams with oncall. But it's not normal. Our users not being able to shitpost is simply not worth my weekend or evening. And outside of Google you don't even get paid for oncall at most big tech companies! Company losing millions of dollars an hour, but somehow not willing to pay me a dime to jump in at 3AM? Looks like it's not my problem!
  - scns 1y ago
    > And outside of Google you don't even get paid for oncall at most big tech companies. What the redacted?
    - redwall_hp 1y ago
      Welcome to the typical American salary abuse. There's even a specific legal cutout exempting information technology, scientific and artistic fields from the overtime pay requirements of the Fair Labor Standards Act. There's a similar cutout for management, which is how companies like GameStop squeeze their retail managers. They just don't give enough payroll hours for regular employees, so the salaried (but poorly paid) manager has to cover all of the gaps.
  - sneak 1y ago
    > But this is not normal. Our users not being able to shitpost is simply not worth my weekend or evening. It is completely normal for staff to have to work 24/7 for critical services. Plumbing, HVAC, power plant engineers, doctors, nurses, hospital support staff, taxi drivers, system and network engineers - these people keep our modern world alive, all day, every day. Weekends, midnights, holidays, every hour of every day someone is AT WORK to make sure our society functions. Not only is it normal, it is essential and required. It’s ok that you don’t like having to work nights or weekends or holidays. But some people absolutely have to. Be thankful there are EMTs and surgeons and power and network engineers working instead of being with their families on holidays or in the wee hours of the night.
    - xvector 1y ago
      No one dies if our users can't shitpost until tomorrow morning. I'm glad there are people willing to do oncall. Especially for critical services. But the software engineering profession as a whole would benefit from negotiating concessions for oncall. We have normalized work interfering with life so the company can squeeze a couple extra millions from ads. And for what? Nontrivial amount of ad revenue lost? Not my problem if the company can't pay me to mitigate.
      - disgruntledphd2 1y ago
        > Nontrivial amount of ad revenue lost? Not my problem if the company can't pay me to mitigate. Interestingly, when I worked on analytics around bugs we found that often (in the ads space), there actually wasn't an impact when advertisers were unable to create ads, as they just created all of them when the interface started working again. Now, if it had been the ad serving or pacing mechanisms then it would've been a lot of money, but not all outages are created equal.
      - sneak 1y ago
        Not all websites are for shitposting. I can’t talk to my clients for whom I am on call because Signal is down. I also can’t communicate with my immediate family. There are tons of systems positively critical to society downstream from these services. Some can tolerate downtime. Many can’t.
  - tilolebo 1y ago
    "Your shitposting is very important to us, please stay on the site"
  - einarfd 1y ago
    When I used to be on call for Cisco WebEx services. I got paid extra, and got extra time of. Even if nothing happened. In addition we where enough people on the rotation, so I didn't have to do it that often. I believe the rules varied based on jurisdiction, and I think some had worse deals, and some even better. But I was happy with our setup in Norway. Tbh I do not think we would have had, what we had if it wasn't for the local laws and regulations. Sometimes worker friendly laws can be nice.
    - weakfish 1y ago
      As I was reading the parent, I was thinking “hm, doesn’t match my experience at Cisco!” So it’s funny to see your comment right after.
  - sksksk 1y ago
    It's also unneccesary at large companies, since there'll likely be enough offices globally to have a follow the sun model.
    - decimalenough 1y ago
      Follow the sun does not happen by itself. Very few if any engineering teams are equally split across thirds of the globe in such a way that (say) Asia can cover if both EMEA and the Americas are offline. Having two sites cover the pager is common, but even then you only have 16 working hours at best and somebody has to take the pager early/late.
      - macintux 1y ago
        Not to mention that the knowledge, skills, experience to troubleshoot/recover is rarely evenly distributed across the teams.
  - secondcoming 1y ago
    We get TOIL for being on call.
drevil-v2 1y ago
Easiest day for engineers on-call everywhere except AWS staff. There’s nothing you can do except wait for AWS to come back online. Pour one out for the customer service teams of affected businesses instead
- codeduck 1y ago
  and by one I trust you mean a bottle.
- darkwater 1y ago
  Well, but tomorrow there will be CTOs asking for a contingency plan if AWS goes down, even if planning, preparing, executing and keeping it up to date as the infra evolves will cost more than the X hours of AWS outage. There are certainly organizations for which that cost is lower than the overall damage of services being down due to AWS fault, but tomorrow we will hear CTOs from smaller orgs as well.
  - brazukadev 1y ago
    Lots of NextJS CTOs are gonna need to think about it for the first time too
  - noir_lord 1y ago
    They’ll ask, in a week they’ll have other priorities and in a month they’ll have forgotten about it. This will hold until the next time AWS had a major outage, rinse and repeat.
    - darkwater 1y ago
      It's so true it hurts. If you are new in any infra/platform management position you will be scared as hell this week. Then you will just learn that feeling will just disappear by itself in a few days.
      - noir_lord 1y ago
        Yep, when I was a young programmer I lived in dread of an outage or worse been responsible for a serious bug in production, then I got to watch what happened when it happened to others (and that time I dropped the prod database at half past four on a Friday). When everything is some varying degree of broken at all times been responsible for a brief uptick in the background brokenness isn't the drama you think it is. It would be different if the systems I worked on where true life and death (ATC/Emergency Services etc) but in reality the blast radius from my fucking up somewhere is monetary and even at the biggest company I worked for constrained (while 100+K per hour from an outage sounds horrific - in reality the vast majority of that was made up when the service was back online, people still needed to order the thing in the end).
DebtDeflation 1y ago
>Can't vibe code your way out of this issue... I feel bad for the people impacted by the outage. But at the same time there's a part of me that says we need a cataclysmic event to shake the C-Suite out of their current mindset of laying off all of their workers to replace them with AI, the cheapest people they can find in India, or in some cases with nothing at all, in order to maximize current quarter EPS.
- moomoo11 1y ago
  /ai why is AWS down? can you bring it back up
SilverSlash 1y ago
I expect it's their SREs who are dealing with this mess.

rvz 1y ago

> Can't vibe code your way out of this issue... Exactly. This time, some LLM providers are also down and can't help vibe coders on this issue.

fragmede 1y ago
Qwen3 on lm-studio running fine on my work Mac M3, what's wrong with yours?

LostMyLogin 1y ago

Pour one out for everyone on-call right now.

xvector 1y ago
After some thankless years preventing outages for a big tech company, I will never take an oncall position again in my life. Most miserable working years I have had. It's wild how normalized working on weekends and evenings becomes in teams with oncall. But it's not normal. Our users not being able to shitpost is simply not worth my weekend or evening. And outside of Google you don't even get paid for oncall at most big tech companies! Company losing millions of dollars an hour, but somehow not willing to pay me a dime to jump in at 3AM? Looks like it's not my problem!
- scns 1y ago
  > And outside of Google you don't even get paid for oncall at most big tech companies. What the redacted?
  - redwall_hp 1y ago
    Welcome to the typical American salary abuse. There's even a specific legal cutout exempting information technology, scientific and artistic fields from the overtime pay requirements of the Fair Labor Standards Act. There's a similar cutout for management, which is how companies like GameStop squeeze their retail managers. They just don't give enough payroll hours for regular employees, so the salaried (but poorly paid) manager has to cover all of the gaps.
- sneak 1y ago
  > But this is not normal. Our users not being able to shitpost is simply not worth my weekend or evening. It is completely normal for staff to have to work 24/7 for critical services. Plumbing, HVAC, power plant engineers, doctors, nurses, hospital support staff, taxi drivers, system and network engineers - these people keep our modern world alive, all day, every day. Weekends, midnights, holidays, every hour of every day someone is AT WORK to make sure our society functions. Not only is it normal, it is essential and required. It’s ok that you don’t like having to work nights or weekends or holidays. But some people absolutely have to. Be thankful there are EMTs and surgeons and power and network engineers working instead of being with their families on holidays or in the wee hours of the night.
  - xvector 1y ago
    No one dies if our users can't shitpost until tomorrow morning. I'm glad there are people willing to do oncall. Especially for critical services. But the software engineering profession as a whole would benefit from negotiating concessions for oncall. We have normalized work interfering with life so the company can squeeze a couple extra millions from ads. And for what? Nontrivial amount of ad revenue lost? Not my problem if the company can't pay me to mitigate.
    - disgruntledphd2 1y ago
      > Nontrivial amount of ad revenue lost? Not my problem if the company can't pay me to mitigate. Interestingly, when I worked on analytics around bugs we found that often (in the ads space), there actually wasn't an impact when advertisers were unable to create ads, as they just created all of them when the interface started working again. Now, if it had been the ad serving or pacing mechanisms then it would've been a lot of money, but not all outages are created equal.
    - sneak 1y ago
      Not all websites are for shitposting. I can’t talk to my clients for whom I am on call because Signal is down. I also can’t communicate with my immediate family. There are tons of systems positively critical to society downstream from these services. Some can tolerate downtime. Many can’t.
- tilolebo 1y ago
  "Your shitposting is very important to us, please stay on the site"
- einarfd 1y ago
  When I used to be on call for Cisco WebEx services. I got paid extra, and got extra time of. Even if nothing happened. In addition we where enough people on the rotation, so I didn't have to do it that often. I believe the rules varied based on jurisdiction, and I think some had worse deals, and some even better. But I was happy with our setup in Norway. Tbh I do not think we would have had, what we had if it wasn't for the local laws and regulations. Sometimes worker friendly laws can be nice.
  - weakfish 1y ago
    As I was reading the parent, I was thinking “hm, doesn’t match my experience at Cisco!” So it’s funny to see your comment right after.
- sksksk 1y ago
  It's also unneccesary at large companies, since there'll likely be enough offices globally to have a follow the sun model.
  - decimalenough 1y ago
    Follow the sun does not happen by itself. Very few if any engineering teams are equally split across thirds of the globe in such a way that (say) Asia can cover if both EMEA and the Americas are offline. Having two sites cover the pager is common, but even then you only have 16 working hours at best and somebody has to take the pager early/late.
    - macintux 1y ago
      Not to mention that the knowledge, skills, experience to troubleshoot/recover is rarely evenly distributed across the teams.
- secondcoming 1y ago
  We get TOIL for being on call.

drevil-v2 1y ago

Easiest day for engineers on-call everywhere except AWS staff. There’s nothing you can do except wait for AWS to come back online. Pour one out for the customer service teams of affected businesses instead

codeduck 1y ago
and by one I trust you mean a bottle.
darkwater 1y ago
Well, but tomorrow there will be CTOs asking for a contingency plan if AWS goes down, even if planning, preparing, executing and keeping it up to date as the infra evolves will cost more than the X hours of AWS outage. There are certainly organizations for which that cost is lower than the overall damage of services being down due to AWS fault, but tomorrow we will hear CTOs from smaller orgs as well.
- brazukadev 1y ago
  Lots of NextJS CTOs are gonna need to think about it for the first time too
- noir_lord 1y ago
  They’ll ask, in a week they’ll have other priorities and in a month they’ll have forgotten about it. This will hold until the next time AWS had a major outage, rinse and repeat.
  - darkwater 1y ago
    It's so true it hurts. If you are new in any infra/platform management position you will be scared as hell this week. Then you will just learn that feeling will just disappear by itself in a few days.
    - noir_lord 1y ago
      Yep, when I was a young programmer I lived in dread of an outage or worse been responsible for a serious bug in production, then I got to watch what happened when it happened to others (and that time I dropped the prod database at half past four on a Friday). When everything is some varying degree of broken at all times been responsible for a brief uptick in the background brokenness isn't the drama you think it is. It would be different if the systems I worked on where true life and death (ATC/Emergency Services etc) but in reality the blast radius from my fucking up somewhere is monetary and even at the biggest company I worked for constrained (while 100+K per hour from an outage sounds horrific - in reality the vast majority of that was made up when the service was back online, people still needed to order the thing in the end).

DebtDeflation 1y ago

>Can't vibe code your way out of this issue... I feel bad for the people impacted by the outage. But at the same time there's a part of me that says we need a cataclysmic event to shake the C-Suite out of their current mindset of laying off all of their workers to replace them with AI, the cheapest people they can find in India, or in some cases with nothing at all, in order to maximize current quarter EPS.

moomoo11 1y ago
/ai why is AWS down? can you bring it back up

SilverSlash 1y ago

I expect it's their SREs who are dealing with this mess.