4 ms·

fly.io just has the weirdest outages. It has issues so regularly we dont even need to run mock outages to make sure our system fail overs work.

duxup 2y ago
When I worked for a company who worked with big banks / financial institutions we used to run disaster recovery tests. Effectively a simulated outage where the company would try to run off their backup sites. They ran everything from those sites, it was impressive. Once in a while we'd have a real outage that matched the test we ran as recently as the weekend before. I was helping a bank switch over to the DR site(s) one day during such a real outage and I left my mic open when someone asked me what the commotion was on the upper floors of our HQ. I said "super happy fun surprise disaster recovery test for company X". VP of BIG bank was on the line monitoring and laughed "I'm using that one on the executive call in 15, thanks!" Supposedly it got picked up at the bank internally after the VP made the joke and was an unofficial code for such an outage for a long time.
- NetOpWibby 2y ago
  Thankfully your comment was positive!
- latch 2y ago
  In most BIG banks, "Vice President" is almost an entry-level title. Easily have 1000s of them. For example, this article points out that Goldman Sachs had ~12K VPs out of more than 30K employees: https://web.archive.org/web/20150311012855/https://www.wsj.com/articles/SB10001424052702304692804577281252012689294 https://web.archive.org/web/20150311012855/https://www.wsj.c...
  - jart 2y ago
    VP at Goldman is equivalent to Senior SWE according to levels.fyi and their entry level is Analyst. I'm surprised by the compensation though. I would have thought people working at a place with gold in the name would be making more. Also apparently Morgan Stanley pays their VPs $67k/year.
    - philipwhiuk 2y ago
      Tech outstripped big finance corps tech a while ago. Traders make loads, not the SWEs
    - bormaj 2y ago
      That VP comp number seems quite low fwiw
      - jart 2y ago
        Yes how much longer till we see Morgan Stanley VPs picketing outside demanding a living wage and humming The Internationale.
  - SteveNuts 2y ago
    Just like all Sales folks have heavily inflated titles, no customer wants to think they're dealing with a junior salesperson/loan officer when you're about to hand over your money. It seems like every vendor sales team I work with is an "executive" or "director of sales" even though in reality they're just regular old salespeople.
benreesman 2y ago
In fairness to the fly.io folks (who are extremely serious hackers), they’re standing up a whole cloud provider and they’ve priced it attractively and they’re much customer-friendlier than most alternatives. I don’t envy the difficulty of doing this, but I’m quite confident they’ll iron the bugs out.
- redslazer 2y ago
  The tech is impressive and the pricing is attractive which is why we use them. I just wish there was less black magic.
  - benreesman 2y ago
    I don’t always agree with @tptacek on social/political issues, and I don’t always agree with @xe on the direction of Nix, but these are legends on the technical side of things. And they’re trying to build an equitable relationship between the user of cloud services and the provider, not fund a private space program. If I were in the market for cloud services I’d highly prize a long-term relationship on mutual benefit and fair dealings over a short-term nuisance of being an early adopter. I strongly suspect your investment in fly is going to pay off.
    - throwaway984393 2y ago
      [dead]
    - verelo 2y ago
      I want to believe, but in the meantime they’re killing the product I’ve been working hard to build trust with my own customers though. There is a limit to my idealism, and it’s well and truly in the past.
    - reissbaker 2y ago
      FWIW Xe was let go from Fly earlier this year during a round of layoffs.
deleted 2y ago
[deleted]

duxup 2y ago

When I worked for a company who worked with big banks / financial institutions we used to run disaster recovery tests. Effectively a simulated outage where the company would try to run off their backup sites. They ran everything from those sites, it was impressive. Once in a while we'd have a real outage that matched the test we ran as recently as the weekend before. I was helping a bank switch over to the DR site(s) one day during such a real outage and I left my mic open when someone asked me what the commotion was on the upper floors of our HQ. I said "super happy fun surprise disaster recovery test for company X". VP of BIG bank was on the line monitoring and laughed "I'm using that one on the executive call in 15, thanks!" Supposedly it got picked up at the bank internally after the VP made the joke and was an unofficial code for such an outage for a long time.

NetOpWibby 2y ago
Thankfully your comment was positive!
latch 2y ago
In most BIG banks, "Vice President" is almost an entry-level title. Easily have 1000s of them. For example, this article points out that Goldman Sachs had ~12K VPs out of more than 30K employees: https://web.archive.org/web/20150311012855/https://www.wsj.com/articles/SB10001424052702304692804577281252012689294 https://web.archive.org/web/20150311012855/https://www.wsj.c...
- jart 2y ago
  VP at Goldman is equivalent to Senior SWE according to levels.fyi and their entry level is Analyst. I'm surprised by the compensation though. I would have thought people working at a place with gold in the name would be making more. Also apparently Morgan Stanley pays their VPs $67k/year.
  - philipwhiuk 2y ago
    Tech outstripped big finance corps tech a while ago. Traders make loads, not the SWEs
  - bormaj 2y ago
    That VP comp number seems quite low fwiw
    - jart 2y ago
      Yes how much longer till we see Morgan Stanley VPs picketing outside demanding a living wage and humming The Internationale.
- SteveNuts 2y ago
  Just like all Sales folks have heavily inflated titles, no customer wants to think they're dealing with a junior salesperson/loan officer when you're about to hand over your money. It seems like every vendor sales team I work with is an "executive" or "director of sales" even though in reality they're just regular old salespeople.

benreesman 2y ago

In fairness to the fly.io folks (who are extremely serious hackers), they’re standing up a whole cloud provider and they’ve priced it attractively and they’re much customer-friendlier than most alternatives. I don’t envy the difficulty of doing this, but I’m quite confident they’ll iron the bugs out.

redslazer 2y ago
The tech is impressive and the pricing is attractive which is why we use them. I just wish there was less black magic.
- benreesman 2y ago
  I don’t always agree with @tptacek on social/political issues, and I don’t always agree with @xe on the direction of Nix, but these are legends on the technical side of things. And they’re trying to build an equitable relationship between the user of cloud services and the provider, not fund a private space program. If I were in the market for cloud services I’d highly prize a long-term relationship on mutual benefit and fair dealings over a short-term nuisance of being an early adopter. I strongly suspect your investment in fly is going to pay off.
  - throwaway984393 2y ago
    [dead]
  - verelo 2y ago
    I want to believe, but in the meantime they’re killing the product I’ve been working hard to build trust with my own customers though. There is a limit to my idealism, and it’s well and truly in the past.
  - reissbaker 2y ago
    FWIW Xe was let go from Fly earlier this year during a round of layoffs.

deleted 2y ago

[deleted]