5 ms·

Unfortunately for fly, us-east-1 has better reliability then their product.

tptacek 3y ago
We're not. For a company of our age and scale? To claim competitive reliability to the world's most important data center? Risible. We're making fun of the fact that everybody deploys their apps in a random city in Virginia with like 3 restaurants in it. We're not saying they're dumb to do it, or that Ashburn isn't up to the challenge --- it self-evidently is. (The comment I'm responding to said something about people in glass houses not throwing stones, but was then edited).
- candiddevmike 3y ago
  You guys may want to rebrand once you get your issues sorted out. Fly is going to have the same problem as btrfs, where folks will have a hard time deciding if it's ready for production yet.
  - tptacek 3y ago
    It's comments like this that moved me to claim, in our infra/ops job post, the benefit "The most lovable user base in the whole technology industry, and a company culture that will keep you in touch with them directly."
    - culi 3y ago
      Fly.io was the darling of the webhosting world amongst techworkers particularly because of the honest transparent communication and blog posts explicitly admitting fuck ups Raising a bunch of money just isn't a "good" announcement to most of us. When unpopular things have happened with the company, your communication usually directly acknowledges and focuses on the concerns of the userbase. This serves two purposes: 1. It gives you a chance to show concrete steps to prevent specific concerns we have 2. Perhaps more importantly, it shows we're all on the same page and you're actually in touch with how your userbase thinks Failing on #1 is something you can fix later, but I've never seen a company rebound from losing the trust gained by #2. It loses confidence that future decisions made can also be equally out of touch. I've never seen a company rebound from losing that confidence
      - tptacek 3y ago
        If we're "honest" and "transparent", it's not a conscious strategy to sign up customers. It's just the easiest way for us to write. Meanwhile: we're a hardware-based global hosting company. We've always been externally funded. That's how you build a global hardware-backed hosting company, because servers and colocation and infra engineers are expensive. There are things that happen that disappoint customers that we care a lot about. Reliability issues several months ago absolutely fit that bill. So does hiring: we paused hiring for most of our engineering roles because we weren't happy with the experience candidates were getting. I don't know how "honest" and "transparent" we are about those things, but I do feel like when we're asked about them, we give straight answers. There are other things that happen that disappoint people that we don't care so much about. Being unhappy that we took external funding to build out our fleet and improve reliability would count as one of those things. Again with the straight answers: I wouldn't expect us to spend too much time trying to "rebound" from this announcement.
      - PKop 3y ago
        >Raising a bunch of money just isn't a "good" announcement to most of us Speak for yourself. Announcements of adequate funding inspires confidence fly will have longevity and resources to scale. Other concerns can be tackled in other blog posts, but nothing about the announcement of funding should inspire less confidence, what sense does that make?
        mananaysiempre 3y ago
        The issue is not raising money, precisely, the issue is raising money by selling large amounts of controlling equity (as is usually the case for VC funding and IPOs, with a few notable exceptions). The purchasers will expect their money’s worth, and they will (eventually) not hesitate to exercise the control it bought them in order to get that. VCs in particular will expect much more than their money’s worth, or (most of the time) for the company to die trying to earn it, and they won’t hesitate to force the latter in pursuit of the former. This increase in variance is literally their investment model. So if VCs expect their investments to fail most of the time, and I see a company take VC money, I must have very strong reasons if I’m to disbelieve the VCs and expect the company to succeed—and I’m not following a venture investment strategy when I choose a tech supplier. IPOs are somewhat milder in their intrinsic character, but they still usually lead to professional boards of directors and eventually professional execs, where “professional” usually means “no subject matter expertise”. And hardly anybody runs a public company on customer satisfaction nowadays, unfortunately. So when I see controlling equity sold, my outlook changes from “will this go to shit?” to “when this inevitably goes to shit, will it be in the distant enough future that I don’t care or can plan for the loss?” (that being maybe a decade or so for infra, or implausibly long—like the rest of my natural life—for anything involving a community). As usual, the question remains—why does none of this seem to be priced in? I don’t know.
- hgsgm 3y ago
  Does PR know you are posting this? Number of restaurants is a measurement of hosting quality? Uptime doesn't matter?
  - tptacek 3y ago
    This is neither here nor there, but every time I read a comment like this, all I can think about is that scene in The Office where Dwight Schrute is screaming into his cell phone "Are you saying you invented paper?!"
    - xNeil 3y ago
      Yeah this is pretty funny - if you speak to PR before posting something, it's 'PR-riddled bull** written by some executive', or 'corporate talk'. If you don't speak to PR before posting something, it's 'Did you speak to PR before posting this?' There's no winning.
    - robertlagrant 3y ago
      The Office? Dwight Schrute? ARE YOU SAYING YOU INVENTED THE OFFICE?
  - deleted 3y ago
    [deleted]
  - ralgozino 3y ago
    the whole point is deploying closer to your users, unless your customer base is the owners of those restaurants, it would be better to deploy to other (even all!) regions
- alex_lav 3y ago
  To be fair (this comment is mostly a joke), Ashburn/Loudoun County is pretty nice for a suburb/strip mall hell. I've had some really good food in that area.
- rirze 3y ago
  Ashburn does not have "like three restaurants". The datacenter zoned part of the city has "like three restaurants". If you ever drove >5 minutes from those large shed warehouses, you'd know what you're talking about.
  - mike_d 3y ago
    You are probably thinking of Sterling, Chantilly, etc. Ashburn is literally like the Embassy Suites next to the Five Guys, the Aloft next to the taco place, Equinix, Digital Reality, Verizon, NTT, CenturyLink, etc.
    - rirze 3y ago
      I lived here my entire life, I know where Sterling is and how it's a 15 minute drive from there to Chantilly (which is not in Loudoun County btw). Take a look at this map here: https://www.loudoun.gov/DocumentCenter/View/171141/2022ElectionDistricts https://www.loudoun.gov/DocumentCenter/View/171141/2022Elect... Ashburn is not literally just one shopping plaza. It's a giant area zoned into different parts of datacenter, suburbs, and commercial sections. Even forgetting all of this, just type "restaurants in Ashburn" into Google Maps see the plethora of results. Just because someone has some imaginary boundary of what constitutes as Ashburn doesn't make it definitively small.
      - tptacek 3y ago
        As you can see from the post, we did that. :)
- urda 3y ago
  So you fuss at your users who have valid complaints about the availability of your product? It's a valid complaint when the uptime of a service, program, or feature running is critical to your business. I'm really concerned that Fly's employees like yourself think availability and service uptime isn't important. I think I'll be avoiding Fly on this alone.
  - dcow 3y ago
    > I'm really concerned that Fly's employees like yourself think availability and service uptime isn't important. I'm confused, when did any of Fly's employees say as much?
nickpeterson 3y ago
Well if you’re right maybe in a couple years we’ll see a ‘We razed a bunch of money’ post, seems on brand.
- mrkurt 3y ago
  I was planning an "Our incredible journey" post, but "We razed a bunch of money" is much better.
  - nickpeterson 3y ago
    It’s from the Joker school of money management. Although I believe he only burns his half.
  - deleted 3y ago
    [deleted]
jchw 3y ago
Honestly though, Fly.io feels like it's off into the future in terms of hosting platforms. Undoubtedly they'll have to work on the reliability problems, not denying that - however, for anyone that can tolerate a bit of reliability issues, which to be honest, is more people than will readily admit it, it might just be worth the trouble for developer experience that legitimately feels like it's from the future.
- candiddevmike 3y ago
  What good is a product like this if it's not stable? Most folks can yolo a global infrastructure if they disregard SLAs...
  - jchw 3y ago
    It's definitely not unusably unstable, at least from my PoV, and in fact, I haven't really been hit by most of the instability personally; or at least, if I have, it has gone mostly unnoticed by me. I had an issue where I couldn't deploy for a while, but the services that were up seemed fine. (Disclaimer: I do not use the Redis/database features.) That said, the point I'm trying to make is that it is in fact a lot nicer to use than simply YOLOing whatever onto servers. This model of PaaS is basically what App Engine should have been. It's difficult to want to go back once you've used it.
arandr0x 3y ago
I've worked on the reliability team at a place that made what they call an "edgy app" (long before fly, so we made our own stuff on baremetal) and to be fair, reliability is the actual challenge when you're talking multi-region seamless deployments of scalable web platforms. It makes or breaks that kind of product at least for the original market. Money doesn't contribute all that much to solving it though. It's usually time to make the early mistakes, continuous improvements, etc. But AWS had several decades to solve this, whereas fly clearly hasn't, so I'm not sure I'd count them out yet.

We're not. For a company of our age and scale? To claim competitive reliability to the world's most important data center? Risible. We're making fun of the fact that everybody deploys their apps in a random city in Virginia with like 3 restaurants in it. We're not saying they're dumb to do it, or that Ashburn isn't up to the challenge --- it self-evidently is. (The comment I'm responding to said something about people in glass houses not throwing stones, but was then edited).

candiddevmike 3y ago
You guys may want to rebrand once you get your issues sorted out. Fly is going to have the same problem as btrfs, where folks will have a hard time deciding if it's ready for production yet.
- tptacek 3y ago
  It's comments like this that moved me to claim, in our infra/ops job post, the benefit "The most lovable user base in the whole technology industry, and a company culture that will keep you in touch with them directly."
  - culi 3y ago
    Fly.io was the darling of the webhosting world amongst techworkers particularly because of the honest transparent communication and blog posts explicitly admitting fuck ups Raising a bunch of money just isn't a "good" announcement to most of us. When unpopular things have happened with the company, your communication usually directly acknowledges and focuses on the concerns of the userbase. This serves two purposes: 1. It gives you a chance to show concrete steps to prevent specific concerns we have 2. Perhaps more importantly, it shows we're all on the same page and you're actually in touch with how your userbase thinks Failing on #1 is something you can fix later, but I've never seen a company rebound from losing the trust gained by #2. It loses confidence that future decisions made can also be equally out of touch. I've never seen a company rebound from losing that confidence
    - tptacek 3y ago
      If we're "honest" and "transparent", it's not a conscious strategy to sign up customers. It's just the easiest way for us to write. Meanwhile: we're a hardware-based global hosting company. We've always been externally funded. That's how you build a global hardware-backed hosting company, because servers and colocation and infra engineers are expensive. There are things that happen that disappoint customers that we care a lot about. Reliability issues several months ago absolutely fit that bill. So does hiring: we paused hiring for most of our engineering roles because we weren't happy with the experience candidates were getting. I don't know how "honest" and "transparent" we are about those things, but I do feel like when we're asked about them, we give straight answers. There are other things that happen that disappoint people that we don't care so much about. Being unhappy that we took external funding to build out our fleet and improve reliability would count as one of those things. Again with the straight answers: I wouldn't expect us to spend too much time trying to "rebound" from this announcement.
    - PKop 3y ago
      >Raising a bunch of money just isn't a "good" announcement to most of us Speak for yourself. Announcements of adequate funding inspires confidence fly will have longevity and resources to scale. Other concerns can be tackled in other blog posts, but nothing about the announcement of funding should inspire less confidence, what sense does that make?
      - mananaysiempre 3y ago
        The issue is not raising money, precisely, the issue is raising money by selling large amounts of controlling equity (as is usually the case for VC funding and IPOs, with a few notable exceptions). The purchasers will expect their money’s worth, and they will (eventually) not hesitate to exercise the control it bought them in order to get that. VCs in particular will expect much more than their money’s worth, or (most of the time) for the company to die trying to earn it, and they won’t hesitate to force the latter in pursuit of the former. This increase in variance is literally their investment model. So if VCs expect their investments to fail most of the time, and I see a company take VC money, I must have very strong reasons if I’m to disbelieve the VCs and expect the company to succeed—and I’m not following a venture investment strategy when I choose a tech supplier. IPOs are somewhat milder in their intrinsic character, but they still usually lead to professional boards of directors and eventually professional execs, where “professional” usually means “no subject matter expertise”. And hardly anybody runs a public company on customer satisfaction nowadays, unfortunately. So when I see controlling equity sold, my outlook changes from “will this go to shit?” to “when this inevitably goes to shit, will it be in the distant enough future that I don’t care or can plan for the loss?” (that being maybe a decade or so for infra, or implausibly long—like the rest of my natural life—for anything involving a community). As usual, the question remains—why does none of this seem to be priced in? I don’t know.
hgsgm 3y ago
Does PR know you are posting this? Number of restaurants is a measurement of hosting quality? Uptime doesn't matter?
- tptacek 3y ago
  This is neither here nor there, but every time I read a comment like this, all I can think about is that scene in The Office where Dwight Schrute is screaming into his cell phone "Are you saying you invented paper?!"
  - xNeil 3y ago
    Yeah this is pretty funny - if you speak to PR before posting something, it's 'PR-riddled bull** written by some executive', or 'corporate talk'. If you don't speak to PR before posting something, it's 'Did you speak to PR before posting this?' There's no winning.
  - robertlagrant 3y ago
    The Office? Dwight Schrute? ARE YOU SAYING YOU INVENTED THE OFFICE?
- deleted 3y ago
  [deleted]
- ralgozino 3y ago
  the whole point is deploying closer to your users, unless your customer base is the owners of those restaurants, it would be better to deploy to other (even all!) regions
alex_lav 3y ago
To be fair (this comment is mostly a joke), Ashburn/Loudoun County is pretty nice for a suburb/strip mall hell. I've had some really good food in that area.
rirze 3y ago
Ashburn does not have "like three restaurants". The datacenter zoned part of the city has "like three restaurants". If you ever drove >5 minutes from those large shed warehouses, you'd know what you're talking about.
- mike_d 3y ago
  You are probably thinking of Sterling, Chantilly, etc. Ashburn is literally like the Embassy Suites next to the Five Guys, the Aloft next to the taco place, Equinix, Digital Reality, Verizon, NTT, CenturyLink, etc.
  - rirze 3y ago
    I lived here my entire life, I know where Sterling is and how it's a 15 minute drive from there to Chantilly (which is not in Loudoun County btw). Take a look at this map here: https://www.loudoun.gov/DocumentCenter/View/171141/2022ElectionDistricts https://www.loudoun.gov/DocumentCenter/View/171141/2022Elect... Ashburn is not literally just one shopping plaza. It's a giant area zoned into different parts of datacenter, suburbs, and commercial sections. Even forgetting all of this, just type "restaurants in Ashburn" into Google Maps see the plethora of results. Just because someone has some imaginary boundary of what constitutes as Ashburn doesn't make it definitively small.
    - tptacek 3y ago
      As you can see from the post, we did that. :)
urda 3y ago
So you fuss at your users who have valid complaints about the availability of your product? It's a valid complaint when the uptime of a service, program, or feature running is critical to your business. I'm really concerned that Fly's employees like yourself think availability and service uptime isn't important. I think I'll be avoiding Fly on this alone.
- dcow 3y ago
  > I'm really concerned that Fly's employees like yourself think availability and service uptime isn't important. I'm confused, when did any of Fly's employees say as much?

nickpeterson 3y ago

Well if you’re right maybe in a couple years we’ll see a ‘We razed a bunch of money’ post, seems on brand.

mrkurt 3y ago
I was planning an "Our incredible journey" post, but "We razed a bunch of money" is much better.
- nickpeterson 3y ago
  It’s from the Joker school of money management. Although I believe he only burns his half.
- deleted 3y ago
  [deleted]

jchw 3y ago

Honestly though, Fly.io feels like it's off into the future in terms of hosting platforms. Undoubtedly they'll have to work on the reliability problems, not denying that - however, for anyone that can tolerate a bit of reliability issues, which to be honest, is more people than will readily admit it, it might just be worth the trouble for developer experience that legitimately feels like it's from the future.

candiddevmike 3y ago
What good is a product like this if it's not stable? Most folks can yolo a global infrastructure if they disregard SLAs...
- jchw 3y ago
  It's definitely not unusably unstable, at least from my PoV, and in fact, I haven't really been hit by most of the instability personally; or at least, if I have, it has gone mostly unnoticed by me. I had an issue where I couldn't deploy for a while, but the services that were up seemed fine. (Disclaimer: I do not use the Redis/database features.) That said, the point I'm trying to make is that it is in fact a lot nicer to use than simply YOLOing whatever onto servers. This model of PaaS is basically what App Engine should have been. It's difficult to want to go back once you've used it.

arandr0x 3y ago

I've worked on the reliability team at a place that made what they call an "edgy app" (long before fly, so we made our own stuff on baremetal) and to be fair, reliability is the actual challenge when you're talking multi-region seamless deployments of scalable web platforms. It makes or breaks that kind of product at least for the original market. Money doesn't contribute all that much to solving it though. It's usually time to make the early mistakes, continuous improvements, etc. But AWS had several decades to solve this, whereas fly clearly hasn't, so I'm not sure I'd count them out yet.