7 ms·

After all the chatter this week, I've come to the conclusion that Heroku froze at the perfect time for my 4 person company. All of these so called "features" ar

After all the chatter this week, I've come to the conclusion that Heroku froze at the perfect time for my 4 person company. All of these so called "features" are exactly what we don't want or need. 1. Multi-region deployment only work if your database is globally distributed too. However, making your database globally distributed creates a set of new problems, most of which take time away from your core business. 2. File persistence is fine but not typically necessary. S3 works just fine. It's easy to forget that most companies are a handful of people or just solo devs. At the same time, most money comes from the enterprise, so products that reach sufficient traction tend to shift their focus to serving the needs of these larger clients. I'm really glad Heroku froze when it did. Markets always demand growth at all costs, and I find it incredibly refreshing that Heroku ended up staying in its lane. IMO it was and remains the best PaaS for indie devs and small teams.

tomjakubowski 4y ago
> Multi-region deployment only work if your database is globally distributed too. However, making your database globally distributed creates a set of new problems, most of which take time away from your core business. Guess what? fly.io offers a turnkey distributed/replicated Postgres for just this reason. You use an HTTP header to route writes to the region hosting your primary. https://fly.io/docs/getting-started/multi-region-databases/ https://fly.io/docs/getting-started/multi-region-databases/ You do still need to consider the possibility of read replicas being behind the primary when designing your application. If your design considers that from day 1, I think it takes less away from solving your business problems. Alternatively, you can also just ignore all the multi-region stuff and deploy to one place, as if it was old-school Heroku :-)
- nickjj 4y ago
  > Guess what? fly.io offers a turnkey distributed/replicated Postgres for just this reason. You use an HTTP header to route writes to the region hosting your primary. Doesn't this take away a lot of the benefits of global distribution? For example if you pay Fly hundreds of dollars a month to distribute your small app in a few datacenters around the globe but your primary DB is in California then everyone from the EU is going to have about 150-200ms round trip latency every time you write to your DB because you can't get around the limitations of the speed of light. Now we're back to non-distributed latency times every time you want to write to the DB which is quite often in a lot of types of apps. If you want to cache mostly static read-only pages at the CDN level you can do this with a number of services. Fly has about 20 datacenters, hosting a small'ish web app that's distributed across them will be over $200 / month without counting extra storage or bandwidth just for the web app portion. Their pg pricing isn't clear but a fairly small cluster is $33.40 / month for 2GB of memory and 40GB of storage. Based on their pricing page it sounds like that's the cost for 1 datacenter, so if you wanted read-replicas in a bunch of other places it adds up. Before you know it you might be at $500 / month to host something that will have similar latency on DB writes as a $20 / month DigitalOcean server that you self manage, Fly also charges you $2 / month per Let's Encrypt wildcard cert where as that's free from Let's Encrypt directly.
  - deleted 4y ago
    [deleted]
  - anamexis 4y ago
    Yes, there are hundreds of different ways you could accomplish this. Fly.io is a convenient and easy to use one.
  - manmal 4y ago
    You don’t need to route every write to primary though, but only those writes that have dependencies on other writes. Things like telemetry can be written in edge instances. Depends on your application of course, but in many cases that should be only a tiny fraction of all requests needing redirects to primary. And why would you get 20 instances, all around the world right out of the gate? 6-7 probably do the job quite well, but maybe you don’t even need that many. Depending on where most of your customers are, you could get good results with 3-4 for most users.
    - nickjj 4y ago
      > You don’t need to route every write to primary though, but only those writes that have dependencies on other writes. Thanks, can you give an example of how that works? Did you write your own fork of Postgres or are you using a third party solution like BDR? Also do you have a few use cases where you'd want writes being dependent on another write? > 6-7 probably do the job quite well You could, let's call it 5. For a 2gb set up would that be about $50 for the web app, $50 for the background workers, $160ish for postgres and then $50 for Redis? We're still at $300+? I was thinking maybe 5 background workers wasn't necessary but frameworks like Rails will put a bunch of things through a background worker where you would want low latency even if they're happening in the background because it's not only things like sending an email where it doesn't matter if it's delayed for 2 seconds behind the scenes. It's performing various Hotwire Turbo actions which render templates and modify records where you'd want to see those things reflected in the web UI as soon as possible.
      - nicoburns 4y ago
        For low-latency workers like that it might make sense to just run them on the same instance as the web servers.
sanjayio 4y ago
I’m always confused why edge services are always selling points given point 1. The most basic of backend services won’t be able to completely utilize edge services.
- anon3949494 4y ago
  Yep, for anyone confused on how this works: You'd still be sending writes to a single region (leader). If the leader is located across the world from the request's origin, there will be a significant latency. Not to mention you need to wait for that write to replicate across the world before it becomes generally available.
  - mrkurt 4y ago
    This is the distribute-your-Rails-app-without-making-any-code-changes version of that story. It works great for apps that are 51% or more read heavy. You drop our library in, add a region, and off you go. The library takes care of eventual consistency issues. HTTP requests that write to the DB are basically the same speed as "Heroku, but in one place". If you're building infrastructure for all the full stack devs you can target, this is a good way to do it. Distributing write heavy work loads is an application architecture problem. You can do it with something like CockroachDB, but you have to model your data specifically to solve that problem. We have maybe 5 customers who've made that leap. In our experience, people get a huge boost from read replicas without needing to change their app (or learn to model data for geo-distribution).
    - anon3949494 4y ago
      It's also trivial to serve read requests from a caching layer or via a CDN. At any sufficient scale, you're probably going to need a CDN anyway, whether your database is replicated or not. You don't want every read to hit your database.
      - jrockway 4y ago
        I don't think this is that trivial. I've never seen it done correctly. It typically manifests itself as not being able to read your own writes, and I see this all the time (often from companies that have blog posts about how smart their caching algorithm is). For example, you add something to a list, then you're redirected to the list, and it's not there. Then you press refresh and it is. I guess that's acceptable because people don't really look for the feedback; why do users add the same thing to the list twice, why does everyone hit the refresh button after adding an item to the list, etc. It's because the bug happens after the user is committed to using your service (contract, cost of switching too high; so you don't see adding the cache layer correspond to "churn"), and that it's annoying but not annoying enough to file a support ticket (so you don't see adding the cache layer correspond to increased support burden). All I can say is, be careful. I wouldn't annoy my users to save a small amount of money. That the industry as a whole is oblivious to quality doesn't mean that it's okay for you to be oblivious about quality. (Corollary: relaxing the transactional isolation level on your database to increase performance is very hard to reason about it. Do some tests and your eyes will pop out of your head.)
twblalock 4y ago
Even small companies should be multi-region, if they care about uptime.
- ceejayoz 4y ago
  Very few companies have uptime requirements so critical they can justify this. Small companies with limited ops resources may struggle to make a multi-region setup work more reliably than a single-region one.
- tomnipotent 4y ago
  No, they shouldn't. In many instances it's cheaper to tolerate downtime than to pay to avoid it, especially when there's no SLA involved.
  - treeman79 4y ago
    Most of the time. If heroku is having downtime. Then Amazon is having downtime. Then half the internet is down. Let customers know Amazon is down. Sit back and relax.
    - tomnipotent 4y ago
      Uptime isn't an axiom. Most software isn't mission critical and most users won't notice if it's down for 30 minutes once or twice a month, and for everything else we have SLA's to manage professional expectations.
    - nosvince 4y ago
      Wow, that's a horrible way of thinking about the user experience. And honestly, I'm not surprised. That's why companies that really care about the user experience will always steal market share from those that don't.
      - neojebfkekeej 4y ago
        It’s actually small companies that care about user experience that will often make these trade-offs. Less time managing multi-cloud deployments means more time spent building our core product and talking to users.
closeparen 4y ago
Hot take: if people spent half the energy doing multi-region that they today spend screwing around with Kubernetes, they’d be a hell of a lot more reliable.
- jpgvm 4y ago
  I think people misconstrue the benefits of k8s to be related to reliability or similar. Ultimately it's about the API and the consistency and productivity it offers. For larger teams having a well defined API that delineates applications from infrastructure that doesn't require extreme specialist knowledge (it still requires some specialist knowledge but vastly less than direct manipulation of resources via something like Terraform) is a massive productivity boost. Of course none of that matters if you have 4 developers like OP but for folks like myself that routinely end up at places with 300+ engineers then it's a huge deal.
  - Trasmatta 4y ago
    > I think people misconstrue the benefits of k8s to be related to reliability or similar. Ultimately it's about the API and the consistency and productivity it offers I think this is the first time I've heard somebody say one of the benefits of kubernetes was productivity.
    - vlunkr 4y ago
      Really? I think it's a pretty obvious benefit. If you bundle something into a container, you can probably run it in kubernetes. This uniformity makes it incredibly easy to deploy and scale new applications.
      - otterley 4y ago
        The same is true of ECS, but with a much simpler API, much tighter integration with load balancers, a no-charge control plane, and not having to upgrade every cluster every 3-6 months.
donmcronald 4y ago
> It's easy to forget that most companies are a handful of people or just solo devs. I have the same complaint all the way down to simple sysadmin tasks. Ex: MS365 has a lot of churn on features and changes. It’s like they think everyone has a team of admins for it when in reality a lot of small businesses would be satisfied with a simple, email only product they can manage without help.
hthrowaway5 4y ago
As someone that was a very active Heroku user for years and then worked there for years: I wouldn't trust it as my host. There is nowhere near enough people maintaining it in order to have confidence it'll run without reliability or security issues. They aren't exactly in a position to retain or attract talent either. I thought Cedar was going to fall over years ago but ironically I think people migrating off the platform are helping it stay alive.
iammiles 4y ago
I strongly agree with your last paragraph. I used Heroku for my wedding website and I would 100% use it again on a project site. In about 15 minutes I was able to take my site from localhost to a custom domain with SSL with just a little more than a git push. I can't think of many solutions that are simpler than that.
- teaearlgraycold 4y ago
  Vercel and Github pages would be better IMO if it's a static site.
  - snoman 4y ago
    If it’s a static site then just dump it in an s3 bucket and be on your way.
    - manigandham 4y ago
      Static websites have their own issues but an S3 bucket is probably the worst hosting mechanism for them these days. The other services mentioned are much nicer and easier to deal with.
- bastawhiz 4y ago
  This is literally Netlify's core offering.
throwaway892238 4y ago
> 2. File persistence is fine but not typically necessary. S3 works just fine. I'm so glad you pointed this out. Cloud-native development is an important factor in newly architected systems. Defaulting to an S3 API for persistent I/O brings loads of benefits over using traditional file I/O, and brings significant new design considerations. Until a majority of software developers learn exactly how and why to use these new designs, we'll be stuck with outmoded platforms catering to old designs.
smt88 4y ago
> Multi-region deployment only work if your database is globally distributed too. However, making your database globally distributed creates a set of new problems, most of which take time away from your core business. I have used multi-region for every production database I've deployed in the last ~8 years, and it took < 10 seconds of extra time. It's a core feature of services like RDS on AWS. There is a benefit if you're multi-region (but not global) because individual regions go down all the time. It costs more every month, but if you have a B2B business, it's worth the extra cost.
- DylanSp 4y ago
  For RDS, are you talking about multi-region, or multi-AZ? I know the latter is easy, but I don't think the first is, though maybe Aurora makes it easier.
  - smt88 4y ago
    Multi-region, not just multi-AZ[1]. 1. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.XRgn.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...
    - DylanSp 4y ago
      Oh, that's handy. Thanks for sharing!
amerine 4y ago
We’re not “froze”. The last week has been exhausting fud.
- hnthrow124 4y ago
  That’s a lie. The official freeze for new features started in 2017 and customers can see themselves by reading the changelog.

tomjakubowski 4y ago

> Multi-region deployment only work if your database is globally distributed too. However, making your database globally distributed creates a set of new problems, most of which take time away from your core business. Guess what? fly.io offers a turnkey distributed/replicated Postgres for just this reason. You use an HTTP header to route writes to the region hosting your primary. https://fly.io/docs/getting-started/multi-region-databases/ https://fly.io/docs/getting-started/multi-region-databases/ You do still need to consider the possibility of read replicas being behind the primary when designing your application. If your design considers that from day 1, I think it takes less away from solving your business problems. Alternatively, you can also just ignore all the multi-region stuff and deploy to one place, as if it was old-school Heroku :-)

nickjj 4y ago
> Guess what? fly.io offers a turnkey distributed/replicated Postgres for just this reason. You use an HTTP header to route writes to the region hosting your primary. Doesn't this take away a lot of the benefits of global distribution? For example if you pay Fly hundreds of dollars a month to distribute your small app in a few datacenters around the globe but your primary DB is in California then everyone from the EU is going to have about 150-200ms round trip latency every time you write to your DB because you can't get around the limitations of the speed of light. Now we're back to non-distributed latency times every time you want to write to the DB which is quite often in a lot of types of apps. If you want to cache mostly static read-only pages at the CDN level you can do this with a number of services. Fly has about 20 datacenters, hosting a small'ish web app that's distributed across them will be over $200 / month without counting extra storage or bandwidth just for the web app portion. Their pg pricing isn't clear but a fairly small cluster is $33.40 / month for 2GB of memory and 40GB of storage. Based on their pricing page it sounds like that's the cost for 1 datacenter, so if you wanted read-replicas in a bunch of other places it adds up. Before you know it you might be at $500 / month to host something that will have similar latency on DB writes as a $20 / month DigitalOcean server that you self manage, Fly also charges you $2 / month per Let's Encrypt wildcard cert where as that's free from Let's Encrypt directly.
- deleted 4y ago
  [deleted]
- anamexis 4y ago
  Yes, there are hundreds of different ways you could accomplish this. Fly.io is a convenient and easy to use one.
- manmal 4y ago
  You don’t need to route every write to primary though, but only those writes that have dependencies on other writes. Things like telemetry can be written in edge instances. Depends on your application of course, but in many cases that should be only a tiny fraction of all requests needing redirects to primary. And why would you get 20 instances, all around the world right out of the gate? 6-7 probably do the job quite well, but maybe you don’t even need that many. Depending on where most of your customers are, you could get good results with 3-4 for most users.
  - nickjj 4y ago
    > You don’t need to route every write to primary though, but only those writes that have dependencies on other writes. Thanks, can you give an example of how that works? Did you write your own fork of Postgres or are you using a third party solution like BDR? Also do you have a few use cases where you'd want writes being dependent on another write? > 6-7 probably do the job quite well You could, let's call it 5. For a 2gb set up would that be about $50 for the web app, $50 for the background workers, $160ish for postgres and then $50 for Redis? We're still at $300+? I was thinking maybe 5 background workers wasn't necessary but frameworks like Rails will put a bunch of things through a background worker where you would want low latency even if they're happening in the background because it's not only things like sending an email where it doesn't matter if it's delayed for 2 seconds behind the scenes. It's performing various Hotwire Turbo actions which render templates and modify records where you'd want to see those things reflected in the web UI as soon as possible.
    - nicoburns 4y ago
      For low-latency workers like that it might make sense to just run them on the same instance as the web servers.

sanjayio 4y ago

I’m always confused why edge services are always selling points given point 1. The most basic of backend services won’t be able to completely utilize edge services.

anon3949494 4y ago
Yep, for anyone confused on how this works: You'd still be sending writes to a single region (leader). If the leader is located across the world from the request's origin, there will be a significant latency. Not to mention you need to wait for that write to replicate across the world before it becomes generally available.
- mrkurt 4y ago
  This is the distribute-your-Rails-app-without-making-any-code-changes version of that story. It works great for apps that are 51% or more read heavy. You drop our library in, add a region, and off you go. The library takes care of eventual consistency issues. HTTP requests that write to the DB are basically the same speed as "Heroku, but in one place". If you're building infrastructure for all the full stack devs you can target, this is a good way to do it. Distributing write heavy work loads is an application architecture problem. You can do it with something like CockroachDB, but you have to model your data specifically to solve that problem. We have maybe 5 customers who've made that leap. In our experience, people get a huge boost from read replicas without needing to change their app (or learn to model data for geo-distribution).
  - anon3949494 4y ago
    It's also trivial to serve read requests from a caching layer or via a CDN. At any sufficient scale, you're probably going to need a CDN anyway, whether your database is replicated or not. You don't want every read to hit your database.
    - jrockway 4y ago
      I don't think this is that trivial. I've never seen it done correctly. It typically manifests itself as not being able to read your own writes, and I see this all the time (often from companies that have blog posts about how smart their caching algorithm is). For example, you add something to a list, then you're redirected to the list, and it's not there. Then you press refresh and it is. I guess that's acceptable because people don't really look for the feedback; why do users add the same thing to the list twice, why does everyone hit the refresh button after adding an item to the list, etc. It's because the bug happens after the user is committed to using your service (contract, cost of switching too high; so you don't see adding the cache layer correspond to "churn"), and that it's annoying but not annoying enough to file a support ticket (so you don't see adding the cache layer correspond to increased support burden). All I can say is, be careful. I wouldn't annoy my users to save a small amount of money. That the industry as a whole is oblivious to quality doesn't mean that it's okay for you to be oblivious about quality. (Corollary: relaxing the transactional isolation level on your database to increase performance is very hard to reason about it. Do some tests and your eyes will pop out of your head.)

twblalock 4y ago

Even small companies should be multi-region, if they care about uptime.

ceejayoz 4y ago
Very few companies have uptime requirements so critical they can justify this. Small companies with limited ops resources may struggle to make a multi-region setup work more reliably than a single-region one.
tomnipotent 4y ago
No, they shouldn't. In many instances it's cheaper to tolerate downtime than to pay to avoid it, especially when there's no SLA involved.
- treeman79 4y ago
  Most of the time. If heroku is having downtime. Then Amazon is having downtime. Then half the internet is down. Let customers know Amazon is down. Sit back and relax.
  - tomnipotent 4y ago
    Uptime isn't an axiom. Most software isn't mission critical and most users won't notice if it's down for 30 minutes once or twice a month, and for everything else we have SLA's to manage professional expectations.
  - nosvince 4y ago
    Wow, that's a horrible way of thinking about the user experience. And honestly, I'm not surprised. That's why companies that really care about the user experience will always steal market share from those that don't.
    - neojebfkekeej 4y ago
      It’s actually small companies that care about user experience that will often make these trade-offs. Less time managing multi-cloud deployments means more time spent building our core product and talking to users.

closeparen 4y ago

Hot take: if people spent half the energy doing multi-region that they today spend screwing around with Kubernetes, they’d be a hell of a lot more reliable.

jpgvm 4y ago
I think people misconstrue the benefits of k8s to be related to reliability or similar. Ultimately it's about the API and the consistency and productivity it offers. For larger teams having a well defined API that delineates applications from infrastructure that doesn't require extreme specialist knowledge (it still requires some specialist knowledge but vastly less than direct manipulation of resources via something like Terraform) is a massive productivity boost. Of course none of that matters if you have 4 developers like OP but for folks like myself that routinely end up at places with 300+ engineers then it's a huge deal.
- Trasmatta 4y ago
  > I think people misconstrue the benefits of k8s to be related to reliability or similar. Ultimately it's about the API and the consistency and productivity it offers I think this is the first time I've heard somebody say one of the benefits of kubernetes was productivity.
  - vlunkr 4y ago
    Really? I think it's a pretty obvious benefit. If you bundle something into a container, you can probably run it in kubernetes. This uniformity makes it incredibly easy to deploy and scale new applications.
    - otterley 4y ago
      The same is true of ECS, but with a much simpler API, much tighter integration with load balancers, a no-charge control plane, and not having to upgrade every cluster every 3-6 months.

donmcronald 4y ago

> It's easy to forget that most companies are a handful of people or just solo devs. I have the same complaint all the way down to simple sysadmin tasks. Ex: MS365 has a lot of churn on features and changes. It’s like they think everyone has a team of admins for it when in reality a lot of small businesses would be satisfied with a simple, email only product they can manage without help.

hthrowaway5 4y ago

As someone that was a very active Heroku user for years and then worked there for years: I wouldn't trust it as my host. There is nowhere near enough people maintaining it in order to have confidence it'll run without reliability or security issues. They aren't exactly in a position to retain or attract talent either. I thought Cedar was going to fall over years ago but ironically I think people migrating off the platform are helping it stay alive.

iammiles 4y ago

I strongly agree with your last paragraph. I used Heroku for my wedding website and I would 100% use it again on a project site. In about 15 minutes I was able to take my site from localhost to a custom domain with SSL with just a little more than a git push. I can't think of many solutions that are simpler than that.

teaearlgraycold 4y ago
Vercel and Github pages would be better IMO if it's a static site.
- snoman 4y ago
  If it’s a static site then just dump it in an s3 bucket and be on your way.
  - manigandham 4y ago
    Static websites have their own issues but an S3 bucket is probably the worst hosting mechanism for them these days. The other services mentioned are much nicer and easier to deal with.
bastawhiz 4y ago
This is literally Netlify's core offering.

throwaway892238 4y ago

> 2. File persistence is fine but not typically necessary. S3 works just fine. I'm so glad you pointed this out. Cloud-native development is an important factor in newly architected systems. Defaulting to an S3 API for persistent I/O brings loads of benefits over using traditional file I/O, and brings significant new design considerations. Until a majority of software developers learn exactly how and why to use these new designs, we'll be stuck with outmoded platforms catering to old designs.

smt88 4y ago

> Multi-region deployment only work if your database is globally distributed too. However, making your database globally distributed creates a set of new problems, most of which take time away from your core business. I have used multi-region for every production database I've deployed in the last ~8 years, and it took < 10 seconds of extra time. It's a core feature of services like RDS on AWS. There is a benefit if you're multi-region (but not global) because individual regions go down all the time. It costs more every month, but if you have a B2B business, it's worth the extra cost.

DylanSp 4y ago
For RDS, are you talking about multi-region, or multi-AZ? I know the latter is easy, but I don't think the first is, though maybe Aurora makes it easier.
- smt88 4y ago
  Multi-region, not just multi-AZ[1]. 1. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.XRgn.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...
  - DylanSp 4y ago
    Oh, that's handy. Thanks for sharing!

amerine 4y ago

We’re not “froze”. The last week has been exhausting fud.

hnthrow124 4y ago
That’s a lie. The official freeze for new features started in 2017 and customers can see themselves by reading the changelog.