7 ms·

Heroku Postgres Official Maintenance

50 points by rdegges 14y ago

falcolas 14y ago
A good reason to take a few minutes and make sure your backups are up-to-date and working, and that your application won't blow up in unexpected ways when the DB is unreachable.
neya 14y ago
I'm yet to see an update/fix on Heroku's recent PR nightmare on the random routing issue. Anyone?
- coolsunglasses 14y ago
  I'm not sure why this point isn't being made, but they're not going to change anything about how the routing works, least of all for Bamboo. Heroku is past the point of caring about pushing the envelope, stateful routing is an infrastructure risk and overhead that primarily their least informed customers need or want. It's a high margin operation is a low margin business, they're trying not to fuck it up. So stop asking.
  - neya 14y ago
    As a customer, no matter what they do, whether it's a high margin risk or low margin profits, I'm supposed to be informed of their decision - whether they try to fix it or not, because I was one of the affected. They promised they would do so in their previous blog posts, hence the curiosity. >So stop asking. I have every right to ask and I will rightfully so. A disguised internet profile about sunglasses means nothing to me.
- joevandyk 14y ago
  They are about to push an update that lets you run larger dynos, which reduces the risk of having all the requests to a dyno stuck doing something long-lived.
  - redguava 14y ago
    I believe the larger dynos are mostly to better handle using Unicorn as the webserver. Unicorn allows multiple workers to run per dyno, and has it's own smart routing, so it compensates well for Heroku's routing system. The downside to Unicorn was that the workers share a dyno's memory and often you couldn't have more than 2 per dyno with 512mb memory. This change helps that a lot.
- dserodio 14y ago
  They've put together a FAQ about routing: https://blog.heroku.com/archives/2013/4/2/routing_and_web_performance_on_heroku_a_faq https://blog.heroku.com/archives/2013/4/2/routing_and_web_pe...
pilif 14y ago
So. Us normal users will get the security fix on Thursday, but Heroku is somewhat special and gets early access? What does one have to do to become special? I don't like how this is being handled, no matter how serious the issue. I could kind of understand what the postgres team was doing until this point where they show that while everybody is equal, some are more equal than others and get early access to security fixes. Doing this does nothing but motivate the bad guys even more to find the hole on their own. The initial announcement got me scared already, but this just made it worse.
- mrkurt 14y ago
  Heroku is special, they host a huge number of Postgres setups that are publicly accessible and somewhat easily discovered. It's not a matter of equality, though, it's one of trust/privacy. Heroku is an easy company for the Postgres guys to trust since they have a pre-existing, healthy relationship.
  - tensor 14y ago
    Meanwhile they are holding back a security fix for numerous other companies that also take security extremely seriously. This creates a situation where companies considering posgresql will now have to ask "will I get security fixes as soon as they are ready or will I intentionally be left vulnerable while more privileged users get early access?" Not a good precedent as far as I'm concerned.
    - andrewljohnson 14y ago
      It is open source. If you are so worried about the SLA you get from this open source software, maybe you should a) pay for better support or b) fix it yourself.
      - eksith 14y ago
        That's a cheeky cop-out. The original criticism is still valid, and we don't know how many companies outside of Heroku (if any) are getting the patch ahead of time. We don't all have the resources to a) Pay Oracle grade prices for support or b) the man power to comb through millions of lines of code to fix it ourselves. If these are the best excuses for lack of full transparency one could could come up with, then why would anyone choose a Linux distro? Or any of the *BSDs for that matter? Transparency and an expectation of due diligence is why a lot of companies (in IT and elsewhere) still stay in business while choosing Open Source software. Besides, Postgres isn't some run-of-the-mill startup with a dropped letter domain name. It's a venerable software foundation with almost 20 years of work and experience behind it. Which makes this a questionable course of action on their part despite the need for discretionary release.
gingerlime 14y ago
I was just recently considering whether we should move our postgresql over to Heroku. On one hand, it sounds promising that they deal with important security updates, and can do it even before the fix is officially announced. Very impressive. On the other hand, not being able to schedule the downtime to fit with your app/userbase is quite annoying. I understand this could be a potentially big security vulnerability that needs urgent fixing. But each organization is different and their risks are therefore different. Some people might prefer to take the risk of a few hours delay to apply the fix (being exposed) in exchange for not having to experience unscheduled downtime of this nature... It seems a little awkward to me that Heroku is taking this decision for its entire customer-base in such a way.
ryan_f 14y ago
This is rather disconcerting. They are given us a window of 48 hours where our site will be down for 60 seconds. On top of that we cannot schedule the time with them. 60 seconds may seem like a small inconvenience but it is one still the same. I host a few clients on their servers and I need to relay this information. I sound like a jackass being that I can't give more specifics on the timing of the server being down to my clients. How does Heroku not feel the same? I am appreciative of the update but that is also what I pay them to do.
- fein 14y ago
  > How does Heroku not feel the same? Because they, as a hosting provider, have several orders of magnitude more clients to please than yourself. The only way to be fair to everyone is that everyone gets treated like shit (obviously embellished). All you should be doing to CYA is telling your clients exactly why this is happening and that its Heroku, and not you, that holds the blame at this point. The caveat would be those difficult customers (suits, exec managers, etc) that you have to handle with kid gloves. For these people... I don't think there's a graceful way to present the upcoming issue.
  - gingerlime 14y ago
    I doubt customers typically know or care whether you use heroku, Amazon or any other provider. What they want is to see their system up and running. That said, I wonder if with the heroku postgresql setup, this vulnerability could mean that if even one db is compromised, it could allow access to other databases too. Maybe heroku runs several postgresql db's on the same virtual/physical hosts?. If that's the case, then heroku simply can't afford to let even one database stay exposed because it would risk all others. Whether or not this is the case, I don't know, but it kinda feels like running your app on a shared-hosting account...
    - criley 14y ago
      >What they want is to see their system up and running. This would be a good chance to take a step back and think about what's more important: using Heroku or controlling your up-time?
thruflo 14y ago
It may be de-rigueur to bash Heroku in the wake of the Rap Genius drama/revelations but this kind of "downtime to apply critical security patch" announcement is precisely why I choose them. I spotted the announcement of the postgres vulnerability and thought "sounds serious". Now I read that my production databases will be fixed as soon as the patch is available without me having to engage with the details.
nevinera 14y ago
>Due to the nature of this update, a scheduled time is not possible. What, it takes place outside of linear time? This crap is unacceptable. Even my $2.00 shared host tells me when they're going to take down my site without asking.
- viraptor 14y ago
  More likely it takes around 5min per host if everything goes right. If they have 100 hosts and get delayed on the 10th and 15th for some reason, all other scheduled times would get shifted... And the upside is that they spot failures as they appear, rather than "oops, the last 5 hosts we did at the same time do not start up, there's some big issue". </speculation>
JuDue 14y ago
Just as an aside, how good is Heroku's Postgres performance, speed wise?
- guywithabike 14y ago
  Don't ask for anecdotes -- test it yourself with conditions similar to those that you'd be operating under.
  - JuDue 14y ago
    Hey, reviews and anecdotes save time. Everyone does it.
tubbo 14y ago
Aww, Heroku is like our own little Miss Cleo "I foresee...mass confusion in your near (like, this week) future..."

50 points by rdegges 14y ago

A good reason to take a few minutes and make sure your backups are up-to-date and working, and that your application won't blow up in unexpected ways when the DB is unreachable.

neya 14y ago

I'm yet to see an update/fix on Heroku's recent PR nightmare on the random routing issue. Anyone?

coolsunglasses 14y ago
I'm not sure why this point isn't being made, but they're not going to change anything about how the routing works, least of all for Bamboo. Heroku is past the point of caring about pushing the envelope, stateful routing is an infrastructure risk and overhead that primarily their least informed customers need or want. It's a high margin operation is a low margin business, they're trying not to fuck it up. So stop asking.
- neya 14y ago
  As a customer, no matter what they do, whether it's a high margin risk or low margin profits, I'm supposed to be informed of their decision - whether they try to fix it or not, because I was one of the affected. They promised they would do so in their previous blog posts, hence the curiosity. >So stop asking. I have every right to ask and I will rightfully so. A disguised internet profile about sunglasses means nothing to me.
joevandyk 14y ago
They are about to push an update that lets you run larger dynos, which reduces the risk of having all the requests to a dyno stuck doing something long-lived.
- redguava 14y ago
  I believe the larger dynos are mostly to better handle using Unicorn as the webserver. Unicorn allows multiple workers to run per dyno, and has it's own smart routing, so it compensates well for Heroku's routing system. The downside to Unicorn was that the workers share a dyno's memory and often you couldn't have more than 2 per dyno with 512mb memory. This change helps that a lot.
dserodio 14y ago
They've put together a FAQ about routing: https://blog.heroku.com/archives/2013/4/2/routing_and_web_performance_on_heroku_a_faq https://blog.heroku.com/archives/2013/4/2/routing_and_web_pe...

pilif 14y ago

So. Us normal users will get the security fix on Thursday, but Heroku is somewhat special and gets early access? What does one have to do to become special? I don't like how this is being handled, no matter how serious the issue. I could kind of understand what the postgres team was doing until this point where they show that while everybody is equal, some are more equal than others and get early access to security fixes. Doing this does nothing but motivate the bad guys even more to find the hole on their own. The initial announcement got me scared already, but this just made it worse.

mrkurt 14y ago
Heroku is special, they host a huge number of Postgres setups that are publicly accessible and somewhat easily discovered. It's not a matter of equality, though, it's one of trust/privacy. Heroku is an easy company for the Postgres guys to trust since they have a pre-existing, healthy relationship.
- tensor 14y ago
  Meanwhile they are holding back a security fix for numerous other companies that also take security extremely seriously. This creates a situation where companies considering posgresql will now have to ask "will I get security fixes as soon as they are ready or will I intentionally be left vulnerable while more privileged users get early access?" Not a good precedent as far as I'm concerned.
  - andrewljohnson 14y ago
    It is open source. If you are so worried about the SLA you get from this open source software, maybe you should a) pay for better support or b) fix it yourself.
    - eksith 14y ago
      That's a cheeky cop-out. The original criticism is still valid, and we don't know how many companies outside of Heroku (if any) are getting the patch ahead of time. We don't all have the resources to a) Pay Oracle grade prices for support or b) the man power to comb through millions of lines of code to fix it ourselves. If these are the best excuses for lack of full transparency one could could come up with, then why would anyone choose a Linux distro? Or any of the *BSDs for that matter? Transparency and an expectation of due diligence is why a lot of companies (in IT and elsewhere) still stay in business while choosing Open Source software. Besides, Postgres isn't some run-of-the-mill startup with a dropped letter domain name. It's a venerable software foundation with almost 20 years of work and experience behind it. Which makes this a questionable course of action on their part despite the need for discretionary release.

gingerlime 14y ago

I was just recently considering whether we should move our postgresql over to Heroku. On one hand, it sounds promising that they deal with important security updates, and can do it even before the fix is officially announced. Very impressive. On the other hand, not being able to schedule the downtime to fit with your app/userbase is quite annoying. I understand this could be a potentially big security vulnerability that needs urgent fixing. But each organization is different and their risks are therefore different. Some people might prefer to take the risk of a few hours delay to apply the fix (being exposed) in exchange for not having to experience unscheduled downtime of this nature... It seems a little awkward to me that Heroku is taking this decision for its entire customer-base in such a way.

ryan_f 14y ago

This is rather disconcerting. They are given us a window of 48 hours where our site will be down for 60 seconds. On top of that we cannot schedule the time with them. 60 seconds may seem like a small inconvenience but it is one still the same. I host a few clients on their servers and I need to relay this information. I sound like a jackass being that I can't give more specifics on the timing of the server being down to my clients. How does Heroku not feel the same? I am appreciative of the update but that is also what I pay them to do.

fein 14y ago
> How does Heroku not feel the same? Because they, as a hosting provider, have several orders of magnitude more clients to please than yourself. The only way to be fair to everyone is that everyone gets treated like shit (obviously embellished). All you should be doing to CYA is telling your clients exactly why this is happening and that its Heroku, and not you, that holds the blame at this point. The caveat would be those difficult customers (suits, exec managers, etc) that you have to handle with kid gloves. For these people... I don't think there's a graceful way to present the upcoming issue.
- gingerlime 14y ago
  I doubt customers typically know or care whether you use heroku, Amazon or any other provider. What they want is to see their system up and running. That said, I wonder if with the heroku postgresql setup, this vulnerability could mean that if even one db is compromised, it could allow access to other databases too. Maybe heroku runs several postgresql db's on the same virtual/physical hosts?. If that's the case, then heroku simply can't afford to let even one database stay exposed because it would risk all others. Whether or not this is the case, I don't know, but it kinda feels like running your app on a shared-hosting account...
  - criley 14y ago
    >What they want is to see their system up and running. This would be a good chance to take a step back and think about what's more important: using Heroku or controlling your up-time?

thruflo 14y ago

It may be de-rigueur to bash Heroku in the wake of the Rap Genius drama/revelations but this kind of "downtime to apply critical security patch" announcement is precisely why I choose them. I spotted the announcement of the postgres vulnerability and thought "sounds serious". Now I read that my production databases will be fixed as soon as the patch is available without me having to engage with the details.

nevinera 14y ago

>Due to the nature of this update, a scheduled time is not possible. What, it takes place outside of linear time? This crap is unacceptable. Even my $2.00 shared host tells me when they're going to take down my site without asking.

viraptor 14y ago
More likely it takes around 5min per host if everything goes right. If they have 100 hosts and get delayed on the 10th and 15th for some reason, all other scheduled times would get shifted... And the upside is that they spot failures as they appear, rather than "oops, the last 5 hosts we did at the same time do not start up, there's some big issue". </speculation>

JuDue 14y ago

Just as an aside, how good is Heroku's Postgres performance, speed wise?

guywithabike 14y ago
Don't ask for anecdotes -- test it yourself with conditions similar to those that you'd be operating under.
- JuDue 14y ago
  Hey, reviews and anecdotes save time. Everyone does it.

tubbo 14y ago

Aww, Heroku is like our own little Miss Cleo "I foresee...mass confusion in your near (like, this week) future..."