5 ms·

Any insights you can share? I'm looking at this now, and it's not encouraging compared to dedicated hardware.

by wooster 14y ago

sickpig 14y ago
Not from heroku guys but definitely a worth reading http://thebuild.com/blog/2012/05/18/running-postgresql-on-aws/ http://thebuild.com/blog/2012/05/18/running-postgresql-on-aw...
pvh 14y ago
You could always use Heroku Postgres. Our whole reason to exist is to ensure you can work fearlessly with your data. At this point, I think running your own data infrastructure is like having a generator in your garage instead of using the power grid.
- pestaa 14y ago
  It sounds good until the power grid goes down and the only person in the street with the lights on is with the generator in his garage.
  - wiredfool 14y ago
    And then a few hours later, the CO2 leaks into his house.
    - pestaa 14y ago
      At least he can fix it.
- bradgessler 14y ago
  This is a bad metaphor. Utility computing isn't quite up to the same level as electrical utilities. Heroku has suffered from more downtime in the past year than I have power outages. Also, most commercial operations that tap into public utilities have some sort of backup plan for when the power goes out. A dramatic example of that is the power staying on in the Goldman Sacs building during Hurricane Sandy (http://www.inquisitr.com/381743/hurricane-sandy-rages-but-the-lights-stay-on-at-goldman-sachs/ http://www.inquisitr.com/381743/hurricane-sandy-rages-but-th...) More visibility would be greatly appreciated into what you're doing behind the scenes and increase your customers' confidence and expectations of using the Heroku pg cloud service. I hope you provide more depth to future answers as opposed to, "Just trust us". I've found that in practice, things never work out that way.
  - pvh 14y ago
    That's a fair criticism - thanks for the rebuttal. (Edit) Also - what kind of visibility are you looking for that we don't offer today? Please feel free to email me (my email is rather guessable) with whatever you have.
- patrickgzill 14y ago
  What is your uptime over the last 12 months? As opposed to e.g. the power grid in non-Sandy hit areas?
- keypusher 14y ago
  What if you are a Fortune 500 company, or a part of the government? A defense contractor? Heath insurance provider? I work for a company that does "private cloud", mostly large enterprise/government places that want the flexibility of having their data accessible anywhere, but are not willing or able (for security reasons) to have Amazon or some other provider host their data. If you already have data center infrastructure and expertise in-house, the long-run cost and benefits of buying your own hardware for high-traffic applications can often outweigh renting space and/or paying on a per-transaction basis from AWS. Public cloud is great, but it's not the right solution for everyone.
jedberg 14y ago
It's been a while since I've been in the weeds, but this is what I remember: It's a balancing act between max_connections and shared_buffers. Each instance type will have a sweet spot for your use case -- you'll have to find it through experimenting. Read this: http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Serve... And this: http://www.postgresql.org/docs/9.2/static/kernel-resources.html http://www.postgresql.org/docs/9.2/static/kernel-resources.h... Give as much RAM to Postgres as possible -- let it do the memory management. Sometimes going to an instance with double the RAM will give you more than double the performance. No swap on the box. The moment you hit swap you're screwed. Vacuum often, maybe continuously, if the database has a lot of updates or deletes. Do your capacity planning so you can vacuum all the time. I'll update if I think of more.
- wooster 14y ago
  Thanks!
- bratsche 14y ago
  I've heard the "no swap" argument before, but can you explain this a little further? It seems like there are two potential situations here: 1/ You have swap, you hit swap. Massive slowdown. 2/ You don't have swap, run out of memory and processes die. #1 is not a good situation, but it seems preferable to #2 doesn't it?
  - mike_ivanov 14y ago
    "No swap" is a common misconception. Having no swap is dangerous and can actually slow down your system.
- mike_ivanov 14y ago
  > No swap on the box. Wrong. ...some common myths: 1. Swap space does not inherently slow down your system. In fact, not having swap space doesn't mean you won't swap pages. It merely means that Linux has fewer choices about what RAM can be reused when a demand hits. Thus, it is possible for the throughput of a system that has no swap space to be lower than that of a system that has some. 2. Swap space is used for modified anonymous pages only. Your programs, shared libraries and filesystem cache are never written there under any circumstances. 3. Given items 1 and 2 above, the philosophy of “minimization of swap space” is really just a concern about wasted disk space. ... http://www.linuxjournal.com/article/10678 http://www.linuxjournal.com/article/10678 Edit: formatting

Any insights you can share? I'm looking at this now, and it's not encouraging compared to dedicated hardware.

by wooster 14y ago

Any insights you can share? I'm looking at this now, and it's not encouraging compared to dedicated hardware.

Not from heroku guys but definitely a worth reading http://thebuild.com/blog/2012/05/18/running-postgresql-on-aws/ http://thebuild.com/blog/2012/05/18/running-postgresql-on-aw...

pvh 14y ago

You could always use Heroku Postgres. Our whole reason to exist is to ensure you can work fearlessly with your data. At this point, I think running your own data infrastructure is like having a generator in your garage instead of using the power grid.

pestaa 14y ago
It sounds good until the power grid goes down and the only person in the street with the lights on is with the generator in his garage.
- wiredfool 14y ago
  And then a few hours later, the CO2 leaks into his house.
  - pestaa 14y ago
    At least he can fix it.
bradgessler 14y ago
This is a bad metaphor. Utility computing isn't quite up to the same level as electrical utilities. Heroku has suffered from more downtime in the past year than I have power outages. Also, most commercial operations that tap into public utilities have some sort of backup plan for when the power goes out. A dramatic example of that is the power staying on in the Goldman Sacs building during Hurricane Sandy (http://www.inquisitr.com/381743/hurricane-sandy-rages-but-the-lights-stay-on-at-goldman-sachs/ http://www.inquisitr.com/381743/hurricane-sandy-rages-but-th...) More visibility would be greatly appreciated into what you're doing behind the scenes and increase your customers' confidence and expectations of using the Heroku pg cloud service. I hope you provide more depth to future answers as opposed to, "Just trust us". I've found that in practice, things never work out that way.
- pvh 14y ago
  That's a fair criticism - thanks for the rebuttal. (Edit) Also - what kind of visibility are you looking for that we don't offer today? Please feel free to email me (my email is rather guessable) with whatever you have.
patrickgzill 14y ago
What is your uptime over the last 12 months? As opposed to e.g. the power grid in non-Sandy hit areas?
keypusher 14y ago
What if you are a Fortune 500 company, or a part of the government? A defense contractor? Heath insurance provider? I work for a company that does "private cloud", mostly large enterprise/government places that want the flexibility of having their data accessible anywhere, but are not willing or able (for security reasons) to have Amazon or some other provider host their data. If you already have data center infrastructure and expertise in-house, the long-run cost and benefits of buying your own hardware for high-traffic applications can often outweigh renting space and/or paying on a per-transaction basis from AWS. Public cloud is great, but it's not the right solution for everyone.

jedberg 14y ago

It's been a while since I've been in the weeds, but this is what I remember: It's a balancing act between max_connections and shared_buffers. Each instance type will have a sweet spot for your use case -- you'll have to find it through experimenting. Read this: http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Serve... And this: http://www.postgresql.org/docs/9.2/static/kernel-resources.html http://www.postgresql.org/docs/9.2/static/kernel-resources.h... Give as much RAM to Postgres as possible -- let it do the memory management. Sometimes going to an instance with double the RAM will give you more than double the performance. No swap on the box. The moment you hit swap you're screwed. Vacuum often, maybe continuously, if the database has a lot of updates or deletes. Do your capacity planning so you can vacuum all the time. I'll update if I think of more.

wooster 14y ago
Thanks!
bratsche 14y ago
I've heard the "no swap" argument before, but can you explain this a little further? It seems like there are two potential situations here: 1/ You have swap, you hit swap. Massive slowdown. 2/ You don't have swap, run out of memory and processes die. #1 is not a good situation, but it seems preferable to #2 doesn't it?
- mike_ivanov 14y ago
  "No swap" is a common misconception. Having no swap is dangerous and can actually slow down your system.
mike_ivanov 14y ago
> No swap on the box. Wrong. ...some common myths: 1. Swap space does not inherently slow down your system. In fact, not having swap space doesn't mean you won't swap pages. It merely means that Linux has fewer choices about what RAM can be reused when a demand hits. Thus, it is possible for the throughput of a system that has no swap space to be lower than that of a system that has some. 2. Swap space is used for modified anonymous pages only. Your programs, shared libraries and filesystem cache are never written there under any circumstances. 3. Given items 1 and 2 above, the philosophy of “minimization of swap space” is really just a concern about wasted disk space. ... http://www.linuxjournal.com/article/10678 http://www.linuxjournal.com/article/10678 Edit: formatting