5 ms·

Heroku releases Followers into General Availability

50 points by jorde 14y ago

adrianpike 14y ago
Can someone more versed in the current state of Postgres chime in on something for me? > "One use case that has historically been challenging in database management is setting up a read replica, often referred to as a read slave." Having only inherited MySQL stuff in production, this is pretty trivial in MySQL land - I was under the impression that Postgres also had a solution for this, with Slony and recently as a part of Postgres core. Is my info wrong on this? I thought master/slave replication was a pretty solved problem at this point, am I wrong?
- hgimenez 14y ago
  Postgres, as of 9.0, has binary log streaming replication built in, truth. Slony ad other logical replication solutions out there are quite hard to set up and come with their gotchas. A rock solid one command (cli)/click (web) is a whole different story though, making it super easy to form more complex DB topologies and adapting them as your needs change.
blissofbeing 14y ago
This is nice, but it would be even better if we could provision a follower in another AWS Region or at least another AZ.
- hgimenez 14y ago
  Followers are automatically provisioned on a different AZ than their leaders.
  - thibaut_barrere 14y ago
    A very nice complement to my ruby fronts hosted at DotCloud. I really can't wait to use this!
thibaut_barrere 14y ago
I can't wait to give them my bucks, once they become Safe Harbor compliant!
denibertovic 14y ago
this is just so awesome... I have become quite well versed in setting up streaming replication by myself (postgres 9.1) and have a good system when i need to provision one but a one click solution is a whole other level!
jhuckestein 14y ago
After heroku's most recent outage, I've been trying to find a way to replicate my heroku database to a non-heroku server and so far haven't found an easy way to do this. Has anybody gotten the log streaming to work with a postgres box that is hosted somewhere else (in my case on Rackspace)?
- amalag 14y ago
  What about just heroku in a different physical region, like replicate from east coast to west coast?
- fdr 14y ago
  This doesn't work via the hot standby/follower mechanism because one needs access to a user with the replication role, which Heroku does not provide. The snag for that is that that stream is architecture, operating system, and postgres-minor-version specific, and that is why we generally do not feel inclined to support that as an interface. Postgres is learning logical replication that doesn't have these restrictions. I sure hope it can make 9.3 -- there's still a lot of ground and risk to cover -- and maybe that will deliver hope of what you seek. In addition, the archives (base backups and archived logs) would also have to be exposed to deal with clients that may have been disconnected for a reasonable period of time, and using a direct interface to that at the storage level is not something I think is very pleasant or maintainable, because I want the freedom to change our archiving layout as we see fit to enable disaster recovery/followers as-is. I think converting those archives to streaming protocol traffic is the way to go, but this doesn't handle the issue of base-backup propagation, which is a serious blocker. So, there's a lot of work to do all around...if anyone thinks this general kind of work is really interesting and perhaps would like to work on it on Heroku's behalf, he/she might want to consider emailing me at daniel@heroku.com.
aneth4 14y ago
Can anyone comment on how this affects consistency from a writing client? For instance, let's say I update a user record with a new email address, then display then redirect to an action to display the user record. The write may not have reached the replica, so I may get the old email address, correct? It seems this might require quite a bit more thought than simply enabling a replica.
- mikkelewis 14y ago
  This is just a general problem of slave lag. In this case you should read from master on redirect.
- drm237 14y ago
  When Facebook opened their second data center, they basically had one be read/write and the other just read (if I remember correctly). You could do something similar and if a user causes a write or update, keep track in a cookie and use only the master for their reads and writes for a period of time equal (or longer) than the typical replication length. Then when the time expires, start using the slaves for their reads. I'm sure there are better ways than this, it's just one that I know has been used in production with some success.
- hgimenez 14y ago
  A little-known feature of Postgres' streaming replication is Synchronous Replication or 2-safe replication, available from version 9.1 onward. It allows you to specify that a given commit must be durable on a standby server prior to the server returning to the client. You can turn this on cluster wide, or just for single transactions using session variables. In your example, you can specify that this update must be synchronously replicated, and as such the consequent request will show the user the updated data. Read more about it here: http://www.postgresql.org/docs/9.1/static/warm-standby.html#SYNCHRONOUS-REPLICATION http://www.postgresql.org/docs/9.1/static/warm-standby.html#...
  - recuter 14y ago
    Wait, so you want to block the response until a cross database RPC completes? That sounds like it would not be very performant and possibly introduce some other problems. I'm guessing this is best used sparingly with specific transactional queries.
garindra 14y ago
A question : is there any way for my application code to transparently "recognize" that (at runtime) there is a new follower PG database that is assigned to the application and that it should route some of the reads to the new follower DB? Or do I have to manually change my app's db configuration and include the new follower's address? Does Postgres actually natively support this and is there any client library that makes use of that? I know that Zookeeper serves this sort of purpose (automatic service discovery etc.), but I have no idea whether/how it works with Postgres.
zimbatm 14y ago
Are their Postgresql offering running on top of EBS instances ?
- thibaut_barrere 14y ago
  Maybe here (but I didn't investigate closely): http://enterprisedb.com/ http://enterprisedb.com/

50 points by jorde 14y ago

Can someone more versed in the current state of Postgres chime in on something for me? > "One use case that has historically been challenging in database management is setting up a read replica, often referred to as a read slave." Having only inherited MySQL stuff in production, this is pretty trivial in MySQL land - I was under the impression that Postgres also had a solution for this, with Slony and recently as a part of Postgres core. Is my info wrong on this? I thought master/slave replication was a pretty solved problem at this point, am I wrong?

hgimenez 14y ago
Postgres, as of 9.0, has binary log streaming replication built in, truth. Slony ad other logical replication solutions out there are quite hard to set up and come with their gotchas. A rock solid one command (cli)/click (web) is a whole different story though, making it super easy to form more complex DB topologies and adapting them as your needs change.

blissofbeing 14y ago

This is nice, but it would be even better if we could provision a follower in another AWS Region or at least another AZ.

hgimenez 14y ago
Followers are automatically provisioned on a different AZ than their leaders.
- thibaut_barrere 14y ago
  A very nice complement to my ruby fronts hosted at DotCloud. I really can't wait to use this!

thibaut_barrere 14y ago

I can't wait to give them my bucks, once they become Safe Harbor compliant!

denibertovic 14y ago

this is just so awesome... I have become quite well versed in setting up streaming replication by myself (postgres 9.1) and have a good system when i need to provision one but a one click solution is a whole other level!

jhuckestein 14y ago

After heroku's most recent outage, I've been trying to find a way to replicate my heroku database to a non-heroku server and so far haven't found an easy way to do this. Has anybody gotten the log streaming to work with a postgres box that is hosted somewhere else (in my case on Rackspace)?

amalag 14y ago
What about just heroku in a different physical region, like replicate from east coast to west coast?
fdr 14y ago
This doesn't work via the hot standby/follower mechanism because one needs access to a user with the replication role, which Heroku does not provide. The snag for that is that that stream is architecture, operating system, and postgres-minor-version specific, and that is why we generally do not feel inclined to support that as an interface. Postgres is learning logical replication that doesn't have these restrictions. I sure hope it can make 9.3 -- there's still a lot of ground and risk to cover -- and maybe that will deliver hope of what you seek. In addition, the archives (base backups and archived logs) would also have to be exposed to deal with clients that may have been disconnected for a reasonable period of time, and using a direct interface to that at the storage level is not something I think is very pleasant or maintainable, because I want the freedom to change our archiving layout as we see fit to enable disaster recovery/followers as-is. I think converting those archives to streaming protocol traffic is the way to go, but this doesn't handle the issue of base-backup propagation, which is a serious blocker. So, there's a lot of work to do all around...if anyone thinks this general kind of work is really interesting and perhaps would like to work on it on Heroku's behalf, he/she might want to consider emailing me at daniel@heroku.com.

aneth4 14y ago

Can anyone comment on how this affects consistency from a writing client? For instance, let's say I update a user record with a new email address, then display then redirect to an action to display the user record. The write may not have reached the replica, so I may get the old email address, correct? It seems this might require quite a bit more thought than simply enabling a replica.

mikkelewis 14y ago
This is just a general problem of slave lag. In this case you should read from master on redirect.
drm237 14y ago
When Facebook opened their second data center, they basically had one be read/write and the other just read (if I remember correctly). You could do something similar and if a user causes a write or update, keep track in a cookie and use only the master for their reads and writes for a period of time equal (or longer) than the typical replication length. Then when the time expires, start using the slaves for their reads. I'm sure there are better ways than this, it's just one that I know has been used in production with some success.
hgimenez 14y ago
A little-known feature of Postgres' streaming replication is Synchronous Replication or 2-safe replication, available from version 9.1 onward. It allows you to specify that a given commit must be durable on a standby server prior to the server returning to the client. You can turn this on cluster wide, or just for single transactions using session variables. In your example, you can specify that this update must be synchronously replicated, and as such the consequent request will show the user the updated data. Read more about it here: http://www.postgresql.org/docs/9.1/static/warm-standby.html#SYNCHRONOUS-REPLICATION http://www.postgresql.org/docs/9.1/static/warm-standby.html#...
- recuter 14y ago
  Wait, so you want to block the response until a cross database RPC completes? That sounds like it would not be very performant and possibly introduce some other problems. I'm guessing this is best used sparingly with specific transactional queries.

garindra 14y ago

A question : is there any way for my application code to transparently "recognize" that (at runtime) there is a new follower PG database that is assigned to the application and that it should route some of the reads to the new follower DB? Or do I have to manually change my app's db configuration and include the new follower's address? Does Postgres actually natively support this and is there any client library that makes use of that? I know that Zookeeper serves this sort of purpose (automatic service discovery etc.), but I have no idea whether/how it works with Postgres.

zimbatm 14y ago

Are their Postgresql offering running on top of EBS instances ?

thibaut_barrere 14y ago
Maybe here (but I didn't investigate closely): http://enterprisedb.com/ http://enterprisedb.com/