5 ms·
Yup. A lot of heavy users of Postgres eventually hit the same barrier. Here's another take from Uber: https://www.uber.com/blog/postgres-to-mysql-migration/ htt
by cyberax 4y ago
Yup. A lot of heavy users of Postgres eventually hit the same barrier. Here's another take from Uber: https://www.uber.com/blog/postgres-to-mysql-migration/ https://www.uber.com/blog/postgres-to-mysql-migration/
I had a similar personal experience. In my previous job we used Postgres to implement a task queuing system, and it created a major bottleneck, resulting in tons of concurrency failures and bloat.
And most dangerously, the system failed catastrophically under load. As the load increased, most transactions ended up in concurrent failures, so very little actual work got committed. This increased the amount of outstanding tasks, resulting in even higher rate of concurrent failures.
And this can happen suddenly, one moment the system behaves well, with tasks being processed at a good rate, and the next moment the queue blows up and nothing works.
I re-implemented this system using pessimistic locking, and it turned out to work much better. Even under very high load, the system could still make forward progress.
The downside was having to make sure that no deadlocks can happen.
- osigurdson 4y ago>> In my previous job we used Postgres to implement a task queuing system, and it created a major bottleneck, resulting in tons of concurrency failures and bloat Yet, every month or two an article about doing exactly this is upvoted to near the top of HN. It can of course work but might hard to replace years later once "barnacles" have grown on it. Every situation is different of course.
- guywhocodes 4y agoI've built this queue system probably 5 times, first 2-3 were failures as consumer concurrency was 1 without us noticing for hours. The bloat comes from updating the work instead of deleting I assume, did for me. There are definitely many ways to not do it right but kinda works.
- osigurdson 4y agoYeah, it is strange that hokey, home grown solutions built on top of Postgres are suddenly in vogue.
- johnthescott 4y agoskip locks are the secret for queues in pg. did you use skip locks?
- DiabloD3 4y agoI remember when Uber got roasted by the postgresql mailing list over this: ultimately, a post mortem was done on all of Uber's claims, and it was basically proven that they were incompetent, did not read any available "best practices" guides, did not seek any external help, and treated it like it was some sort of mysql-esque database and used it as wrong as humanly possible. Uber's workload at the time, ironically, was not enough to make a postgresql server running moderately decent hardware to fall over if you actually read the manual. Uber's engineering team will never be able to live this down.
- klysm 4y agoMigrating entire workload is way more fun and exciting than reading manual. How else are you going to demonstrate your impact!
- endorphine 4y agoAnyone has a link to that mailing list thread to share?
- ncann 4y agoFrom what I can Google it seems to be the opposite of that, where they acknowledged Postgres's shortcoming in the mailing list: https://www.reddit.com/r/programming/comments/4vms8x/why_we_lost_uber_as_a_user_postgresql_mailing_list/ https://www.reddit.com/r/programming/comments/4vms8x/why_we_... https://www.postgresql.org/message-id/5797D5A1.5030009%40agliodbs.com https://www.postgresql.org/message-id/5797D5A1.5030009%40agl...
- fathyb 4y agoI didn't look at the reddit link but the full mailing list thread is more nuanced than that: https://www.postgresql.org/message-id/flat/579795DF.10502%40commandprompt.com https://www.postgresql.org/message-id/flat/579795DF.10502%40...
- emmelaich 4y agoDo you have a link to the mailing list discussion?