7 ms·

Personally, this kind of thing actually gives me _more_ confidence in Postgres rather than less. The core team's responsiveness to this bug report was incredibl

Personally, this kind of thing actually gives me _more_ confidence in Postgres rather than less. The core team's responsiveness to this bug report was incredibly impressive. Around June 4th, the article's author comes in with a bug report that basically says "I hammered Postgres with a whole bunch of artificial load and made something happen" [1]. By the 8th, a preliminary patch is ready for review [2]. That includes all the time to get the author's testing bootstrap up and running, reproduce, diagnose the bug (which, lest us forget, is the part of all of this that is actually hard), and assemble a fix. It's worth noting that it's no one's job per se on the Postgres project of fix this kind of thing — the hope is that someone will take interest, step up, and find a solution — and as unlikely as that sounds to work in most environments, amazingly, it usually does for Postgres. Of note to the hacker types here, Peter Geoghegan was able to track the bug down through the use of rr [4] [5], which allowed an entire problematic run to be captured, and then stepped through forwards _and_ backwards (the latter being the key for not having to run the simulation over and over again) until the problematic code was identified and a fix could be developed. --- [1] https://www.postgresql.org/message-id/CAH2-Wzm9kNAK0cbzGAvDtdJi-rj_ngsBbRX0i_DKdjYxqJnzNA%40mail.gmail.com https://www.postgresql.org/message-id/CAH2-Wzm9kNAK0cbzGAvDt... [2] https://www.postgresql.org/message-id/CAH2-Wzk%2BFHVJvSS9VPPJ_K9w4xwqeVyfnkzYWtWrBzXJSJcMVQ%40mail.gmail.com https://www.postgresql.org/message-id/CAH2-Wzk%2BFHVJvSS9VPP... [3] https://www.postgresql.org/message-id/CAH2-WznTb6-0fjW4WPzNQh4mFvBH86J7bqZpNqteVUzo8p%3D6Hg%40mail.gmail.com https://www.postgresql.org/message-id/CAH2-WznTb6-0fjW4WPzNQ... [4] https://en.wikipedia.org/wiki/Rr_(debugging) https://en.wikipedia.org/wiki/Rr_(debugging) [5] https://www.postgresql.org/message-id/CAH2-WznTb6-0fjW4WPzNQh4mFvBH86J7bqZpNqteVUzo8p%3D6Hg%40mail.gmail.com https://www.postgresql.org/message-id/CAH2-WznTb6-0fjW4WPzNQ...

aphyr 6y ago
Yeah, the PostgreSQL team really knocked it out of the park on this one. It was a pleasure working with them. :)
- MoOmer 6y ago
  To be fair, you have a great batting average in identifying issues to allow for improvement. Thanks for your work
- AtlasBarfed 6y ago
  With distributed and multicore being the path forward with the end of Moores law, your work has been instrumental in helping open source distributed systems improve. Since distributed systems are so difficult and complicated, it enables salespeople and zealots to both deny issues and overstate capability. Your work is a shining star in that darkness. Thank you.
- archon810 6y ago
  For those not entirely sure, this is Kyle Kingsbury, the author of the main article.
gen220 6y ago
Thank you for this comment that gives credit where it's due, this is a very impressive set of threads to read through. And I agree. For me, one of the most important measures of the reliability of a system is how that system responds too information that it might be wrong. If the response is defensiveness, evasiveness, or persuasive in any way, i.e. of the "it's not that bad" variety, run for the hills. This, on the other hand is technical, validating, and prompt. Every system has bugs, but depending on these cultural features, not every system is capable of systematically removing those bugs. With logs like these, the pg community continues to prove capable. Kudos!
- shawn-butler 6y ago
  What are the storage requirements for using rr for intense or longer debugging sessions?
  - gen220 6y ago
    this paper describes rr, which for context was designed to be used on commodity hardware: https://arxiv.org/pdf/1610.02144.pdf https://arxiv.org/pdf/1610.02144.pdf section 4.4 talks about disk requirements: > Memory-mapped files are almost entirely just the executables and libraries loaded by tracees. As long as the original files don’t change and are not removed, which is usually true in practice, their clones take up no additional space and require no data writes > Like cloned files, cloned file blocks do not consume space as long as the underlying data they’re cloned from persists. they conclude the section with: > In any case, in real-world usage trace storage has not been a concern I imagine that over "longer debugging sessions" the metadata footprint would expand linearly, but probably with a constant smaller than the logs for the average program.
  - petergeoghegan 6y ago
    The exact recording in question was about 125MB, and that was after I materialized it using "rr pack". I'd say that the storage overhead is unlikely to be a concern in almost all cases. It's just something that you need to keep an eye on.
- 6y ago
bredren 6y ago
Thanks for this summary. I take for granted that I have a Postgres, powerful And reliable database that I get to use for free in all my projects and work.
jwr 6y ago
Indeed — it's great to see a vendor (team, in this case) that doesn't try to downplay a Jepsen result, and instead fixes the issues. However, there is one more takeaway here. I've heard too many times "just use Postgres", repeated as an unthinking mantra. But there are no obvious solutions in the complex world of databases. And this isn't even a multi-node scenario!
- Scarbutt 6y ago
  "Just use Postgres" may have become meme but for good reason and is well grounded IMO. Many immature databases with not much wide use are better avoided though, we manage to break datomic three times during development, the first two bugs were fixed in a week, the third took a month, which they called in their changelog "Fix: Prevent a rare scenario where retracting non-existent entities could prevent future transactions from succeeding" so yeah, we went back to "just use postgres", who wants to go through the nightmare of hitting those bugs in production and who knows how many more?scary situation.
- pnathan 6y ago
  "Use PostGres until you have an engineering - data driven rationale not to" is my standard answer for non-blob data storage when a project starts. Why? because when `n` is small (tables, rows, connections), postgres works well enough, and if `n` should ever become large, we'll have interest in funding the work to do a migration, if that's appropriate - and we'll be able to evaluate the system at scale with real data.
  - jmalicki 6y ago
    postgres works great for `n` in ranges in excess of (10k, 10B, 1k) at least, which not many would consider small. The only real justification I've heard is postgres' story on replication still lags mysql, and the write amplification bit which they've done a lot of work on for pg13 (see [0]) Even for many very large companies I've seen, noSQL databases are mostly used as caches that can afford to be delayed or a bit out of sync, not as the database of record. [0] https://eng.uber.com/postgres-to-mysql-migration/ https://eng.uber.com/postgres-to-mysql-migration/

aphyr 6y ago

Yeah, the PostgreSQL team really knocked it out of the park on this one. It was a pleasure working with them. :)

MoOmer 6y ago
To be fair, you have a great batting average in identifying issues to allow for improvement. Thanks for your work
AtlasBarfed 6y ago
With distributed and multicore being the path forward with the end of Moores law, your work has been instrumental in helping open source distributed systems improve. Since distributed systems are so difficult and complicated, it enables salespeople and zealots to both deny issues and overstate capability. Your work is a shining star in that darkness. Thank you.
archon810 6y ago
For those not entirely sure, this is Kyle Kingsbury, the author of the main article.

gen220 6y ago

Thank you for this comment that gives credit where it's due, this is a very impressive set of threads to read through. And I agree. For me, one of the most important measures of the reliability of a system is how that system responds too information that it might be wrong. If the response is defensiveness, evasiveness, or persuasive in any way, i.e. of the "it's not that bad" variety, run for the hills. This, on the other hand is technical, validating, and prompt. Every system has bugs, but depending on these cultural features, not every system is capable of systematically removing those bugs. With logs like these, the pg community continues to prove capable. Kudos!

shawn-butler 6y ago
What are the storage requirements for using rr for intense or longer debugging sessions?
- gen220 6y ago
  this paper describes rr, which for context was designed to be used on commodity hardware: https://arxiv.org/pdf/1610.02144.pdf https://arxiv.org/pdf/1610.02144.pdf section 4.4 talks about disk requirements: > Memory-mapped files are almost entirely just the executables and libraries loaded by tracees. As long as the original files don’t change and are not removed, which is usually true in practice, their clones take up no additional space and require no data writes > Like cloned files, cloned file blocks do not consume space as long as the underlying data they’re cloned from persists. they conclude the section with: > In any case, in real-world usage trace storage has not been a concern I imagine that over "longer debugging sessions" the metadata footprint would expand linearly, but probably with a constant smaller than the logs for the average program.
- petergeoghegan 6y ago
  The exact recording in question was about 125MB, and that was after I materialized it using "rr pack". I'd say that the storage overhead is unlikely to be a concern in almost all cases. It's just something that you need to keep an eye on.
6y ago

bredren 6y ago

Thanks for this summary. I take for granted that I have a Postgres, powerful And reliable database that I get to use for free in all my projects and work.

jwr 6y ago

Indeed — it's great to see a vendor (team, in this case) that doesn't try to downplay a Jepsen result, and instead fixes the issues. However, there is one more takeaway here. I've heard too many times "just use Postgres", repeated as an unthinking mantra. But there are no obvious solutions in the complex world of databases. And this isn't even a multi-node scenario!

Scarbutt 6y ago
"Just use Postgres" may have become meme but for good reason and is well grounded IMO. Many immature databases with not much wide use are better avoided though, we manage to break datomic three times during development, the first two bugs were fixed in a week, the third took a month, which they called in their changelog "Fix: Prevent a rare scenario where retracting non-existent entities could prevent future transactions from succeeding" so yeah, we went back to "just use postgres", who wants to go through the nightmare of hitting those bugs in production and who knows how many more?scary situation.
pnathan 6y ago
"Use PostGres until you have an engineering - data driven rationale not to" is my standard answer for non-blob data storage when a project starts. Why? because when `n` is small (tables, rows, connections), postgres works well enough, and if `n` should ever become large, we'll have interest in funding the work to do a migration, if that's appropriate - and we'll be able to evaluate the system at scale with real data.
- jmalicki 6y ago
  postgres works great for `n` in ranges in excess of (10k, 10B, 1k) at least, which not many would consider small. The only real justification I've heard is postgres' story on replication still lags mysql, and the write amplification bit which they've done a lot of work on for pg13 (see [0]) Even for many very large companies I've seen, noSQL databases are mostly used as caches that can afford to be delayed or a bit out of sync, not as the database of record. [0] https://eng.uber.com/postgres-to-mysql-migration/ https://eng.uber.com/postgres-to-mysql-migration/