4 ms·

Thank you for this comment that gives credit where it's due, this is a very impressive set of threads to read through. And I agree. For me, one of the most imp

by gen220 6y ago

Thank you for this comment that gives credit where it's due, this is a very impressive set of threads to read through. And I agree. For me, one of the most important measures of the reliability of a system is how that system responds too information that it might be wrong. If the response is defensiveness, evasiveness, or persuasive in any way, i.e. of the "it's not that bad" variety, run for the hills. This, on the other hand is technical, validating, and prompt. Every system has bugs, but depending on these cultural features, not every system is capable of systematically removing those bugs. With logs like these, the pg community continues to prove capable. Kudos!

shawn-butler 6y ago
What are the storage requirements for using rr for intense or longer debugging sessions?
- gen220 6y ago
  this paper describes rr, which for context was designed to be used on commodity hardware: https://arxiv.org/pdf/1610.02144.pdf https://arxiv.org/pdf/1610.02144.pdf section 4.4 talks about disk requirements: > Memory-mapped files are almost entirely just the executables and libraries loaded by tracees. As long as the original files don’t change and are not removed, which is usually true in practice, their clones take up no additional space and require no data writes > Like cloned files, cloned file blocks do not consume space as long as the underlying data they’re cloned from persists. they conclude the section with: > In any case, in real-world usage trace storage has not been a concern I imagine that over "longer debugging sessions" the metadata footprint would expand linearly, but probably with a constant smaller than the logs for the average program.
- petergeoghegan 6y ago
  The exact recording in question was about 125MB, and that was after I materialized it using "rr pack". I'd say that the storage overhead is unlikely to be a concern in almost all cases. It's just something that you need to keep an eye on.
tetha 6y ago
>If the response is defensiveness, evasiveness, or persuasive in any way, i.e. of the "it's not that bad" variety, run for the hills. This, on the other hand is technical, validating, and prompt. This resonates with me with teams inside the company as well. We have a few teams that just deflect issues. Find any issue in the bug report, be it an FQDN in a log search, and poof it goes. Back to sender, don't care. Engineers in my team just don't care to report bugs there anymore, regardless how simple. Usually, it's faster and less frustrating to just work around it or ignore it. You could be fighting windmills for weeks, or just fudge around it. Other teams, far more receptive with bugs.. engineers end up curious and just poke around until they understand what's up. And then you have bug reports like "Ok, if I create these 57 things over here, and toggle thing 32 to off, and then toggle 2 things indexed by prime numbers on, then my database connection fails. I've reproduced this from empty VMs. If 32 is on, I need to toggle two perfect squares on, but not 3". And then a lot of things just get fixed.
jennyyang 6y ago
Practically everyone in distributed systems knows who Aphyr is, and to get a bug reported by him is like a badge of honor. I am willing to bet that people at PostgrSQL jousted over who got to work with him on this bug, because it would be such an amazing learning opportunity. I know I would want to!
emmelaich 6y ago
If you reject the "it's not that bad" response then you are saying all bugs are equal in severity / urgency / priority. Which is obviously not true. A better response might be something like "I understand the problem is X however the fix might take Y months and the next release will deprecate that feature anyway. Can you quantify or guesstimate the impact on our product if we allow this bug to stay unfixed?"
- gen220 6y ago
  Yeah, in retrospect, I disagree with myself, too. (and agre with your rationale!) :) I think if I were to try and salvage the quote, I'd change it to "It's probably not that bad". In this telling, I'm meaning to imply that the responder has made an apriori value judgement about a bug without determining scope and cause. I agree with your ideal response. The only caveat I'd personally make, is that I'd recommend "planting the flag" by providing your own guesstimate of product impact, and ask the reporter for their opinion on moving it. In my perspective, it's important to acknowledge that is has an impact, and it's especially important to convey your sense of impact concretely with some napkin math. If you're terribly wrong, the reporter will try to convince you of it[1], and you'll learn something along the way. It also conveys empathy (i.e. we're both trying to solve your problem, even if it's only a problem for you). [1]: https://meta.wikimedia.org/wiki/Cunningham%27s_Law https://meta.wikimedia.org/wiki/Cunningham%27s_Law

by gen220 6y ago

shawn-butler 6y ago

What are the storage requirements for using rr for intense or longer debugging sessions?

gen220 6y ago
this paper describes rr, which for context was designed to be used on commodity hardware: https://arxiv.org/pdf/1610.02144.pdf https://arxiv.org/pdf/1610.02144.pdf section 4.4 talks about disk requirements: > Memory-mapped files are almost entirely just the executables and libraries loaded by tracees. As long as the original files don’t change and are not removed, which is usually true in practice, their clones take up no additional space and require no data writes > Like cloned files, cloned file blocks do not consume space as long as the underlying data they’re cloned from persists. they conclude the section with: > In any case, in real-world usage trace storage has not been a concern I imagine that over "longer debugging sessions" the metadata footprint would expand linearly, but probably with a constant smaller than the logs for the average program.
petergeoghegan 6y ago
The exact recording in question was about 125MB, and that was after I materialized it using "rr pack". I'd say that the storage overhead is unlikely to be a concern in almost all cases. It's just something that you need to keep an eye on.

tetha 6y ago

>If the response is defensiveness, evasiveness, or persuasive in any way, i.e. of the "it's not that bad" variety, run for the hills. This, on the other hand is technical, validating, and prompt. This resonates with me with teams inside the company as well. We have a few teams that just deflect issues. Find any issue in the bug report, be it an FQDN in a log search, and poof it goes. Back to sender, don't care. Engineers in my team just don't care to report bugs there anymore, regardless how simple. Usually, it's faster and less frustrating to just work around it or ignore it. You could be fighting windmills for weeks, or just fudge around it. Other teams, far more receptive with bugs.. engineers end up curious and just poke around until they understand what's up. And then you have bug reports like "Ok, if I create these 57 things over here, and toggle thing 32 to off, and then toggle 2 things indexed by prime numbers on, then my database connection fails. I've reproduced this from empty VMs. If 32 is on, I need to toggle two perfect squares on, but not 3". And then a lot of things just get fixed.

jennyyang 6y ago

Practically everyone in distributed systems knows who Aphyr is, and to get a bug reported by him is like a badge of honor. I am willing to bet that people at PostgrSQL jousted over who got to work with him on this bug, because it would be such an amazing learning opportunity. I know I would want to!

emmelaich 6y ago

If you reject the "it's not that bad" response then you are saying all bugs are equal in severity / urgency / priority. Which is obviously not true. A better response might be something like "I understand the problem is X however the fix might take Y months and the next release will deprecate that feature anyway. Can you quantify or guesstimate the impact on our product if we allow this bug to stay unfixed?"

gen220 6y ago
Yeah, in retrospect, I disagree with myself, too. (and agre with your rationale!) :) I think if I were to try and salvage the quote, I'd change it to "It's probably not that bad". In this telling, I'm meaning to imply that the responder has made an apriori value judgement about a bug without determining scope and cause. I agree with your ideal response. The only caveat I'd personally make, is that I'd recommend "planting the flag" by providing your own guesstimate of product impact, and ask the reporter for their opinion on moving it. In my perspective, it's important to acknowledge that is has an impact, and it's especially important to convey your sense of impact concretely with some napkin math. If you're terribly wrong, the reporter will try to convince you of it[1], and you'll learn something along the way. It also conveys empathy (i.e. we're both trying to solve your problem, even if it's only a problem for you). [1]: https://meta.wikimedia.org/wiki/Cunningham%27s_Law https://meta.wikimedia.org/wiki/Cunningham%27s_Law