4 ms·

huh. on one hand, this is awesome! something like this should be fantastic for things like citus’s columnar storage - a pluggable storage system can be a huge b

huh. on one hand, this is awesome! something like this should be fantastic for things like citus’s columnar storage - a pluggable storage system can be a huge boon. on the other hand there have been some bugs popping up on pg-bugs like the parallel queries that look a little questionable. hopefully no negative impact for anyone upstream while everything gets hammered out. context: i’m The maintainer of a fairly popular pg extension editing to note that the bugs popping up have nothing to do with this, just the hope that it doesn’t become ... overwhelming

macdice 8y ago
Hi Jerry, Yeah, parallelism is hard... but speaking as someone involved in several bug fixes in that area, I can tell you that we put a lot of effort into chasing those bugs down, and even made changes to allow extensions that were doing illegal things in _PG_init() to keep working under parallelism. Bugs are inevitable, and in complicated code it can take someone else's workload to uncover them. IMHO the important question is how well you deal with them, and I think we do a good job at providing work-arounds (primarily by providing GUCs so you can turn new stuff off it it's causing problems), and getting fixes into the tree ASAP. I think the worse recent case was the "DSM handle collision" one, which was hiding in code committed years ago but only discoverable with very particular timing and newer query plans. All currently known problems fixed in that department, but unfortunately those patches missed the February cut-off for 11.2 and will have to wait until May.
- jerrysievert 8y ago
  > but speaking as someone involved in several bug fixes in that area, I can tell you that we put a lot of effort into chasing those bugs down, yup! i've definitely seen the efforts, and really appreciate the work that everyone is doing to solve them - it's great seeing that kind of commitment. > IMHO the important question is how well you deal with them you are absolutely correct! and i can't fault how quickly bugs in parallelism have been triaged and fixed. but that wasn't what i was getting at - just that there have been a lot of fairly large changes (i guess you can call it a big increase in velocity?) that still seem to have (or uncovered as you said) a lot more to do - parallelism was just an example of a big change that seems to have some pretty big impacts on stability and planning and adding another big change makes me wary since that has direct impact on code that i maintain. it wasn't an attack, and i hope that it didn't come across that way, just an observation ... the more major changes the more balls in the air (to borrow a juggling metaphor), and pg has a ton of them in the air right now. > I think we do a good job at providing work-arounds (primarily by providing GUCs so you can turn new stuff off it it's causing problems) GUC's are helpful, but there are times when direct access to postgresql.conf isn't really feasible ... and setting GUC's at the beginning of every session can become cumbersome/untenable. also, GUC's aren't always feasible when some major changes have negative impact across whole stacks (the major changes in CTE's and how they impacted planning is one that i can think of off the top of my head). > illegal things in _PG_init() just curious, can you point to some of these? i want to do my own extension audits and considering that plv8 does/did a lot of things based on postgres/src/pl/* it would be good to make sure it's not doing any of those as well. regardless, thanks for the response - it is appreciated.
  - macdice 8y ago
    >> illegal things in _PG_init() > just curious, can you point to some of these? i want to do my own extension audits and considering that plv8 does/did a lot of things based on postgres/src/pl/* it would be good to make sure it's not doing any of those as well. https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=6c3c9d418918373a9535ad3d3bd357f652a367e3 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit...

macdice 8y ago

Hi Jerry, Yeah, parallelism is hard... but speaking as someone involved in several bug fixes in that area, I can tell you that we put a lot of effort into chasing those bugs down, and even made changes to allow extensions that were doing illegal things in _PG_init() to keep working under parallelism. Bugs are inevitable, and in complicated code it can take someone else's workload to uncover them. IMHO the important question is how well you deal with them, and I think we do a good job at providing work-arounds (primarily by providing GUCs so you can turn new stuff off it it's causing problems), and getting fixes into the tree ASAP. I think the worse recent case was the "DSM handle collision" one, which was hiding in code committed years ago but only discoverable with very particular timing and newer query plans. All currently known problems fixed in that department, but unfortunately those patches missed the February cut-off for 11.2 and will have to wait until May.

jerrysievert 8y ago
> but speaking as someone involved in several bug fixes in that area, I can tell you that we put a lot of effort into chasing those bugs down, yup! i've definitely seen the efforts, and really appreciate the work that everyone is doing to solve them - it's great seeing that kind of commitment. > IMHO the important question is how well you deal with them you are absolutely correct! and i can't fault how quickly bugs in parallelism have been triaged and fixed. but that wasn't what i was getting at - just that there have been a lot of fairly large changes (i guess you can call it a big increase in velocity?) that still seem to have (or uncovered as you said) a lot more to do - parallelism was just an example of a big change that seems to have some pretty big impacts on stability and planning and adding another big change makes me wary since that has direct impact on code that i maintain. it wasn't an attack, and i hope that it didn't come across that way, just an observation ... the more major changes the more balls in the air (to borrow a juggling metaphor), and pg has a ton of them in the air right now. > I think we do a good job at providing work-arounds (primarily by providing GUCs so you can turn new stuff off it it's causing problems) GUC's are helpful, but there are times when direct access to postgresql.conf isn't really feasible ... and setting GUC's at the beginning of every session can become cumbersome/untenable. also, GUC's aren't always feasible when some major changes have negative impact across whole stacks (the major changes in CTE's and how they impacted planning is one that i can think of off the top of my head). > illegal things in _PG_init() just curious, can you point to some of these? i want to do my own extension audits and considering that plv8 does/did a lot of things based on postgres/src/pl/* it would be good to make sure it's not doing any of those as well. regardless, thanks for the response - it is appreciated.
- macdice 8y ago
  >> illegal things in _PG_init() > just curious, can you point to some of these? i want to do my own extension audits and considering that plv8 does/did a lot of things based on postgres/src/pl/* it would be good to make sure it's not doing any of those as well. https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=6c3c9d418918373a9535ad3d3bd357f652a367e3 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit...