4 ms·

> Perhaps try to give others a bit more benefit of the doubt. I wonder whether you'd be willing to go into a meta-discussion on that, perhaps not public? I se

> Perhaps try to give others a bit more benefit of the doubt. I wonder whether you'd be willing to go into a meta-discussion on that, perhaps not public? I seem to hit this kind of stalemate too often and need someone's help to close the gap. I can see your email address in your profile. I don't see how consistent and repeated "can't work" answers can lead to a collaboration on an actual solution. I'm not even the proposer of the optimization, but I can see it has merit and doesn't have high intrinsic complexity. > always packing columns would turn an operation that doesn't currently rewrite into a rewriting operation I don't think this is necessarily true. What is the operation that would rewrite that doesn't already, assuming the same shape of solutions already used? > The per-row overhead would be too large - we already have problems with that. I read this as "the proposal can't work because [...]", on another thing that feels like a technicality. There are likely all sorts of bit-packing and other options that would make the overhead low, and the Postgres per-row overhead is already so large that it seems doubtful that it'd be that big a deal. (For example, if you've very few columns, you don't need much space for the counts, and if you've more columns, the extra space for a few sizes isn't going to have much impact. Also, any active transaction range isn't going to have a large number of table shapes in it, so rows could likely use a table-definition-version number in place of any column counts. Or you could probably look it up in some bookkeeping structure via xmin and drop the per-row details. The fast path would remain when there are no recent shape changes.)

anarazel 4y ago
> I wonder whether you'd be willing to go into a meta-discussion on that, perhaps not public? I seem to hit this kind of stalemate too often and need someone's help to close the gap. I can see your email address in your profile. Feel free to email me. I'll be offline for part of the next two weeks though. > I don't see how consistent and repeated "can't work" answers can lead to a collaboration on an actual solution. I'm not even the proposer of the optimization, but I can see it has merit and doesn't have high intrinsic complexity. Well, it's easy to say that something doesn't have a high intrinsic complexity from the outside, if you haven't spent years in the guts of a system. > > always packing columns would turn an operation that doesn't currently rewrite into a rewriting operation > I don't think this is necessarily true. What is the operation that would rewrite that doesn't already, assuming the same shape of solutions already used? Adding a column to a table. Right now that doesn't require a rewrite, if it were always optimally packed it would (or alternatively increase the per-row overhead). > > The per-row overhead would be too large - we already have problems with that. > I read this as "the proposal can't work because [...]", on another thing that feels like a technicality. There are likely all sorts of bit-packing and other options that would make the overhead low, and the Postgres per-row overhead is already so large that it seems doubtful that it'd be that big a deal. Shrug. I've spent a lot of time working on postgres, including heap storage. Within the constraints of where we are, I am quite confident that it's not realistic to add that. There's larger possible redesigns (and I've spent working on attempts) that could make it more realistic, but it'd be a lot of work - and I'm doubtful it's the most productive way to spend the effort. I think time would be much more wisely spent removing the need for most of the alignment padding - for by-value types that's solely needed because of an ancient design choice, namely that system catalogs need a C struct compatible layout. That can be addressed in less harmful ways (two things discussed so far, 1) only pad by-value columns in catalog tables 2) auto-generate mapping code to/from the catalog tuple format to struct format). With that a lot of the padding overhead vanishes - many by-reference types have lower alignment requirements. I might also be a bit overly sensitive around this - I hear "why don't you simply do this [immensely complicated and fragile sketch of a proposal]" a LOT.
- ghusbands 4y ago
  Very clear and understood, thank you. I think I've not been helping by mainly talking about a hypothetical world in which Postgres had the feature from the start. Implementing it now would be far less worthwhile.

anarazel 4y ago

> I wonder whether you'd be willing to go into a meta-discussion on that, perhaps not public? I seem to hit this kind of stalemate too often and need someone's help to close the gap. I can see your email address in your profile. Feel free to email me. I'll be offline for part of the next two weeks though. > I don't see how consistent and repeated "can't work" answers can lead to a collaboration on an actual solution. I'm not even the proposer of the optimization, but I can see it has merit and doesn't have high intrinsic complexity. Well, it's easy to say that something doesn't have a high intrinsic complexity from the outside, if you haven't spent years in the guts of a system. > > always packing columns would turn an operation that doesn't currently rewrite into a rewriting operation > I don't think this is necessarily true. What is the operation that would rewrite that doesn't already, assuming the same shape of solutions already used? Adding a column to a table. Right now that doesn't require a rewrite, if it were always optimally packed it would (or alternatively increase the per-row overhead). > > The per-row overhead would be too large - we already have problems with that. > I read this as "the proposal can't work because [...]", on another thing that feels like a technicality. There are likely all sorts of bit-packing and other options that would make the overhead low, and the Postgres per-row overhead is already so large that it seems doubtful that it'd be that big a deal. Shrug. I've spent a lot of time working on postgres, including heap storage. Within the constraints of where we are, I am quite confident that it's not realistic to add that. There's larger possible redesigns (and I've spent working on attempts) that could make it more realistic, but it'd be a lot of work - and I'm doubtful it's the most productive way to spend the effort. I think time would be much more wisely spent removing the need for most of the alignment padding - for by-value types that's solely needed because of an ancient design choice, namely that system catalogs need a C struct compatible layout. That can be addressed in less harmful ways (two things discussed so far, 1) only pad by-value columns in catalog tables 2) auto-generate mapping code to/from the catalog tuple format to struct format). With that a lot of the padding overhead vanishes - many by-reference types have lower alignment requirements. I might also be a bit overly sensitive around this - I hear "why don't you simply do this [immensely complicated and fragile sketch of a proposal]" a LOT.

ghusbands 4y ago
Very clear and understood, thank you. I think I've not been helping by mainly talking about a hypothetical world in which Postgres had the feature from the start. Implementing it now would be far less worthwhile.