4 ms·

> Also, note that Postgres already has features that are in the same area - null bitmaps put the null-ness of columns at the start of the row, and adding a nint

> Also, note that Postgres already has features that are in the same area - null bitmaps put the null-ness of columns at the start of the row, and adding a ninth column moves all of the data in a row. It's all just implementation detail, not show-stoppers. It doesn't move all the data for existing rows, because every row has an indicator about the number of columns in that specific row. Compare that to adding a column and efficient packing requiring that the entire table be rewritten to fit the new column at its optimal place...

ghusbands 4y ago
In the same vein, you could just have an indicator of the number of columns of each alignment, since the simplest solution is to group columns by alignment and add new ones to the end of each. Perhaps try to fully evaluate the idea rather than just shooting it down at the first perceived hurdle. Repeatedly introducing strawmen like "entire table be rewritten" doesn't really help the discussion; the performance impediments and solutions are largely the same shape as what Postgres already does elsewhere. It's all just feeling a bit like the Windows Terminal team telling Casey that the obvious optimisations won't work or require a PhD to fully evaluate or such. It's not that hard and any systems developer could do it. It probably wouldn't be worth the effort, compared to other things, though.
- anarazel 4y ago
  > Perhaps try to fully evaluate the idea rather than just shooting it down at the first perceived hurdle. I'm not trying to shoot improvements in this area down? I've spent time reviewing patches in the past, and I plan to do so in the future. Perhaps try to give others a bit more benefit of the doubt. > Repeatedly introducing strawmen like "entire table be rewritten" doesn't really help the discussion; the performance impediments and solutions are largely the same shape as what Postgres already does elsewhere. I don't think it's a strawman. For one there's plenty complaints about those rewrites where we have them, so we try to avoid adding more of them. For another, always packing columns would turn an operation that doesn't currently rewrite into a rewriting operation - which'd wreck havoc for people with existing upgrade scripts etc. > In the same vein, you could just have an indicator of the number of columns of each alignment, since the simplest solution is to group columns by alignment and add new ones to the end of each. The per-row overhead would be too large - we already have problems with that.
  - ghusbands 4y ago
    > Perhaps try to give others a bit more benefit of the doubt. I wonder whether you'd be willing to go into a meta-discussion on that, perhaps not public? I seem to hit this kind of stalemate too often and need someone's help to close the gap. I can see your email address in your profile. I don't see how consistent and repeated "can't work" answers can lead to a collaboration on an actual solution. I'm not even the proposer of the optimization, but I can see it has merit and doesn't have high intrinsic complexity. > always packing columns would turn an operation that doesn't currently rewrite into a rewriting operation I don't think this is necessarily true. What is the operation that would rewrite that doesn't already, assuming the same shape of solutions already used? > The per-row overhead would be too large - we already have problems with that. I read this as "the proposal can't work because [...]", on another thing that feels like a technicality. There are likely all sorts of bit-packing and other options that would make the overhead low, and the Postgres per-row overhead is already so large that it seems doubtful that it'd be that big a deal. (For example, if you've very few columns, you don't need much space for the counts, and if you've more columns, the extra space for a few sizes isn't going to have much impact. Also, any active transaction range isn't going to have a large number of table shapes in it, so rows could likely use a table-definition-version number in place of any column counts. Or you could probably look it up in some bookkeeping structure via xmin and drop the per-row details. The fast path would remain when there are no recent shape changes.)
    - anarazel 4y ago
      > I wonder whether you'd be willing to go into a meta-discussion on that, perhaps not public? I seem to hit this kind of stalemate too often and need someone's help to close the gap. I can see your email address in your profile. Feel free to email me. I'll be offline for part of the next two weeks though. > I don't see how consistent and repeated "can't work" answers can lead to a collaboration on an actual solution. I'm not even the proposer of the optimization, but I can see it has merit and doesn't have high intrinsic complexity. Well, it's easy to say that something doesn't have a high intrinsic complexity from the outside, if you haven't spent years in the guts of a system. > > always packing columns would turn an operation that doesn't currently rewrite into a rewriting operation > I don't think this is necessarily true. What is the operation that would rewrite that doesn't already, assuming the same shape of solutions already used? Adding a column to a table. Right now that doesn't require a rewrite, if it were always optimally packed it would (or alternatively increase the per-row overhead). > > The per-row overhead would be too large - we already have problems with that. > I read this as "the proposal can't work because [...]", on another thing that feels like a technicality. There are likely all sorts of bit-packing and other options that would make the overhead low, and the Postgres per-row overhead is already so large that it seems doubtful that it'd be that big a deal. Shrug. I've spent a lot of time working on postgres, including heap storage. Within the constraints of where we are, I am quite confident that it's not realistic to add that. There's larger possible redesigns (and I've spent working on attempts) that could make it more realistic, but it'd be a lot of work - and I'm doubtful it's the most productive way to spend the effort. I think time would be much more wisely spent removing the need for most of the alignment padding - for by-value types that's solely needed because of an ancient design choice, namely that system catalogs need a C struct compatible layout. That can be addressed in less harmful ways (two things discussed so far, 1) only pad by-value columns in catalog tables 2) auto-generate mapping code to/from the catalog tuple format to struct format). With that a lot of the padding overhead vanishes - many by-reference types have lower alignment requirements. I might also be a bit overly sensitive around this - I hear "why don't you simply do this [immensely complicated and fragile sketch of a proposal]" a LOT.

ghusbands 4y ago

In the same vein, you could just have an indicator of the number of columns of each alignment, since the simplest solution is to group columns by alignment and add new ones to the end of each. Perhaps try to fully evaluate the idea rather than just shooting it down at the first perceived hurdle. Repeatedly introducing strawmen like "entire table be rewritten" doesn't really help the discussion; the performance impediments and solutions are largely the same shape as what Postgres already does elsewhere. It's all just feeling a bit like the Windows Terminal team telling Casey that the obvious optimisations won't work or require a PhD to fully evaluate or such. It's not that hard and any systems developer could do it. It probably wouldn't be worth the effort, compared to other things, though.

anarazel 4y ago
> Perhaps try to fully evaluate the idea rather than just shooting it down at the first perceived hurdle. I'm not trying to shoot improvements in this area down? I've spent time reviewing patches in the past, and I plan to do so in the future. Perhaps try to give others a bit more benefit of the doubt. > Repeatedly introducing strawmen like "entire table be rewritten" doesn't really help the discussion; the performance impediments and solutions are largely the same shape as what Postgres already does elsewhere. I don't think it's a strawman. For one there's plenty complaints about those rewrites where we have them, so we try to avoid adding more of them. For another, always packing columns would turn an operation that doesn't currently rewrite into a rewriting operation - which'd wreck havoc for people with existing upgrade scripts etc. > In the same vein, you could just have an indicator of the number of columns of each alignment, since the simplest solution is to group columns by alignment and add new ones to the end of each. The per-row overhead would be too large - we already have problems with that.
- ghusbands 4y ago
  > Perhaps try to give others a bit more benefit of the doubt. I wonder whether you'd be willing to go into a meta-discussion on that, perhaps not public? I seem to hit this kind of stalemate too often and need someone's help to close the gap. I can see your email address in your profile. I don't see how consistent and repeated "can't work" answers can lead to a collaboration on an actual solution. I'm not even the proposer of the optimization, but I can see it has merit and doesn't have high intrinsic complexity. > always packing columns would turn an operation that doesn't currently rewrite into a rewriting operation I don't think this is necessarily true. What is the operation that would rewrite that doesn't already, assuming the same shape of solutions already used? > The per-row overhead would be too large - we already have problems with that. I read this as "the proposal can't work because [...]", on another thing that feels like a technicality. There are likely all sorts of bit-packing and other options that would make the overhead low, and the Postgres per-row overhead is already so large that it seems doubtful that it'd be that big a deal. (For example, if you've very few columns, you don't need much space for the counts, and if you've more columns, the extra space for a few sizes isn't going to have much impact. Also, any active transaction range isn't going to have a large number of table shapes in it, so rows could likely use a table-definition-version number in place of any column counts. Or you could probably look it up in some bookkeeping structure via xmin and drop the per-row details. The fast path would remain when there are no recent shape changes.)
  - anarazel 4y ago
    > I wonder whether you'd be willing to go into a meta-discussion on that, perhaps not public? I seem to hit this kind of stalemate too often and need someone's help to close the gap. I can see your email address in your profile. Feel free to email me. I'll be offline for part of the next two weeks though. > I don't see how consistent and repeated "can't work" answers can lead to a collaboration on an actual solution. I'm not even the proposer of the optimization, but I can see it has merit and doesn't have high intrinsic complexity. Well, it's easy to say that something doesn't have a high intrinsic complexity from the outside, if you haven't spent years in the guts of a system. > > always packing columns would turn an operation that doesn't currently rewrite into a rewriting operation > I don't think this is necessarily true. What is the operation that would rewrite that doesn't already, assuming the same shape of solutions already used? Adding a column to a table. Right now that doesn't require a rewrite, if it were always optimally packed it would (or alternatively increase the per-row overhead). > > The per-row overhead would be too large - we already have problems with that. > I read this as "the proposal can't work because [...]", on another thing that feels like a technicality. There are likely all sorts of bit-packing and other options that would make the overhead low, and the Postgres per-row overhead is already so large that it seems doubtful that it'd be that big a deal. Shrug. I've spent a lot of time working on postgres, including heap storage. Within the constraints of where we are, I am quite confident that it's not realistic to add that. There's larger possible redesigns (and I've spent working on attempts) that could make it more realistic, but it'd be a lot of work - and I'm doubtful it's the most productive way to spend the effort. I think time would be much more wisely spent removing the need for most of the alignment padding - for by-value types that's solely needed because of an ancient design choice, namely that system catalogs need a C struct compatible layout. That can be addressed in less harmful ways (two things discussed so far, 1) only pad by-value columns in catalog tables 2) auto-generate mapping code to/from the catalog tuple format to struct format). With that a lot of the padding overhead vanishes - many by-reference types have lower alignment requirements. I might also be a bit overly sensitive around this - I hear "why don't you simply do this [immensely complicated and fragile sketch of a proposal]" a LOT.