9 ms·

Actually as great as Postgres is and as generally approachable the community is - my experience was the same a few times and I read it on the mailing list happe

by jarym 8y ago

Actually as great as Postgres is and as generally approachable the community is - my experience was the same a few times and I read it on the mailing list happening to others: Someone comes along with a patch or idea. Bunch of big Postgres people come knock it and it dies right there. Happened to me when I suggested a more multi-tenant approach back around 2010 and today we have Citus. I was told that (paraphrased) no users were asking for that sort of thing. I see it kind of happening with the Foreign Key Array patch that the author asked for help to rebase and no one bothered to reply. Someone suggested replacing the Postmaster with a threaded approach so it could scale better (and showed benchmarks of their implementation handling more connections). Community response was there were already 3rd party connection pools that do the job. An outsider looking in considers this nuts - most people would not run additional components if they did not need to! Another example: NAMEDATALEN restricts every identifier to 63 bytes - a limit that causes frequent issues (more so now that we have partitioning) and also a problem for multi-lingual table names. It’s been proposed to increase this limit a few times. Every time the answer has been: abbreviate your table names or recompile your own version of Postgres. Could name a few other examples too I’ve noticed over the years and just sighed at. I don’t expect every idea to be accepted or even welcomed - but there is that sense of bias against change.

anarazel 8y ago
Yea, that's definitely something that happens. Partially due to some conservatism, partially due to having way more patches than review bandwidth, ... > Another example: NAMEDATALEN restricts every identifier to 63 bytes - a limit that causes frequent issues (more so now that we have partitioning) and also a problem for multi-lingual table names. It’s been proposed to increase this limit a few times. Every time the answer has been: abbreviate your table names or recompile your own version of Postgres. I think if it were easy, it'd immediately be fixed, but there's a fair bit of complexity in fixing it nicely. For partially historical and partially good reasons object names are allocated with NAMEDATALEN space, even for shorter names. So just increasing it would further waste memory... We're going to have to fix this properly one of these days.
- jarym 8y ago
  Re NAMEDATALEN specifically - you at least acknowledge it eventually needs fixing. On the mailing list there were a lot of important Postgres people who don’t seem to agree. Agreeing that there is something worthy of fixing is a first step. It should have happened with this NFS patch and imo some other stuff. The considerations for how and when should be dear with separately.
  - pgaddict 8y ago
    Well, that's the thing - changing NAMEDATALEN is a seemingly small change, but it'll require much more work than just increasing the value. Increasing the value does not seem like a great option, because (a) how long before people start complaining about the new one and (b) it wastes even more memory. So I assume we'd switch to a variable-length strings, which however affects memory management, changes a lot of other stuff from fixed-length to variable-length, etc. So testing / benchmarking needed and all of that. Which is why people are not enthusiastic about changing it, when there are fairly simple workarounds (assuming keeping the names short is considered to be a workaround).
    - deleted 8y ago
      [deleted]
    - marcosdumay 8y ago
      > (a) how long before people start complaining about the new one Very likely many years, or even never. People don't use large names because they like it, they always prefer small ones. How much memory are we talking about?
      - pgaddict 8y ago
        > Very likely many years, or even never. People don't use large names because they like it, they always prefer small ones. Well, we don't have exactly a barrage of complaints about the current limit either. > How much memory are we talking about? Good question. The thing is - it's not just about table names. NameData is used for any object name, so it affects pretty much any system catalog storing name. A simple grep on the repo says it affects about 40 catalogs (out of 80), including pg_attribute, pg_class, pg_operator, pg_proc, pg_type (which tend to be fairly large). So the amount of additional memory may be quite significant, because all of this is cached in various places.
Squithrilve 8y ago
> Someone comes along with a patch or idea. Bunch of big Postgres people come knock it and it dies right there. This is not unique to Postgres. I've seen this behavior on many development mailing lists (e.g. Mutt-dev).
darkerside 8y ago
While this can and does suck when it happens to you, this is exactly what it takes to keep products focused so they don't die death by a thousand cuts (or feature requests). For every awesome feature embarked upon, there's an opportunity cost of bug fixes, stability updates, tech debt reductions, and other inglorious but necessary work. Aggressively de-scoping is the difficult but necessary work of keeping a product alive in a competitive marketplace. And yes, it's a marketplace even if the product is open source.
- eeeeeeeeeeeee 8y ago
  Yep, I think it’s an unfortunate side effect of dealing with an onslaught of bug reports, many of which are user error or someone else’s problem. It’s common in any kind of user support, you start seeing the same report over and over. I even saw something similar when I went to the ER recently. Even doctors will pickup on one thing you say and draw conclusions from that and dismiss everything else.
  - partomniscient 8y ago
    "Even doctors will pickup on one thing you say and draw conclusions from that and dismiss everything else." This pattern seems really common, and is what scares me about the future in general. The 'experts' concentrate on the stuff they understand / are best to the detriment of where the actual focus needs to be. In a lot of cases, this is despite the insistence of the supposed non-expert who is the one suffering as a result. Some of the worst cases are as a child, where you get in trouble twice. First for wasting adults time because you didn't tell them properly, and then again for pointing out that you did.
  - coolestuk 8y ago
    I've had this experience with doctors so often it is chilling. Elderly relatives writhing in pain, only to have doctors say it's indigestion (it was a perforated ulcer and an uncle had previously died from a wrongly diagnosed ulcer perforating). My partner was misdiagnosed with flu when it was pneumonia which then developed into pleurisy (I'd never seen either of the latter, but was telling the doctor that's what the symptoms looked like - 15 years later he still suffers pain from the pleurisy). I had an arm paralyzed through severe pain and the consultant doctor planned an operation "to cut the nerve" - I said I thought it was a frozen shoulder and that such a procedure was unnecessary; 6 months later the paralysis began to subside and the consultant agreed it was a frozen shoulder). Another relative died of bowel cancer that was said to be back pain (she died in the hospital where she worked). I know of several people who were telling the doctor they had cancer, only to have the doctors dismiss it as trivial, with most of these people dying because of their untreated cancer. As a child I had joint pains for years that were diagnosed as "growing pains" but turned out to a hip disease (younger cousins ended up with the same condition and because I'd already had it, they were more readily diagnosed by family members). In both directions (treating trivial as serious and treating serious as trivial) I've seen so many mistakes. I'd be much happier to see a doctor google the symptoms rather than jump to a conclusion about what is wrong. There's a famous anecdote where junior doctors are taught the importance of observation, by senior doctors tricking them into tasting urine. It doesn't seem to be a lesson they learn. Even when their own objective test results are contra-indicative of their pre-judgement I've seen doctors scratch their heads but stick with their incorrect pre-judgement. When doctors I know have a family member go into hospital, you should see how attentive my doctor friends get concerning what is being said and done to their relatives. Some doctors will not even allow relatives to go into hospital for non-emergency treatment at certain times of year (because of timetabling there can be very inexperienced doctors on duty at certain times of the year).
coldtea 8y ago
Well, just proposing things and waiting for the team to implement them is easy. The core team has also to prioritize, deal with already planned features, and shoot down tons of people with inane ideas as well (not just good ones).
jerrysievert 8y ago
> Could name a few other examples too I’ve noticed over the years and just sighed at. I don’t expect every idea to be accepted or even welcomed - but there is that sense of bias against change. not just bias against change. while there are some very talented and friendly people in the pg community, there are a few bad actors that are openly hostile and aggressive, that feel that they must be part of every discussion. it gets worse when it is a change that they made that is causing an issue, as ego takes a front seat. unfortunately, these few bad actors make dealing with the pg mailing lists in general very unpleasant, and have made myself (a popular extension maintainer) and others try to keep interaction to an absolute minimum. that's not good for the community.
- pgaddict 8y ago
  I'm honestly curious who are those bad actors, and examples of such behavior. I'm not trolling you, I really am curious - because that does not match my experience with the community at all. I'm sure there were cases of inappropriate / rude communication, but AFAICS those were rare one-off incidents. While you're apparently talking about long-term and consistent behavior. So I wonder who you're talking about? I can't think of anyone who'd consistently behave like that - particularly not among senior community members.
  - jerrysievert 8y ago
    I know that you read the mailing lists, just spending a week on bugs, or following any major or minor discussions on hackers should be enough for you to figure a couple of them out, but I'm not going to personally call anyone out by name on hacker news. my comment, which is based on the multiple interactions I've had with the community, stands as is: some fantastic people, and a few aggressive bad actors that spoil things.
ksec 8y ago
I kind of wish the pg community came and read this. I guess this is one of the reason why MySQL were much more dominant in the early days and pg had relatively small usage.
- pgaddict 8y ago
  IMHO there were other / more important reasons why MySQL was initially more successful.
garyclarke27 8y ago
Yes 63 Identifier limit is a real pain, especially for constraint names, where a long name can convey a lot of valuable info. NOT IN FKs ie opposite of current FK’s for IS IN, would be useful for XOR constraints I would guess they could be implemented quite easily using existing FK infrastructure.
- ams6110 8y ago
  Not the worst thing in the world. I have trouble reading code with very long identifiers. Conventionally, a line of code should not exceed 80 characters or so. That's pretty hard if identifers are 60+ characters by themselves. If your project has standard abbreviations for things, "63 characters ought to be enough for anybody"
  - majewsky 8y ago
    I had a project last year where DB names were autogenerated per tenant, with the tenant ID being a UUID, so you're left with 63-36=27 characters. Starts to feel narrow. (Putting tenant_id as a column was not an option because for each tenant, a third-party software was started that wanted to have its own DB.)
samfisher83 8y ago
I don't think this is really limited to Postgres. Any relatively big OS project the core committers they tend to do what they do. I submitted a patch to a pretty big OS project and pinged the person assigned to the area. That person pretty much wrote his own version of my patch. I guess it got fixed so that was good. I guess the good thing about open source is if you want you can just build your own version with whatever you want. However smaller projects are more receptive since they want all the attention and help they can get.
- anarazel 8y ago
  I think the "big project" angle is that maintainability is of higher priority than for smaller project. So more minor things need to to be addressed than in smaller projects. And that then either leads to being very nitpicky in reviews (which costs both sides cycles), or polishing patches pretty liberally (which costs the submitter experience, but reduces the time required for both sides).
amachefe 8y ago
> and today we have Citus. After reading this, I have been wondering if the other requests/ideas are not startup ideas
- pgaddict 8y ago
  I'm sure some of them are. And Citus is not the only startup pushing PostgreSQL in a way the community did not want to. If you look at https://wiki.postgresql.org/wiki/PostgreSQL_derived_databases https://wiki.postgresql.org/wiki/PostgreSQL_derived_database..., it's a pretty long list - some of the product are successful, some are dead. And then there are products adding some extra sauce on top of PostgreSQL (e.g. timescaleDB).

by jarym 8y ago

anarazel 8y ago

Yea, that's definitely something that happens. Partially due to some conservatism, partially due to having way more patches than review bandwidth, ... > Another example: NAMEDATALEN restricts every identifier to 63 bytes - a limit that causes frequent issues (more so now that we have partitioning) and also a problem for multi-lingual table names. It’s been proposed to increase this limit a few times. Every time the answer has been: abbreviate your table names or recompile your own version of Postgres. I think if it were easy, it'd immediately be fixed, but there's a fair bit of complexity in fixing it nicely. For partially historical and partially good reasons object names are allocated with NAMEDATALEN space, even for shorter names. So just increasing it would further waste memory... We're going to have to fix this properly one of these days.

jarym 8y ago
Re NAMEDATALEN specifically - you at least acknowledge it eventually needs fixing. On the mailing list there were a lot of important Postgres people who don’t seem to agree. Agreeing that there is something worthy of fixing is a first step. It should have happened with this NFS patch and imo some other stuff. The considerations for how and when should be dear with separately.
- pgaddict 8y ago
  Well, that's the thing - changing NAMEDATALEN is a seemingly small change, but it'll require much more work than just increasing the value. Increasing the value does not seem like a great option, because (a) how long before people start complaining about the new one and (b) it wastes even more memory. So I assume we'd switch to a variable-length strings, which however affects memory management, changes a lot of other stuff from fixed-length to variable-length, etc. So testing / benchmarking needed and all of that. Which is why people are not enthusiastic about changing it, when there are fairly simple workarounds (assuming keeping the names short is considered to be a workaround).
  - deleted 8y ago
    [deleted]
  - marcosdumay 8y ago
    > (a) how long before people start complaining about the new one Very likely many years, or even never. People don't use large names because they like it, they always prefer small ones. How much memory are we talking about?
    - pgaddict 8y ago
      > Very likely many years, or even never. People don't use large names because they like it, they always prefer small ones. Well, we don't have exactly a barrage of complaints about the current limit either. > How much memory are we talking about? Good question. The thing is - it's not just about table names. NameData is used for any object name, so it affects pretty much any system catalog storing name. A simple grep on the repo says it affects about 40 catalogs (out of 80), including pg_attribute, pg_class, pg_operator, pg_proc, pg_type (which tend to be fairly large). So the amount of additional memory may be quite significant, because all of this is cached in various places.

Squithrilve 8y ago

> Someone comes along with a patch or idea. Bunch of big Postgres people come knock it and it dies right there. This is not unique to Postgres. I've seen this behavior on many development mailing lists (e.g. Mutt-dev).

darkerside 8y ago

While this can and does suck when it happens to you, this is exactly what it takes to keep products focused so they don't die death by a thousand cuts (or feature requests). For every awesome feature embarked upon, there's an opportunity cost of bug fixes, stability updates, tech debt reductions, and other inglorious but necessary work. Aggressively de-scoping is the difficult but necessary work of keeping a product alive in a competitive marketplace. And yes, it's a marketplace even if the product is open source.

eeeeeeeeeeeee 8y ago
Yep, I think it’s an unfortunate side effect of dealing with an onslaught of bug reports, many of which are user error or someone else’s problem. It’s common in any kind of user support, you start seeing the same report over and over. I even saw something similar when I went to the ER recently. Even doctors will pickup on one thing you say and draw conclusions from that and dismiss everything else.
- partomniscient 8y ago
  "Even doctors will pickup on one thing you say and draw conclusions from that and dismiss everything else." This pattern seems really common, and is what scares me about the future in general. The 'experts' concentrate on the stuff they understand / are best to the detriment of where the actual focus needs to be. In a lot of cases, this is despite the insistence of the supposed non-expert who is the one suffering as a result. Some of the worst cases are as a child, where you get in trouble twice. First for wasting adults time because you didn't tell them properly, and then again for pointing out that you did.
- coolestuk 8y ago
  I've had this experience with doctors so often it is chilling. Elderly relatives writhing in pain, only to have doctors say it's indigestion (it was a perforated ulcer and an uncle had previously died from a wrongly diagnosed ulcer perforating). My partner was misdiagnosed with flu when it was pneumonia which then developed into pleurisy (I'd never seen either of the latter, but was telling the doctor that's what the symptoms looked like - 15 years later he still suffers pain from the pleurisy). I had an arm paralyzed through severe pain and the consultant doctor planned an operation "to cut the nerve" - I said I thought it was a frozen shoulder and that such a procedure was unnecessary; 6 months later the paralysis began to subside and the consultant agreed it was a frozen shoulder). Another relative died of bowel cancer that was said to be back pain (she died in the hospital where she worked). I know of several people who were telling the doctor they had cancer, only to have the doctors dismiss it as trivial, with most of these people dying because of their untreated cancer. As a child I had joint pains for years that were diagnosed as "growing pains" but turned out to a hip disease (younger cousins ended up with the same condition and because I'd already had it, they were more readily diagnosed by family members). In both directions (treating trivial as serious and treating serious as trivial) I've seen so many mistakes. I'd be much happier to see a doctor google the symptoms rather than jump to a conclusion about what is wrong. There's a famous anecdote where junior doctors are taught the importance of observation, by senior doctors tricking them into tasting urine. It doesn't seem to be a lesson they learn. Even when their own objective test results are contra-indicative of their pre-judgement I've seen doctors scratch their heads but stick with their incorrect pre-judgement. When doctors I know have a family member go into hospital, you should see how attentive my doctor friends get concerning what is being said and done to their relatives. Some doctors will not even allow relatives to go into hospital for non-emergency treatment at certain times of year (because of timetabling there can be very inexperienced doctors on duty at certain times of the year).

coldtea 8y ago

Well, just proposing things and waiting for the team to implement them is easy. The core team has also to prioritize, deal with already planned features, and shoot down tons of people with inane ideas as well (not just good ones).

jerrysievert 8y ago

> Could name a few other examples too I’ve noticed over the years and just sighed at. I don’t expect every idea to be accepted or even welcomed - but there is that sense of bias against change. not just bias against change. while there are some very talented and friendly people in the pg community, there are a few bad actors that are openly hostile and aggressive, that feel that they must be part of every discussion. it gets worse when it is a change that they made that is causing an issue, as ego takes a front seat. unfortunately, these few bad actors make dealing with the pg mailing lists in general very unpleasant, and have made myself (a popular extension maintainer) and others try to keep interaction to an absolute minimum. that's not good for the community.

pgaddict 8y ago
I'm honestly curious who are those bad actors, and examples of such behavior. I'm not trolling you, I really am curious - because that does not match my experience with the community at all. I'm sure there were cases of inappropriate / rude communication, but AFAICS those were rare one-off incidents. While you're apparently talking about long-term and consistent behavior. So I wonder who you're talking about? I can't think of anyone who'd consistently behave like that - particularly not among senior community members.
- jerrysievert 8y ago
  I know that you read the mailing lists, just spending a week on bugs, or following any major or minor discussions on hackers should be enough for you to figure a couple of them out, but I'm not going to personally call anyone out by name on hacker news. my comment, which is based on the multiple interactions I've had with the community, stands as is: some fantastic people, and a few aggressive bad actors that spoil things.

ksec 8y ago

I kind of wish the pg community came and read this. I guess this is one of the reason why MySQL were much more dominant in the early days and pg had relatively small usage.

pgaddict 8y ago
IMHO there were other / more important reasons why MySQL was initially more successful.

garyclarke27 8y ago

Yes 63 Identifier limit is a real pain, especially for constraint names, where a long name can convey a lot of valuable info. NOT IN FKs ie opposite of current FK’s for IS IN, would be useful for XOR constraints I would guess they could be implemented quite easily using existing FK infrastructure.

ams6110 8y ago
Not the worst thing in the world. I have trouble reading code with very long identifiers. Conventionally, a line of code should not exceed 80 characters or so. That's pretty hard if identifers are 60+ characters by themselves. If your project has standard abbreviations for things, "63 characters ought to be enough for anybody"
- majewsky 8y ago
  I had a project last year where DB names were autogenerated per tenant, with the tenant ID being a UUID, so you're left with 63-36=27 characters. Starts to feel narrow. (Putting tenant_id as a column was not an option because for each tenant, a third-party software was started that wanted to have its own DB.)

samfisher83 8y ago

I don't think this is really limited to Postgres. Any relatively big OS project the core committers they tend to do what they do. I submitted a patch to a pretty big OS project and pinged the person assigned to the area. That person pretty much wrote his own version of my patch. I guess it got fixed so that was good. I guess the good thing about open source is if you want you can just build your own version with whatever you want. However smaller projects are more receptive since they want all the attention and help they can get.

anarazel 8y ago
I think the "big project" angle is that maintainability is of higher priority than for smaller project. So more minor things need to to be addressed than in smaller projects. And that then either leads to being very nitpicky in reviews (which costs both sides cycles), or polishing patches pretty liberally (which costs the submitter experience, but reduces the time required for both sides).

amachefe 8y ago

> and today we have Citus. After reading this, I have been wondering if the other requests/ideas are not startup ideas

pgaddict 8y ago
I'm sure some of them are. And Citus is not the only startup pushing PostgreSQL in a way the community did not want to. If you look at https://wiki.postgresql.org/wiki/PostgreSQL_derived_databases https://wiki.postgresql.org/wiki/PostgreSQL_derived_database..., it's a pretty long list - some of the product are successful, some are dead. And then there are products adding some extra sauce on top of PostgreSQL (e.g. timescaleDB).