9 ms·
Postgres: The next generation
- alberth 3y agoWhat limitations does Postgres have that would cause you to use another data store? Or what big feature / functionally is Postgres missing (if any, like replication?)
- nerdbaggy 3y agoReplication for sure
- cannonpalms 3y agoWould you mind expanding on this? In what ways is PG's replication support lacking versus the competition?
- mattashii 3y agoAssuming you mean replication (as defined in https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION https://www.postgresql.org/docs/current/warm-standby.html#ST... or https://www.postgresql.org/docs/current/logical-replication.html https://www.postgresql.org/docs/current/logical-replication....), then PostgreSQL has it (see those docs for details). If you mean something else, could you elaborate?
- kumarvvr 3y agoAs an interested programmer, but with no experience in C/C++, I would really love to have a series of videos explaining the code in detail, so that I can begin to contribute.
- bagasme 3y ago> Postgres isn’t in any sense struggling to attract new users – there are plenty of 22 year olds defaulting to it today. Yes, I do use Postgres for a lot of my self-hosted apps. For PHP applications, though, I keep using MariaDB as their default/only DB.
- misiek08 3y agoMy observation is: most of people coming to IT these days are just money greedy, no enthusiasts anymore. It’s very sad and it seems like many open source projects die because of that - no back contributions or any help, only “copy and paste from SO and get paycheck”. I’m not saying everyone is like that by my close observations and conversations in people from few companies I contributed in show me 19:1 ratio… Disclaimer: I work for two companies every day because had so much free time because did my job too fast for standards and only wasted time waiting for meetings. Did a lot of side jobs to do something interesting, many time for free or to just use some new hardware and experiment with.
- pjungwir 3y agoI am hoping to be part of that "Next Generation"---although I'm already 46. I'm sure there are others a bit younger. My last talk at PGCon tried to fill some of the gaps around what you must know to hack on Postgres, especially the Executor Phase and TupleTableSlots. I'm not the most qualified, but sometimes it takes a learner to know what learners need. Just the other day I wrote a table of contents for a book about how to contribute to Postgres. I figure it would sell at least ten copies. Maybe a serial publication online would be better. Curious if anyone would be interested in either of those? For now Postgres is more like a hobby for me, but if someone is hiring to do open source Postgres contributions full time, I'm open to a chat. ;-)
- mch82 3y agoI’ve met a bunch of top engineers that kept working into their 70s and beyond because they couldn’t do the cool stuff at home. I’m very excited to see if more people may begin to retire earlier & get involved with open source now that participation is more broadly accessible.
- biehl 3y agoI would be happy to have a look at your book drafts.
- craigkerstiens 3y agoJust coming to say hi Paul! I recall you being very excited and getting the email after you submitted your first patch and loved it. I think a ton of the current set of "next generation" came to Postgres a good bit later. Even Tom himself will tell you he did some stuff with images for a few years which undersells himself (tiff, jpg, png - in some form involved in creation of each of those), then found this Postgres thing and started working on it.
- DrDroop 3y agoI've done a little bit of postgres work with the help of pgrx and I can recommend it as a platform to build data solutions. Another great resource has been the CMU channel: https://www.youtube.com/@CMUDatabaseGroup https://www.youtube.com/@CMUDatabaseGroup
- gavinray 3y agopgrx unfortunately has essentially zero documentation or samples on use outside of extensions. For instance, say you want to write a new Table Access Method handler. There are bindings to TableAM stuff in the core pg-sys SDK but no docs or examples on using it in Rust.
- zombodb 3y agoI don’t think we even expose the TableAM APIs? They are incredibly hard to generate bindings for from the C headers — lots of inline functions and complex #define macros. We have an ambitious goal with pgrx and it’s going to take many years and countless hours of developer effort to get there. It can, however, serve as a way for newcomers to gain experience with Postgres internals. Regarding the TableAM specifically, when we are able to create a safe Rust wrapper around it, that wrapper will be documented.
- gavinray 3y agoAh, curious what this bit does then, I think I might have misunderstood it: https://github.com/pgcentralfoundation/pgrx/blob/c2eac033856971ee0433762fb0fb26fbf73de137/pgrx-pg-sys/src/include.rs#L206-L220 https://github.com/pgcentralfoundation/pgrx/blob/c2eac033856...
- zombodb 3y agoI think maybe what you’re really looking for are the files here: https://github.com/pgcentralfoundation/pgrx/tree/c2eac033856971ee0433762fb0fb26fbf73de137/pgrx-pg-sys/include https://github.com/pgcentralfoundation/pgrx/tree/c2eac033856... Those are the internals we currently expose as unsafe “sys” bindings. As we/contributors identify more that are desired we add them. pgrx’ focus is on providing safe wrappers and general interfaces to the Postgres internals, which is the bulk of our work and is what will take many years. As unsafe bindings go, we could just expose everything, and likely eventually will. There’s just some practical management concerns around doing that without a better namespace organization —- something we’ve been working. The Postgres sources are not small. They are very complex, inconsistent in places, and often follow patterns that are specific to Postgres and not easy to generalize. If you’ve never built an extension with pgrx, give it a shot one afternoon. It’s very exciting to see your own code running in your database.
- candiddevmike 3y agoI wonder if C codebases are going to have problems finding maintainers in the future. Postgres has commercial backing and inertia, but it seems like we are lacking a pipeline of (proficient) C developers.
- deleted 3y ago[deleted]
- OJFord 3y agoI doubt it, universities basically choose between C & Java as the main if not only teaching language. Sure the former might evolve into C++, but it somewhat inherently starts as C, and I suspect you didn't mean 'C developers' to exclude people with C++ experience anyway? An organisation worried about it can easily hire grads/juniors with a corresponding proficiency, and ensure that they still have that by the time they're seniors by employing them to work in C.
- packetlost 3y agoIf people are worried about Postgres, they should be more worried about Linux maintainership.
- piaste 3y agoThere are many more billion-dollar companies that rely on the Linux kernel than there are that rely on Postgres.
- packetlost 3y agoI mean, fair.
- hedora 3y agoI'm guessing this won't be a major problem. I don't know any proficient Rust or C++ developers that are not also proficient C developers. I do wonder when even more C code bases are going to start seeing modules ripped out and replaced with Rust. This is already happening with Linux, curl, and with C++ stuff like Chrome, many MS products, Amazon S3, etc, etc. The biggest explicit holdout I know of is OpenBSD, and that's because they're trying to keep the bootstrap / base install toolchain small.
- wkoszek 3y agoMy aspirational goal is to be in a position to retire early and hack on Postgres full-time, it’s so interesting. It has networking, storage, data, algorithms etc. C is less of a problem tbh. Postgres has good code style and its pretty consistent. The complexity of the internals is, and having a small community may infer the speed in which people help you.
- teaearlgraycold 3y agoWe should have an NSF for FOSS contributions
- nikita 3y agoYou can work full time for someone as a Postgres committer.
- 1500100900 3y agoUnless you're Tom, you'll have to work on tickets assigned to you by some manager.
- samaysharma 3y agoThat's not really true any more. I personally know several Postgres contributors / committers who have a very high amount of control on what projects they work on.
- craigkerstiens 3y agoSo many thoughts on this. The community has definitely ebbed and flowed, on this for a while. A few varying pieces of insight with no intention other than to share a bit more on the PG community. And I'm sure some current and former colleagues already in comment threads are going to correct me on nuance of a lot of this. For several years there were no new committers at all. In recent years the team has tried to be a little more intentional about adding new ones and culling those no longer involved. About 15 years there was a phase of letting a lot of younger people earn their commit bit. I can recall 3 people by name that all got a commit bit before the age of 25, and they may have actually all been under 22. One of those three shortly after moved on to work outside of the Postgres community, another quietly was busy on other things for over 10 years before coming back, and the third was actively involved going forward. I suspect there was some unease of folks getting a commit bit and then sort of falling off a cliff so it slowed for a few years on adding new folks. Edit - sounds like it was less age driven but maybe still slightly related to some folks falling off that there was a slow down in new committers – tldr - you're not getting a commit bit right out of college for Postgres. What to me would be interesting but likely hard to gather is what age to people become a committer to Postgres. It wouldn't surprise me if the average age of getting a commit bit is closer to 45 than not. Many folks contributing come to Postgres after other systems work or just don't consider contributing to they're a bit more seasoned because it feels intimidating–I mean patches sent on a mailing list who does that any more? Postgres thats who.
- monkchips 3y agogreat contribution here from Craig, in terms of the ebbs and flows and useful history. i had no idea about that cluster of folks under 22 with commit bits.
- gavinray 3y agoThe thing about becoming a PG contributor is that the barrier to entry is fairly high. I love Postgres so much I have a PG tattoo, but from the perspective of the two ways you can contribute: - As a random user, in your free time: There's not a ton of "Good first issue" type tickets. Where you can ease your way into PG dev by working on something that doesn't require you to have context on many parts of the PG architecture and at least a little historical knowledge on why things are written the way they are. Also, it can be a bit intimidating to have your patches reviewed by the likes of Tom or Andres. - As a developer for a paid PG company like EDB/PG Pros/Crunchy etc: It's a sort of Catch-22 scenario here, where it's difficult to get hired as a junior without having previous PG hacking experience, but the path to doing that is not the easiest thing in the world. If I was going to work somewhere that wasn't $CURRENT_CO, it'd be somewhere doing PG work, but there's not a lot of viable avenues/inroads there.
- dist-epoch 3y ago> Who is going to be doing the work in 2043? ChatGPT. It won't matter anyway since we won't be around to worry about this question.
- DiggyJohnson 3y ago2043 is only 20 years away. Presumably many of us will be around and productive in that year.
- choffman 3y agoArticle says Postgres launched in 1986. Wikipedia says 1996.
- fanf2 3y agoThe POSTGRES project at Berkeley started in 1986, the open source successor became PostgreSQL in 1996.
- tristan957 3y ago> The implementation of POSTGRES began in 1986 https://www.postgresql.org/docs/current/history.html https://www.postgresql.org/docs/current/history.html Postgres started as a project at Berkeley.
- tristan957 3y agoI am a new contributor to Postgres as of 5 months ago. I turn 27 later this month. While I haven't contributed too much of value, I have a few commits here and there with plans to do other things, like make it easy to build Postgres extensions with Meson and hopefully drop the autotools build of Postgres ASAP. You might also catch me in the pgbouncer or pgvector repositories very soon. The story of how I came to contribute to Postgres is that I got tired of working at a software consultancy where I had worked for 3 years, 2 years during college, 1 year full-time. I had always more envisioned myself as an open-source systems software guy. I found a job at Micron working on an open-source storage engine. It kind of felt like luck to have found that job to be honest, but I felt like the job description was written for me, so I applied. I worked on that project for 2.5 years and loved it. Unfortunately, Micron laid off my entire team back at the end of February. I began searching for a job, and eventually received an offer from MongoDB to work on the C/C++ drivers, but that was rescinded. Then, I really started to lean into my network, and one guy I knew from #mesonbuild on Libera.Chat/Matrix worked on Postgres, so I messaged him and asked if there were any positions open in the Postgres space for someone with my background that he might be aware of. He told me that Neon was hiring, so I sent an application to work on their storage engine, but in the initial interview, my eventual manager thought that I would fit better in the new Postgres team they were forming, which would contribute upstream to Postgres. I am very appreciative to Neon for taking a chance on me. The topic of this blog post is interesting because it just came up during some discussions with other young Postgres contributors at PGConf NYC. Some points that we brought up: - It is hard to get patches looked at, even small ones. With better name recognition in the community, comes more patch reviews it seems, which is most likely the case in most projects, but still, it is a circular issue. - The organization of the Postgres mailing list is not very good. You are forced to drink from the firehose that is pgsql-hackers, whereas the LKML is organized into various subsystems. Modern code forges have value in that you can subscribe to certain tags on PRs/issues, which isn't the case with the current state of pgsql-hackers. - Adding things to a commitfest is a little burdensome. The only way to get a patch through the entire Postgres CI is adding it to the commitfest, and even then, you have to be proactive about checking it, or hope that a committer will tell you to look at the CI failure. - Bug reports are also sent to a mailing list (pgsql-bugs). There is no equivalent to the Linux bugzilla for Postgres for instance. - Patches are sent as attachments to emails, and not necessarily git-format-patched either, whereas the LKML uses git-send-email exclusively from what I can tell. All in all, it kind of seems like tooling in the Postgres contributor community works best for those that have been ingrained in it for 15+ years, which I guess is the case for most things. I don't want this to turn into a "Use GitHub/GitLab" post. Let it be known that I actually think email is the superior way to communicate about patches, but the tooling around the mailing list could improve. Everything seems very disjoint. SourceHut, I think, has done a good job of making mailing list development more approachable for the everyday contributor. Issues, mailing lists, CI/CD, and repositories are all connected to each other. There aren't separate services like they currently are for Postgres. This comment is probably worth a blog post of its own at some point, but I'll end it here. If you are also new to contributing to Postgres, perhaps we can share experiences. Email me at tristan <at> neon.tech or tristan <at> partin.io. Another Postgres contributor that I talked to thought it might be useful to have monthly meetings among non-committer contributors where we can talk about the patches we are working or have posted in order to get reviews from peers.
- throwjabdb 3y agoTL;DR PostgreSQL are aging and Neon os hiring and training juniors instead of existing commiters so the developer base grows.