8 ms·

Looking Back at Postgres

290 points by craigkerstiens 6y ago

kcolford 6y ago
Is this the same Postgres as the PostgreSQL database we all know and love here on HN?
- spearo77 6y ago
  Yes, the abstract doesn't mention it, but the PDF has this– > 1 OPENING > > Postgres was Michael Stonebraker’s most ambitious project—his grand effort to build a one-size-fits-all database system.
- ryanmjacobs 6y ago
  Yessir. Originally inspired by a db system called "Ingress", it was called "Postgres" then eventual PostgreSQL to emphasize the SQL part
- breakingcups 6y ago
  Yes, although originally it did not support SQL but QUEL. see https://en.wikipedia.org/wiki/PostgreSQL#History https://en.wikipedia.org/wiki/PostgreSQL#History and https://en.wikipedia.org/wiki/QUEL_query_languages https://en.wikipedia.org/wiki/QUEL_query_languages
  - takeda 6y ago
    I wish how things would be if it was QUEL that would became the standard. The language looks closer to the access pattern that ORMs using.
boshomi 6y ago
See also: "The Postgres and Illustra Codelines" by Wei Hong: https://dl.acm.org/doi/pdf/10.1145/3226595.3226623 https://dl.acm.org/doi/pdf/10.1145/3226595.3226623
- macdice 6y ago
  I have done some work on parallelism in PostgreSQL, and I was fascinated to discover Wei Hong's work in this area from more than a quarter century ago: https://www2.eecs.berkeley.edu/Pubs/TechRpts/1992/ERL-92-3.pdf https://www2.eecs.berkeley.edu/Pubs/TechRpts/1992/ERL-92-3.p... The POSTGRES 4.2 tarball includes a bunch of his parallel query work, which you can find wrapped in #ifdef sequent. I assume that this was a precursor to XPRS, and it was ripped out of regular POSTGRES after that. I wonder if anyone knows what happened to the XPRS sources...
jansan 6y ago
Totally off-topic, but anybody who is into databases should at least have heard the name dBase, which was the most successful database in the 80s. However, they totally fucked up the release of version IV, which had serious reliability issues (not something you want in a database, in case you did not know). This lead to a dramatic decline and eventually the death of dBase. Who knows how the database and software landscape would look like had they released a version without major bugs.
- beagle3 6y ago
  I think it wasn't the IV version that killed it; After all, there were many dBase compatible programs at the time, e.g. FoxBase which were at least as good and comparably bug free. I suspect it's actually Microsoft Office with Access and a mature Excel that did it. Everyone was using office at the time (for Word, Excel, Powerpoint and the emerging Outlook), so access was "free"; and many things that dBase&co were used for, were actually becoming easier to do in Excel.
- typh00n 6y ago
  Not kidding, my dad still uses dBase databases in his five-man company to this day. (non-IT company) I even re-wrote some of his programs in C# 10 years ago, so he can use some of them with a modern GUI, but not replacing the database to stay compatible with his original software and tools. I was surprised that databases are contained in *.dbf files, and no real database-service is present to process database-calls.
  - eropple 6y ago
    At the time, and depending on the OS...where would the service be? ;)
    - typh00n 6y ago
      I think i do not understand the question ... it is a little warehouse if that is your question. :-)
sxcurry 6y ago
One of the first C programs I wrote pulled data from an Ingres Database, all running on a PDP 11/34. I now use PostgreSQL heavily on AWS. Thanks Michael Stonebraker!
- fcatalan 6y ago
  I'm still developing web services connected to an Ingres instance running on a rickety ~22 year old Sun server with lots of bits and pieces cannibalized from its long dead siblings. We have the replacement system almost ready but won't really be able to swap the thing until the project owner retires (yay politics!). It's important but not time-critical, so if it dies before he goes we can just eat the downtime and take a couple of weeks to migrate.
parhamn 6y ago
> The highest-order lesson I draw comes from the fact that that Postgres defied Fred Brooks’ “Second System Effect” [Bro75]. Brooks argued that designers often follow up on a successful first system with a second system that fails due to being overburdened with features and ideas. Postgres was Stonebraker’s second system... I really hope we pull off some of these software adages.
- VectorLock 6y ago
  >Postgres was Stonebraker’s second system. What was his first system?
  - deleted 6y ago
    [deleted]
  - fiddlerwoaroof 6y ago
    Ingres. https://en.m.wikipedia.org/wiki/Ingres_(database) https://en.m.wikipedia.org/wiki/Ingres_(database) However, if you look at the projects he’s worked on, he has a track record of writing database engines that end up in delivered database systems.
  - jonas21 6y ago
    Ingres. The "post" in Postgres refers to the fact that it was the follow-up to Ingres.
  - arminiusreturns 6y ago
    I am really interested in his latest work with SciDB and VoltDB also. Cool stuff to check out if you haven't yet.
- mjw1007 6y ago
  On the other hand, a lot of what made the original Postgres distinctive has ended up being either removed or considerably downplayed: the "object-relational" features, rules, versioned data, and so on. (I think what the paper refers to as "versioned data" is the "time-travel queries" stuff, not multi-version concurrency control which was added after Stonebraker's time.)
RedShift1 6y ago
A good read but man these PDF's are terrible to read on mobile.
- gen220 6y ago
  Do submitters write academic papers in "native" LaTex these days? Or do they write in some intermediary format like markdown? If it's the latter, it'd be pretty trivial to generate a more mobile-friendly html format. I wonder if arxiv supports something like this.
  - zozbot234 6y ago
    LaTeX2html is a thing too.
  - ufo 6y ago
    Most academic papers are still written directly in Latex.
    - sjy 6y ago
      (In computer science, physics and mathematics.)
  - scott_s 6y ago
    As others said, yes, academic papers are still written in LaTex. However, arXiv does not actually accept a pdf for submissions, but require the actual LaTex sources and they generate their own pdf. As LaTex-to-HTML improves, they could always generate it for all of the submissions for which they have the source.
    - gen220 6y ago
      That would be fantastic! I hope it’s on their long term roadmap :). They probably have the biggest corpus of LaTex source in the world, that’d be pretty useful to the project maintainers.
  - davidgerard 6y ago
felixge 6y ago
I love PostgreSQL history, so this was tons of fun to read me for me. While reading, I also ended up highlighting and sharing some of my favorite parts on twitter: https://twitter.com/felixge/status/1272613965219139585 https://twitter.com/felixge/status/1272613965219139585
danielovichdk 6y ago
I find Postgres a really good database but with really poor tools. I don't want to stir up a religious battle here, but the tooling around MSSQL is just that much better. And the sql dialect is imo more to my taste.
- kiwicopple 6y ago
  I also found this when I switched from MySQL. After a while you discover the right tool for the job, and often some tolls that don’t exist for other databases (things like PostgREST come to mind). Sometime I wish it was a bit more idiomatic though. We’re working on a lot of tooling at https://supabase.io https://supabase.io. We’re open source too. If there is anything in particular you’re missing, let me know
  - RNCTX 6y ago
    Since you guys are doing something I highly agree with and I replied to your announcement post as well... How about a UML/ERD diagram tool which can spit out schemas and, given a schema to start with, migrations? There are a couple dozen of these being offered as SAAS with varying levels of usability and complexity. If you built one of those you'd soon have the attention of everyone who is bitching about paying 20 dollars a month to see their own database in a way that management can visualize for their charts and graphs. ERD diagrams have existed for decades. Even Oracle gives such a tool away. But for Postgres, the easiest way to create a schema with migrations without a credit card is still to write a Django model by hand, afaik.
    - kiwicopple 6y ago
      That's a great idea for the diagrams. I'm slowly working towards dumping schemas/types for the database in various formats over here: https://supabase.github.io/pg-api/ https://supabase.github.io/pg-api/ Once we've stabalized this we will start on git integrations - link up your repo for to keep everything in code - your types, schema, tables, functions, etc. We are also brainstorming ways that we can offer branching - since we give a full database server, we can possibly "branch" the default "postgres" database into various schemas which you can test with. > ERD diagrams have existed for decades If you know of a good open source tool which we should be supporting for this, let me know. Otherwise we don't mind building from scratch!
lazyjones 6y ago
As someone who bet his startup in 2000 on Postgres (with little experience at the time) and registered on HN just to praise Tom Lane for being exceptionally helpful and forthcoming on the mailing lists, I enjoyed this but don't think it's very relevant to Postgres' success in the past 20-odd years - that was entirely due to a few good people who had the right priorities and made good decisions without drama and without being overly opinionated. A very notable exception in the world of open source...
- irjustin 6y ago
  > without being overly opinionated My take is Postgres is extremely opinionated, but in such a way that it is beneficial for the long run. PG adheres to the SQL standard far stricter than MySQL (i.e. select w/o order, ACID). From 2000-2010, it severely slowed the progress of features compared to MySQL (clustering was big topic for a while). But the problems of hacks, speed and probably a significant one, management/owners, caught up with MySQL. Feels like tortoise & the hare.
  - jeffdavis 6y ago
    "Postgres is extremely opinionated" When I think of the term "opinionated" as it applies to software platforms, I think of things that have intended purposes in mind and optimize for the ease of those use cases, potentially to the detriment of people doing other things. It also suggests a "one way to do it" approach. If that is the definition, I don't think Postgres fits. Postgres tries pretty hard to meet users where they are, and tackle uncommon or emerging use cases. Examples include JSON support, support for writing functions in various languages (javascript, python, perl, R, etc.), and C extensions that can hack the engine in innumerable ways.
    - chousuke 6y ago
      I don't think "opinionated" in the context of a database system means refusing to implement useful features. When I think of unopinionated software, I think of software like Jenkins; it feels like a plugin glue framework that does barely anything useful out-of-the-box and leaves it up to the user to create a working configuration out of the tons of plugins available. Aside from being immediately useful just as a basic relational data store, my impression of PostgreSQL is that the "opinion" is apparent in the way that new things get done. There seems to be a culture of either implementing things well or not at all, and to avoid rushing things just because people want the new feature right now.
rurban 6y ago
I wish he described more the problems of their insular approach to avoid SQL. It used something like postquel, nobody used. It only shined under a C/C++ API, but was not usable for users. Just for developers to map their objects to. An opinionated OO database, which didn't follow the standards. And there were much better OO - DB persistency mappings these days, like for Lisp. There you had to write no glue code. Postgres was a turkey. But everything changed when someone added SQL. Soon it became the very best SQL standards implementation. That's when I picked it up again, and stayed with it.

Is this the same Postgres as the PostgreSQL database we all know and love here on HN?

spearo77 6y ago
Yes, the abstract doesn't mention it, but the PDF has this– > 1 OPENING > > Postgres was Michael Stonebraker’s most ambitious project—his grand effort to build a one-size-fits-all database system.
ryanmjacobs 6y ago
Yessir. Originally inspired by a db system called "Ingress", it was called "Postgres" then eventual PostgreSQL to emphasize the SQL part
breakingcups 6y ago
Yes, although originally it did not support SQL but QUEL. see https://en.wikipedia.org/wiki/PostgreSQL#History https://en.wikipedia.org/wiki/PostgreSQL#History and https://en.wikipedia.org/wiki/QUEL_query_languages https://en.wikipedia.org/wiki/QUEL_query_languages
- takeda 6y ago
  I wish how things would be if it was QUEL that would became the standard. The language looks closer to the access pattern that ORMs using.

See also: "The Postgres and Illustra Codelines" by Wei Hong: https://dl.acm.org/doi/pdf/10.1145/3226595.3226623 https://dl.acm.org/doi/pdf/10.1145/3226595.3226623

macdice 6y ago
I have done some work on parallelism in PostgreSQL, and I was fascinated to discover Wei Hong's work in this area from more than a quarter century ago: https://www2.eecs.berkeley.edu/Pubs/TechRpts/1992/ERL-92-3.pdf https://www2.eecs.berkeley.edu/Pubs/TechRpts/1992/ERL-92-3.p... The POSTGRES 4.2 tarball includes a bunch of his parallel query work, which you can find wrapped in #ifdef sequent. I assume that this was a precursor to XPRS, and it was ripped out of regular POSTGRES after that. I wonder if anyone knows what happened to the XPRS sources...

jansan 6y ago

Totally off-topic, but anybody who is into databases should at least have heard the name dBase, which was the most successful database in the 80s. However, they totally fucked up the release of version IV, which had serious reliability issues (not something you want in a database, in case you did not know). This lead to a dramatic decline and eventually the death of dBase. Who knows how the database and software landscape would look like had they released a version without major bugs.

beagle3 6y ago
I think it wasn't the IV version that killed it; After all, there were many dBase compatible programs at the time, e.g. FoxBase which were at least as good and comparably bug free. I suspect it's actually Microsoft Office with Access and a mature Excel that did it. Everyone was using office at the time (for Word, Excel, Powerpoint and the emerging Outlook), so access was "free"; and many things that dBase&co were used for, were actually becoming easier to do in Excel.
typh00n 6y ago
Not kidding, my dad still uses dBase databases in his five-man company to this day. (non-IT company) I even re-wrote some of his programs in C# 10 years ago, so he can use some of them with a modern GUI, but not replacing the database to stay compatible with his original software and tools. I was surprised that databases are contained in *.dbf files, and no real database-service is present to process database-calls.
- eropple 6y ago
  At the time, and depending on the OS...where would the service be? ;)
  - typh00n 6y ago
    I think i do not understand the question ... it is a little warehouse if that is your question. :-)

sxcurry 6y ago

One of the first C programs I wrote pulled data from an Ingres Database, all running on a PDP 11/34. I now use PostgreSQL heavily on AWS. Thanks Michael Stonebraker!

fcatalan 6y ago
I'm still developing web services connected to an Ingres instance running on a rickety ~22 year old Sun server with lots of bits and pieces cannibalized from its long dead siblings. We have the replacement system almost ready but won't really be able to swap the thing until the project owner retires (yay politics!). It's important but not time-critical, so if it dies before he goes we can just eat the downtime and take a couple of weeks to migrate.

parhamn 6y ago

> The highest-order lesson I draw comes from the fact that that Postgres defied Fred Brooks’ “Second System Effect” [Bro75]. Brooks argued that designers often follow up on a successful first system with a second system that fails due to being overburdened with features and ideas. Postgres was Stonebraker’s second system... I really hope we pull off some of these software adages.

VectorLock 6y ago
>Postgres was Stonebraker’s second system. What was his first system?
- deleted 6y ago
  [deleted]
- fiddlerwoaroof 6y ago
  Ingres. https://en.m.wikipedia.org/wiki/Ingres_(database) https://en.m.wikipedia.org/wiki/Ingres_(database) However, if you look at the projects he’s worked on, he has a track record of writing database engines that end up in delivered database systems.
- jonas21 6y ago
  Ingres. The "post" in Postgres refers to the fact that it was the follow-up to Ingres.
- arminiusreturns 6y ago
  I am really interested in his latest work with SciDB and VoltDB also. Cool stuff to check out if you haven't yet.
mjw1007 6y ago
On the other hand, a lot of what made the original Postgres distinctive has ended up being either removed or considerably downplayed: the "object-relational" features, rules, versioned data, and so on. (I think what the paper refers to as "versioned data" is the "time-travel queries" stuff, not multi-version concurrency control which was added after Stonebraker's time.)

RedShift1 6y ago

A good read but man these PDF's are terrible to read on mobile.

gen220 6y ago
Do submitters write academic papers in "native" LaTex these days? Or do they write in some intermediary format like markdown? If it's the latter, it'd be pretty trivial to generate a more mobile-friendly html format. I wonder if arxiv supports something like this.
- zozbot234 6y ago
  LaTeX2html is a thing too.
- ufo 6y ago
  Most academic papers are still written directly in Latex.
  - sjy 6y ago
    (In computer science, physics and mathematics.)
- scott_s 6y ago
  As others said, yes, academic papers are still written in LaTex. However, arXiv does not actually accept a pdf for submissions, but require the actual LaTex sources and they generate their own pdf. As LaTex-to-HTML improves, they could always generate it for all of the submissions for which they have the source.
  - gen220 6y ago
    That would be fantastic! I hope it’s on their long term roadmap :). They probably have the biggest corpus of LaTex source in the world, that’d be pretty useful to the project maintainers.
- davidgerard 6y ago

felixge 6y ago

I love PostgreSQL history, so this was tons of fun to read me for me. While reading, I also ended up highlighting and sharing some of my favorite parts on twitter: https://twitter.com/felixge/status/1272613965219139585 https://twitter.com/felixge/status/1272613965219139585

danielovichdk 6y ago

I find Postgres a really good database but with really poor tools. I don't want to stir up a religious battle here, but the tooling around MSSQL is just that much better. And the sql dialect is imo more to my taste.

kiwicopple 6y ago
I also found this when I switched from MySQL. After a while you discover the right tool for the job, and often some tolls that don’t exist for other databases (things like PostgREST come to mind). Sometime I wish it was a bit more idiomatic though. We’re working on a lot of tooling at https://supabase.io https://supabase.io. We’re open source too. If there is anything in particular you’re missing, let me know
- RNCTX 6y ago
  Since you guys are doing something I highly agree with and I replied to your announcement post as well... How about a UML/ERD diagram tool which can spit out schemas and, given a schema to start with, migrations? There are a couple dozen of these being offered as SAAS with varying levels of usability and complexity. If you built one of those you'd soon have the attention of everyone who is bitching about paying 20 dollars a month to see their own database in a way that management can visualize for their charts and graphs. ERD diagrams have existed for decades. Even Oracle gives such a tool away. But for Postgres, the easiest way to create a schema with migrations without a credit card is still to write a Django model by hand, afaik.
  - kiwicopple 6y ago
    That's a great idea for the diagrams. I'm slowly working towards dumping schemas/types for the database in various formats over here: https://supabase.github.io/pg-api/ https://supabase.github.io/pg-api/ Once we've stabalized this we will start on git integrations - link up your repo for to keep everything in code - your types, schema, tables, functions, etc. We are also brainstorming ways that we can offer branching - since we give a full database server, we can possibly "branch" the default "postgres" database into various schemas which you can test with. > ERD diagrams have existed for decades If you know of a good open source tool which we should be supporting for this, let me know. Otherwise we don't mind building from scratch!

lazyjones 6y ago

As someone who bet his startup in 2000 on Postgres (with little experience at the time) and registered on HN just to praise Tom Lane for being exceptionally helpful and forthcoming on the mailing lists, I enjoyed this but don't think it's very relevant to Postgres' success in the past 20-odd years - that was entirely due to a few good people who had the right priorities and made good decisions without drama and without being overly opinionated. A very notable exception in the world of open source...

irjustin 6y ago
> without being overly opinionated My take is Postgres is extremely opinionated, but in such a way that it is beneficial for the long run. PG adheres to the SQL standard far stricter than MySQL (i.e. select w/o order, ACID). From 2000-2010, it severely slowed the progress of features compared to MySQL (clustering was big topic for a while). But the problems of hacks, speed and probably a significant one, management/owners, caught up with MySQL. Feels like tortoise & the hare.
- jeffdavis 6y ago
  "Postgres is extremely opinionated" When I think of the term "opinionated" as it applies to software platforms, I think of things that have intended purposes in mind and optimize for the ease of those use cases, potentially to the detriment of people doing other things. It also suggests a "one way to do it" approach. If that is the definition, I don't think Postgres fits. Postgres tries pretty hard to meet users where they are, and tackle uncommon or emerging use cases. Examples include JSON support, support for writing functions in various languages (javascript, python, perl, R, etc.), and C extensions that can hack the engine in innumerable ways.
  - chousuke 6y ago
    I don't think "opinionated" in the context of a database system means refusing to implement useful features. When I think of unopinionated software, I think of software like Jenkins; it feels like a plugin glue framework that does barely anything useful out-of-the-box and leaves it up to the user to create a working configuration out of the tons of plugins available. Aside from being immediately useful just as a basic relational data store, my impression of PostgreSQL is that the "opinion" is apparent in the way that new things get done. There seems to be a culture of either implementing things well or not at all, and to avoid rushing things just because people want the new feature right now.

rurban 6y ago

I wish he described more the problems of their insular approach to avoid SQL. It used something like postquel, nobody used. It only shined under a C/C++ API, but was not usable for users. Just for developers to map their objects to. An opinionated OO database, which didn't follow the standards. And there were much better OO - DB persistency mappings these days, like for Lisp. There you had to write no glue code. Postgres was a turkey. But everything changed when someone added SQL. Soon it became the very best SQL standards implementation. That's when I picked it up again, and stayed with it.