23 ms·
PostgreSQL is the worlds’ best database
- throwaway_pdp09 7y ago"In the age old argument about the imperitive vs. the declarative5 programming models, it occurs to me that declarative programming is just imperative programming in a thin disguise. Each declarative token in a database query language ultimately maps to one or several algorithms which apply the declaration in imperitive terms" Oh, profound, profound! It occurs to me the author's a n00b. Anyway, I do MSSQL. Until recently CTEs in PG were an optimisation fence. That would have slaughtered performance for my work. Not that I'm knocking PG, 'best' depends upon the problem.
- mulmen 7y agoMSSSQL people seem really proud of their CTEs. What is it that makes CTEs in MSSQL so great? I wonder if that really disqualifies Postgres from the same task. Are MSSQL CTEs just the hammer for your proverbial nail? Can you use a different approach in Postgres to solve the problem by leveraging its strengths? My day job is a lot of Redshift. We use CTEs and temp tables depending on what we need. It’s based on Postgres but not really comparable when talking about performance optimization.
- throwaway_pdp09 7y agoHere's the last line of my post: "Not that I'm knocking PG, 'best' depends upon the problem" I did not want to get into a war with people emotionally involved in their favourite programming tool. The article said 'best'. By that measure PG was worse. Point is it doesn't matter, it's about what problem you are trying to solve and at what price. In many cases PG is best for that - do you understand? I knew people would see it as an attack on PG. It's so childish and predictable. I solve business problems, not cheerlead for a particular RDBMS. As for what made CTEs so great in MSSQL, I just explained in my first post. Look for 'optimisation fence', and understand that a) PG has changed and b) was not a personal attack. > I wonder if that really disqualifies Postgres from the same task. If you have 100GB of data being materialised as a temp table under the hood instead of having the predicates being pushed down, what do you think? > Are MSSQL CTEs just the hammer for your proverbial nail? Jesus, they're just a tool to get a job done. As for is there a way round it, probably, it just means more work for the programmer, and more stress for the SQL optimiser which will at some point fail as complexity climbs. > We use CTEs and temp tables depending on what we need same.
- cosmiccatnap 7y agoYou do realize you are the one getting emotional because you are clinging to an engine that consistently underperforms in benchmarks across the board and costs you a fortune right? Hope the job security is worth being an expert in a bunch of defunkt practices in 10 years nobody will want.
- throwaway_pdp09 7y ago> consistently underperforms in benchmarks across the board Show me mssql vs PG speed comparison on a recent industry standard benchmark. > and costs you a fortune right? Are you witless enough to think I'm unaware of the wallet-raping, labyrinthine licensing of MSSQL? > Hope the job security is worth... Since you apparently can't understand, let me spell it out for you: I've recently installed PG to start learning it because, depending on the problem, it IS the best solution. I do hope you can stop feeling the need to protect your pet software package and undersand these are in the end just tools to solve problems.
- pritambaral 7y ago> Show me mssql vs PG speed comparison on a recent industry standard benchmark. Does the mssql license allow for this though? Last I heard, they did have a DeWitt clause.
- throwaway_pdp09 7y agoTPC
- mulmen 7y agoIt was an honest question, professional to professional. I did not interpret your post as a personal attack nor did I intend my post as an attack on you. The last time I used MSSQL in anger I was an intern and Michael Jackson was alive. It’s been a while. My question is: “what is it about your work that makes (or made) MSSQL a better choice than Postgres or something else, specifically because of CTE differences.”
- Jonnax 7y agoThis is advertising of course. But if I had to select an SQL DB postgres is my only choice. Perhaps I don't know enough about databases and their differences. Anyone have some pros and cons of others? Like why would I pick MySQL, Microsoft, Oracle, Maria etc over Postgres? Apart from support that you gotta pay for.
- drdaeman 7y agoReplication, failover and high availability are complicated matters. I believe there are other SQL databases that may be said to handle this better than PostgreSQL. At the very least, CockroachDB is much simpler to set up and manage. Cannot vouch for its stability and complexity of abnormal emergency recoveries, though - haven't used it long enough, only had simplest outages (that it had handled flawlessly).
- xdanger 7y agoI've been following YugabyteDB ( https://www.yugabyte.com/ https://www.yugabyte.com/ ) with interest for similar reasons. It's build on postgresql so it supports more features than CockroachDB. They just forked PG on top of their clustered document store. And the newest version (2.1) is production ready (their words). Just haven't had the reason to dive in yet, but seems good. Their PR department just isn't as good as CockroachDBs one.
- ksec 7y ago>Replication, failover and high availability are complicated matters. I wonder why Postgre hasn't improved in these areas and instead leaving it to third party solution. I mean every year these few points are still listed as something that flavours MySQL.
- nicoburns 7y agoIt might be partly because Postgres is largely developed by the providers of "3rd-party" solutions for features like this. You could see it as an open-core model of sorts.
- 7y ago
- deleted 7y ago[deleted]
- gjs278 7y agopostgresql cannot even do master-master. it’s a disgrace. is every service dependent on one machine staying online? or some thin wrapper that “promises” to keep them in sync?
- thorin 7y agoThis talks a lot about Postgres (which is great ) but not about other databases so the title isn't very accurate. If you ignore cost it certainly doesn't explain why you'd work with Postgres rather than Oracle or SQLServer. I've used Oracle for 20+ years, SQLServer for > 10 years, and Postgres from time to time and would say they are all good tools for a general use case.
- DeathArrow 7y agoCost isn't an advantage for the company I work for because we licensed SQL server very cheap in big volume. If devs can work easier and faster with SQL server, then we end up saving money because time is money.
- irjustin 7y agoWhile I disagree that it's the world's best, it is my default for all projects and general RMDBS. I used to be MySQL but it felt like over time it was becoming hack on top of hack for each subsequent version while PG lacked a number of features in the early 2010's it has clearly caught up with great care to its codebase. There are scenarios that PG does not win and should not be used, but if we're talking about the most applications that it covers well, PG is it.
- sgt 7y agoOut of curiosity, what RDBM do you feel is the best?
- cosmiccatnap 7y agoIt's missing the part at the end that says "for general use" Plenty of noSQL and even MySQL/mariaDB features beat psql in the right problem space.
- jwr 7y agoPostgreSQL is great if you: a) are OK with using SQL (this is not obvious) b) do not need a distributed database I've spent a lot of time on looking at database solutions recently, reading through Jepsen reports and thinking about software architectures. The problem with PostgreSQL is that it is essentially a single-point-of-failure solution (yes, I do know about various replication scenarios and solutions, in fact I learned much more about them than I ever wanted to know). This is perfectly fine for many applications, but not nearly enough to call it "the world's best database".
- BozeWolf 7y agoSo what do you suggest then? Do you really care that much about the language? Shouldn't you care more about your data? If you are not OK with SQL there are abstractions available.
- deleted 7y ago[deleted]
- jwr 7y ago> Do you really care that much about the language? Shouldn't you care more about your data? If you are not OK with SQL there are abstractions available. I do care about SQL being a text-based protocol with in-band commands, bringing lots of functionality that I never need (I do not use the fancy query features for exploratory analysis, I have applications with very specific data access patterns). For my needs, I do not need a "query language" at all, just index lookups. And I would much rather not have to worry about escaping my data, because I have to squeeze everything into a simple text pipe.
- guywhocodes 7y agoThis is why I'm always looking at the new cool databases on the block before going back to postgres. The hard truth is that postgres is more annoying to operate than a lot of the modern alternatives when you need HA.
- hnarn 7y agoCan you just shortly elaborate on why PostgreSQL is not good for distributed scenarios for those of us who are/were under the impression that psql does have stable distribution features?
- oerpli 7y agoI used PostgreSQL as student and during research and had to switch to Microsoft SQL Server as my company uses that since forever. For features I used on both, the average quality of the SQL Server implementation is between middling and pathetic (mostly the latter). I cannot recall a single instance where I thought "that's nice" or anything positive about how SQL Server does something. At best, it has been "this is not awful". Things that sucked especially hard: - Datetime API: In postgresql it is a joy to use and most features one would like to have to analyze data are directly there and work as they should. The analysis for my master's thesis was completely done in postgresql [1] (including outputting CSV files that could then be used by the LaTeX document) - without ever feeling constrained by the language or its features. Meanwhile, the way to store a Timespan from C# via EF is to convert it to a string and back or convert it to ticks and store it as long. - The documentation: The pgsql documentation is the best of any project I have ever seen, the one for SQL Server is horrible and doesn't seem to have any redeeming qualities. - Management: SMSS is a usability nightmare and still the recommended choice for doing anything. I assume there are better ways but I haven't found them and if I have, they usually didn't work. - The query language: In psql you can do things like select a < b as ab, b < c as bc , count(*) from t group by 1,2; \crosstabview To get even half of that in SQL Server one would do select case when a < b then 1 else 0 end as ab, case when b < c then 1 else 0 end as bc, count(*) as count group by case when a < b then 1 else 0 end, case when b < c then 1 else 0 end I think crosstab queries are unsupported (there are PIVOTs but I haven't yet looked into them a lot). Maybe my use cases are a bit weird and I haven't encountered the better features of SQL Server yet. Would appreciate if someone could point out what I am doing wrong. [1]: https://github.com/oerpli/MONitERO/blob/master/sql/queries/ringsize_flow.sql https://github.com/oerpli/MONitERO/blob/master/sql/queries/r...
- dangerbird2 7y agoInteresting how mssql has bad documentation. Judging by my experiences with docs for azure and various windows and dotnet APIs, Microsoft's documentation is usually pretty solid. Not that it's surpising, since proprietary rdbs'es tend to prefer "vendor lock-in" over "making the product pleasant to work with so the customer doesn't jump ship"
- philliphaydon 7y ago
- mcny 7y agoLet’s say I have a table for reservations: 1. Item ID 2. Reservation duration What I would like is if I have an item ID of 1 and a reservation duration of today 10AM to 11AM and I try to add another record of item ID 1 and duration 10:39AM to 11:39AM the insert should fail. Apparently it is straightforward in db2 but not in postgresql?
- mrighele 7y agoPostgresql supports storing range of values, in particular ranges of timestamps. I never used them but the manual page shows an example on avoiding overlaps that looks a lot like your case: https://www.postgresql.org/docs/12/rangetypes.html#RANGETYPES-CONSTRAINT https://www.postgresql.org/docs/12/rangetypes.html#RANGETYPE...
- woutgaze 7y agoSomething like this will do the job in PostgreSQL: CREATE TABLE reservations ( item_id uuid, duration tsrange, EXCLUDE USING GIST (item_id WITH =, duration WITH &&) );
- nicoburns 7y agoCan this be made to work if the `id` and `duration` are in different tables (with a joining key)?
- INTPenis 7y agoMy first job in IT, I was 18, it was the early 2000s. I was a nerd but in professional IT I was essentially a blank slate. I lucked up and ended up with a real hacker for a boss. As early as 2001-2002 he had saved entire businesses by migrating them from mysql to postgres. He was a BSD guy and a postgres guy. He made me into a fanboy of both those technologies. So out of sheer luck I've preferred Postgres for over 15 years, and it has never disappointed me. Unlike BSD, which often disappointed me once I learned how much easier and mature Linux was to use.
- laumars 7y ago> Unlike BSD, which often disappointed me once I learned how much easier and mature Linux was to use. I was with you right until that last sentence. I'm not going to offer a counterargument because your statement is extremely genralised and ripe for flamewars but I will say it's not as clear cut as you stated.
- krainboltgreene 7y ago> As early as 2001-2002 he had saved entire businesses by migrating them from mysql to postgres. I'm extremely curious how this worked out.
- INTPenis 7y agoWell I was 18-19 so to me these were myths I heard re-told. But all I know is that they had been throwing hardware on a MySQL install to make it work better. He migrated them to postgres and they got much better performance and could get away with less hardware than they had with mysql. That was as much detail that I remember. Keep in mind I said by sheer luck I became a fanboy. Not by experience and competence. That came later.
- forinti 7y agoI manage Oracle and Postgresql instances and I agree with this, except for one small thing: Postgresql doesn't have packages. Otherwise, it is much simpler to manage than Oracle (which really is a system turned inside out). The documentation is really good and straight to the point (Oracle can't help but turn everything into an ad - nvl uses the best null handling technology on the market and such).
- mulmen 7y agoI spent many years on Oracle. I do not miss smug Tom Kyte blog posts about the pointlessness of BOOLEANs. Nor do I miss endless hours Googling around Burleson Consulting pages because we couldn’t afford support. I do kinda miss how easy it was to do partitioning but I’m on Redshift now so it doesn’t matter.
- fullstop 7y agoOh, man, that shirtless guy. Thanks for putting that back into my mind!
- dangerbird2 7y agoWhile not having packages isn't a huge deal for new projects, it can be a pretty huge stumbling block for dbas who want to migrate from Oracle to Postgres, but need an easy way to port their pl/sql code to pgpl/sql. Most of the syntactic differenes (like pgplqsl code needing to be delimited as a string) can be automated, or handed off to a junior programmer. Not being able to use Oracle's package system would require some major refactoring to how you organize your functions and procedures
- polskibus 7y agoIt would be if it had real clustered indexes like MS SQL, ie. Tables as btrees that are maintained with each update.
- ants_a 7y agoGood news is that the table access method API introduced in version 12 makes it possible to build b-tree based tables as extensions.
- redis_mlc 7y agoIronically, the blog post shoots itself in the foot by starting with Postgres security. Postgres has very poor security compared to MySQL, and in fact, I tell companies implementing compliance policies to shift to MySQL. https://www.cvedetails.com/metasploit-modules/vendor-336/Postgresql.html https://www.cvedetails.com/metasploit-modules/vendor-336/Pos... The reasons are: - Postgres' grant model is overly complex. I haven't seen anybody maintain the grants correctly in production for non-admin read-only users. By contrast, MySQL's are grants are simple to use and simple to understand. - Postgres' COPY FROM and COPY TO have been used to compromise the database by copying ssh keys to the server, amongst other things. - Postgres' version of upsert allowed any command to be run without checking the permissions. So the vaunted "software engineering" behind Postgres is not that solid. - Currently Postgres is subject to around a dozen metasploit vulnerabilities that any script-kiddy can execute. The simple fact is, if you use Postgres, you almost certainly have a security compliance problem. I could make the same arguments about replication, or online schema changes, multi-master writes, or any enterprise database feature. Some constructive advice to the Postgres developers is to take a week and add grant commands to limit COPY FROM and COPY TO, and look at the metasploit options and see what can be done ASAP. I'd appreciate if you're itching to write a hasty response that you actually check your facts first. If you're thinking, "How is it possible that everybody else is wrong about Postgres being the best?", just remember the decade of Mongo fanboism on HN. I cringed during that era, too. Source: MySQL and Postgres DBA.
- halayli 7y agoyour statements are nothing but opinions. It's also dishonest to make them sound like facts. If you care this much about security, you should have taken a look at CVE reports of each db. You'll find that historically MySql has almost double the number of vulnerabilities.
- ilogik 7y agodoes running Postgres on AWS RDS mitigate some of these problems?
- DeathArrow 7y ago
- JacKTrocinskI 7y agoIs PostgreSQL doing anything in the area of autonomous databases? Oracle seems to be doing a lot in this area right now and it seems to be promising.
- ggm 7y agoPostgreSQL and ZFS is a marriage made in heaven. Block aligns to disk behaviour, snapshots make great low latency db dumps trivial. Never regretted making the combo
- yjftsjthsd-h 7y agoAnd compression! I've gotten compression ratios above 3x on real production databases, which is the difference between "we need more disks by next month" and "this hardware will last the foreseeable future". Not to mention that it improves performance when lz4 can decompress faster than the disks can read.
- od0 7y agoI'm getting almost 4x on a 10TB production database! A little surprising the first time you see all the z_wr_iss processes flying :)
- pritambarhate 7y agoCan you please give us more info about your workload? Is it very high throughput DB? Especially for Inserts and Updates? I searched on ZFS and PostgreSQL performance a couple of years ago on the Internet and popular opinion seems to be databases with Copy on Write type of file systems don't give the best performance.
- riku_iki 7y agoI am not deep expert, but my impression is that Postgres MVCC is kinda Copy on Write approach already (no in place updates, but you write new page for each update), so it should be perfect marriage..
- ggm 7y agoI'm on a read-mostly DB. few clients, but constant loads. More inserts than updates.
- smilliken 7y agoFWIW, I've often called out my decision to use PostgreSQL and ZFS (on Linux) as my worst. Memory usage was extremely inefficient. ARC had to be set to half of what it should've been because it varied so dramatically, so half the system memory was wasted. ZFS would occasionally exhaust system memory causing a block on allocations for 10+ minutes almost daily, no ssh or postgresql connections could be opened in the meantime. Sometimes a zfs kernel process would get stuck in a bad state and require restarting the server. Many days were wasted testing different raid schemes to work around dramatic space inefficiencies (2-8x IIRC) with the wrong combinations of using disk block sizes, zfs record sizes, postgresql block sizes, and zfs record compression. Because zfs records are atomic and larger than disk pages, writes have to be prefaced with reads for the other disk pages, adding lots of latency and random IOPS load. Bunch of other issues, I could go on. I've since switched back to ext4 and hardware RAID. Median and average latency dropped an order of magnitude. 99th percentile latency dropped 2 orders of magnitude. These databases are at high load. If they had low load, and I wasn't expecting to grow into high load, I'd consider ZFS since it does have a bunch of nice features.
- bertil 7y agoI’m assuming anyone who has to make that decision already knows this but, while PostgreSQL is great to host a production database, it isn’t a great choice for an analytic database at scale, or to train or store your machine learning features. It works, but it’s not great at scale. You can get away with having a scheduled pg_dump early on, some reports on that, while you figure out an ETL/Messaging process — but picking something that handles concurrent large-scale queries will matter fast.
- yeellow 7y agoCould you name some better alternatives for analytical db?
- DeathArrow 7y agoWe use Teradata for our data warehouse. We have an enormous quantity of data in there (We are a multinational wholesale chain). We however plan to move to either BigQuery or an inhouse solution in the future. But for now Teradata is our workhorse and quite good at what it does.
- bertil 7y agoThere are a lot of commercial solution that are geared towards reporting, from old-school Oracle, TeraData even SAS (would not recommend that one). Some options that came our recently when “Big Data” was a sentence people said: Hive, RedShift, S3, Cassandra. New players that abstract a lot of the problems from the others and add Machine-Learning capacities on top: Google BigQuery, Snowflake. There are non-tabular structure like the graph based ones, Neo4j e.g.: I’ve yet to find a business where that’s a viable option for analytics for the first five years but I like that those are bringing something new. Most of those are great and work: they have connectors to whatever language or tool you want to use. The most promising tool you’ll need to handle most of the transformation is either AirFlow or preferably dbt. Those are is independent from the database, so don’t worry too much about the features that vendors tout. One key thing: monitor all the queries going to that database, find the expensive ones, those with suspicious patterns, etc. Spin-up, response time can matter, but they are rarely a problem for most “slow” analytic use cases; for instance, Google BigQuery takes 10 seconds no matter what you query and it’s fine. On the other hand, concurrency has been an issue for me more than anything: all the analysts and managers trying to update their dashboards on Monday at 10 am. You rapidly get to a point where prices are high and negotiated, so you want to think about your likely usage in the next years before you step into that meeting. Key decision: by default, prefer the tech that is closest to the rest of your stack because ingress is the easiest factor to predict. Looker will be mentioned: it’s on top of all that, downstream from AirFlow/dbt.
- me551ah 7y agoHaving migrated a postgres database to dynamodb, I would say that I'm never going to use postgres again. The problem with all SQL databases is that they are too easy to query and use. You add all kinds of select queries, joins and foreign keys and when traffic hits scramble to make it scale. NoSQL is hard to design but you can atleast be sure that once traffic hits, you don't have to redesign the schema to make it scale.
- hnarn 7y ago> You add all kinds of select queries, joins and foreign keys and when traffic hits scramble to make it scale. NoSQL is hard to design but you can atleast be sure that once traffic hits, you don't have to redesign the schema to make it scale. Surely this depends on how you set up your SQL database to begin with? I'm not familiar with NoSQL, so can you explain why "schemas" aren't necessary and scaling happens automatically?
- me551ah 7y agoYou could design a SQL database to be denormalized from the start, but then you are losing many of the advantages of a SQL database. I never said that schemas aren't necessary, just that using a strict NoSQL database forces you to think about scaling constraints early. This avoids(atleast partially) a schema redesign later.
- DeathArrow 7y agoYou don't have joins in NoSQL so you have to keep data denormalized. That means data isn't consistent and you can have an will have (at a high enough data volume) errors if you aren't extremely cautious on how you update and add new data. That might matter or not in a particular business case.
- kenfoo 7y agoPostgreSQL is also my go-to database for anything relational. My only issue with it though, is the need to run vacuum full (which locks tables for significant amount of time) to free up disk space to the operating system.
- unnouinceput 7y agoFor most of the projects where the DB really mattered, throughout my 10+ freelancer carrier, it came down to one thing that client really cared about. Performance. Nothing else mattered, not license price, not whistles and bells, not hype. My clients wanted to have data in front of their eyes the same second when they clicked the button. And when you have a table with 100 million rows in it, and an application is not loading data from that table to show it to its users in the same second, you already lost the battle. Out of my tries, my clients tries, and my friends tries, only one DB was up to this task. Not Maria, not Access, not Oracle, not NoSQL, not MSSQL, not MySQL - all of them failed. The only one was PostgreSQL. And before starting to bash me, please do this. Make a small application that will show a map, put 100 million points of interest on that map, that are contained in the table we talk about, and now as you scroll the map, select the middle of view as your circle and select on a small radius only those points of interest inside that radius. No more then a thousand points of interest, lets say. When you do that within a second, you got yourself a good database. For me PGSQL was the only one capable to do this reliable.
- Joeri 7y agoIt depends a lot on the implementation strategy. Some databases have GIS features, allowing you to query all points within a radius directly. I know oracle can do this, but it’s a paid addon (spatial and graph) and I don’t know how well it scales. On databases that have good scanning behavior you can use a geohash approach instead. Insert all points with a geohash as key, determine a geohash bounding box that roughly fits your circle, query the rows matching that geohash in their key, run a “within distance” filter on the results client-side. I used that approach in hbase and it could query really fast. I suspect it would perform well on pretty much any mainstream sql db, but admittedly I haven’t tried it.
- rv-de 7y agoOf those several worlds? Is that a subtle joke? I don't get it ...
- NicoJuicy 7y agoFor someone that is using c# , please have a look at the excellent Marten library that shows the awesome functionality of Postgress as a document ( eg. NoSql) + event store. Postgres is not only for Sql, as I see multiple people referencing only the SQL functionality. For those that work with events/projections, Postgress supports js migrations on events so you can update events in you store to the latest version.
- sandGorgon 7y agoi wish there was a management layer that runs and operates high availability postgres databases on AWS/GCP/Azure...but is significantly cheaper. AWS RDS makes it impossible to export snapshots to s3 (because the native snapshot storage is super expensive). EC2 m5.xlarge w/ 5000 GB storage costs 700$ RDS db.m5.xlarge w/ 5000 GB costs 1000$ Why is there no SAAS that lets me BYO my own EC2 and just runs and gives me a dashboardy experience ESPECIALLY for snapshots and restores ?
- kjagiello 7y agoSounds like https://scalegrid.io/ https://scalegrid.io/ could fit your use case
- briffle 7y agoGCP does have HA for postgresql: https://cloud.google.com/sql/docs/postgres/high-availability https://cloud.google.com/sql/docs/postgres/high-availability
- sandGorgon 7y agoGCP managed databases are more expensive than AWS. They claim this is because (unlike AWS), they don't provision iops as "credits". There is an opportunity to play in this price arbitrage space where you can take the cloud instances and manage a database on top of it.
- KristiMKE 7y agoCheck out ScaleGrid's BYOC plans. They are similar to RDS, but give you the option to host in your own AWS or Azure account. ScaleGrid supports MongoDB, Redis, MySQL and Postgres. https://scalegrid.io/pricing.html#section_pricing_byoc https://scalegrid.io/pricing.html#section_pricing_byoc
- jonnypotty 7y agoMeasurably? Im confused.
- tannhaeuser 7y agoIdk, I'm very pro-PostgreSQL but MySQL/InnoDB used to have that one feature for easy replication via logical DB updates (sending UPDATES to replicas) whereas PostgreSQL support for this was considered lacking, using binary block updates instead if I'm remembering correctly. Also, MySQL, with replication, was considered easier to setup and manage by your average web shop. That may be a thing of the past, though. Edit: also, best for what? SQLite is, in a way, the perfect DB is you don't have concurrent writes, and I've worked with vector databases (think bitmap indices on steroids) that totally outclass eg. Oracle and PostgreSQL for analytics-heavy workloads
- andruby 7y agoThat used to be the case, but it's not anymore. Postgresql has logical replication built-in since Postgresql 10 (released 2017-10-05). It's really simple to run these days and the docs are great [0] [0] https://www.postgresql.org/docs/12/logical-replication.html https://www.postgresql.org/docs/12/logical-replication.html
- mekster 7y agoWhen I learned you cannot change the order of columns in PostgreSQL, it sounded like one of the smarter but slightly weird kid in a class room. On the other hand, MySQL, who's not as bright as the smarter kid, is easy to talk to and feels friendlier and is generally the more popular kid. I do appreciate the strictness of PostgreSQL but if I see small weird stuff, I tend to pick the one that is easier to get along with (meaning, more resource found on the net.) Also to mention that MySQL is also getting 'brighter" since version 8.
- thequux 7y agoI'll admit to not being an expert in databases, but I can't figure out why the order of columns in a database would be relevant, unless you're doing `select *` and indexing the resulting columns by number rather than name, which I've always found to be fragile to the point of being useless. What am I missing?
- barryp 7y agoIt doesn't matter for coding or functionality, but it is nice when you're manually eyeballing the results of 'SELECT * ...' statements or browsing tables with something like Navicat, or looking at schema dumps - to have your fields ordered in some way that makes sense to you, rather than strictly in the order they were created. Sure, you can list the fields in the order you want in a SELECT statement, but that's tedious - it's handy to have something reasonable in the table definition. There's a wiki page on the Postgres site: https://wiki.postgresql.org/wiki/Alter_column_position https://wiki.postgresql.org/wiki/Alter_column_position talking about workarounds and a plan from 2006 on how they might implement that feature.
- jbergens 7y agoFor scaling I assume some cloud solution or very distributed solution is better than PostgreSQL. For example Azure SQL Database Hyperscale or CockroachDb. https://redmondmag.com/articles/2019/07/03/microsoft-building-biggest-cloud-database.aspx https://redmondmag.com/articles/2019/07/03/microsoft-buildin...
- qilo 7y agoTechnically, PostgreSQL is a DBMS, not a database. I.e. “PostgreSQL is the worlds’ best DBMS”, but not “best database”. The problem is, if you call the DBMS a database, so what do you call the database (collection of data, tables, etc.)?
- dcwca 7y agoIs this really a problem?
- qilo 7y agoWhen I knew nothing about databases, and was trying to read stuff on the web about those, it was very confusing trying to understand what the author talks about when he uses the term database, is it about a DBMS (software), or it’s about a database (data). So, yeah, it’s a problem, at least for beginners.
- jhoechtl 7y agoBi-temporal tables support badly wanted. https://mariadb.com/kb/en/temporal-data-tables/ https://mariadb.com/kb/en/temporal-data-tables/
- jacques_chester 7y agoYes, though the SQL2011 temporal support was a dud compared to the state of the theoretical art. A strong whiff of vendor-vs-vendor nonsense. Folks in the PostgreSQL community are aware of temporal tables: https://www.pgcon.org/2019/schedule/events/1336.en.html https://www.pgcon.org/2019/schedule/events/1336.en.html https://www.2qpgconf.com/wp-content/uploads/2016/05/Chad-Slaughter-_-Henrietta-Dombrovskaya-Bitemporality-in-PostgreSQL_-What%E2%80%99s-New.pdf https://www.2qpgconf.com/wp-content/uploads/2016/05/Chad-Sla...
- lllr_finger 7y agoPostgres use always reminds me of this presentation: http://boringtechnology.club/ http://boringtechnology.club/ I self-admittedly love esoteric databases and storage engines to a fault. I'll try to shoehorn things like RocksDB into whatever personal project I'm working on. At work however, the motto I spread to the teams I work on is "use Postgres until it hurts". And for many, many teams - Postgres will never hurt. I'm very happy for its continued existence because its been a solid workhorse on various projects I've worked on over the years.
- duckmysick 7y ago> I self-admittedly love esoteric databases and storage engines to a fault. Any interesting notes and observations after using those esoteric tools?
- lllr_finger 7y agoPutting thought into design patterns and a storage engine that accents that choice, it's sometimes possible to eschew caching and distributed storage altogether - saving you from a whole host of complexity. It can also be challenging to accurately assess a database's performance and correctness. I've been using C/Rust bindings when possible for the former, and Aphyr's Jepsen test results for the latter. Unfortunately there's no silver bullet on these topics - you pretty much have to test all your use cases.
- cryptonector 7y agoWhat a great presentation. Thanks for the link! BTW, here's an interesting observation: if you normalize a schema to the max, applying CRDT techniques for eventually-consistent multi-master concurrency is relatively simple, and you can do it using SQL. With PG you could have each instance publish a replication publication of an event log, and each instance could subscribe to a merged log published by any of N instances whose job is to merge those logs, and then each instance could use the merged log to apply local updates in a CRDT manner. If you normalize to the max, this works. For example, instead of having an integer field to hold a count (e.g., of items in a warehouse of some item type) you can have that many rows in a related table to represent the things being counted. Now computing the count gets a bit expensive (you have to GROUP BY and count() aggregate), but on the other hand you get to do CRDT with a boring, off-the-shelf, well-understood technology, with the same trade-offs as you'd have using a new CRDT DB technology, but with all the benefits of the old, boring technology.
- axegon_ 7y agoI'd like to add one more thing about postgesql : the source code is arguably the cleanest code you'll ever see. It's second to none.
- irrational 7y agoWe were on Oracle for 15 years, but the cost was just too high. We decided to move to Postgres. We thought it wouldn’t be as good as Oracle (you get what you pay for, right?), but it ended up being better. Performance is better. Documentation is better (by far). Adherence to SQL standard is better. Many SQL queries are simpler. Null/empty string is sane. Etc. Now I don’t have experience with MySQL, SQLServer, etc., but between Oracle and Postgres, Postgres is definitely the best database.
- briffle 7y agoMy last employer was a large Oracle place, although everything was standard edition. I am so much happier now. Everything that used to be a 'that is an enterprise edition feature' is now baked in. concurrent index creation is amazing. The simplicity and speed of pgBarman vs RMAN is mind boggling. RMAN in Oracle Standard Edition can only do full backups. Full read-only access to a standby? That is huge! (enterprise only for the big O) Realtime replication (not log shipping), also enterprise only for the big O. I'm a sysadmin, so that is what I notice right away, but our developers love it too, for many other reasons.
- nickpeterson 7y agoI think a big portion of this isn't a difference in quality, but rather complexity. Postgres is just easier to understand than Oracle. Oracle has insane amounts of settings and configurable parts. The end result is that people who admin postgres servers tend to learn the levers, but many Oracle DBAs simply don't, or have environments where hand tuning is really hard to justify time-wise. I think Postgres success is it's often in environments where DBAs and devs work closely or are the same person.
- marcosdumay 7y agoOh, God. For everybody there afraid of moving form Oracle to Postgres, I have one fundamental piece of information: Postgres backups work. You can simply backup your database. And you simply restores your backup. And the result is a working database, without errors.
- oftenwrong 7y agoAs mentioned, PostgreSQL's documentation is some of the best in the software world. With most software project documentation, one ends up scouring StackOverflow, mailing list archives, and GitHub issues trying to figure out how to do something. With Postgres, I can just go straight to the official documentation, and read it. It has yet to fail me.
- jakearmitage 7y agoI get all the comparison, but Redis? I still use Redis as a cache on top of my PostgreSQL queries. PostgreSQL simply can't be as fast as a K/V like Redis.
- smt88 7y ago> I still use Redis as a cache on top of my PostgreSQL queries Why? Postgres has a "Redis cache" (in-memory query cache) built in already[1]. Your application layer doesn't have to worry about query caching at all. 1. https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-SHARED-BUFFERS https://www.postgresql.org/docs/current/runtime-config-resou...
- jakearmitage 7y agohttps://www.peterbe.com/plog/redis-vs-postgres-blob-of-json https://www.peterbe.com/plog/redis-vs-postgres-blob-of-json
- ants_a 7y agoWhile I have no doubt that Redis is faster for slinging blobs around, that blog post is not great comparison of the technologies. Both Django ORM and Python psycopg2 driver are not performance oriented tools.
- jatone 7y agothis is so egress of a misrepresentation its absurd. redis in its default configuration is in memory. postgresql stores to disk. I doubt they was running anything in parallel. (python) if you tune postgres by using a RAM disk and actually use parallelism it blows redis out of the water hands down.
- jatone 7y agoyes it can. in fact it can be faster due to the fact it isn't single threaded. learn about table spaces and RAM disks.
- T3RMINATED 7y agoIm surprised PG isnt taking over PHP instead Maria is. wth
- zerubeus 7y agoI'm a simple man, I see PGSQL in a HN post, I upvote, I don't even read the article
- drdec 7y agoI think that PostgreSQL itself is great. However, the developer client tools (pgAdmin) leave something to be desired. The old pgAdmin3 was fine, if lacking bells and whistles. The new pgAdmin4 however, was not very good last time I tried it [1]. It was implemented as a client/server web application and had frequent issues. Can anyone recommend a good client for PostgreSQL? [1] - I see that there have been some new releases in 2020 so I ought to check on them. The version I tried earlier was 4.11.
- nicoburns 7y agoIf you're on a mac, then Postico is pretty good (https://eggerapps.at/postico/ https://eggerapps.at/postico/). Otherwise, TablePlus is cross-platform (and supports multiple DBs) https://tableplus.com/ https://tableplus.com/ Agreed that PgAdmin is awful.
- iruoy 7y agoTablePlus is awesome on MacOS, but not so much on Windows. I use TablePlus for quickly looking stuff up when I'm working on my MacBook, but I use DataGrip when I'm working on Linux.
- egeozcan 7y agoI was on the fence about giving TablePlus a go. Why isn't it awesome on Windows? I'm very interested.
- eyelidlessness 7y agoI use Postico and agree it's pretty great overall. There's some odd behavior in the SQL editor that sometimes bites me when I'm pasting from my code editor (VSCode), where whitespace sometimes picks up some kind of invisible character or something that makes the query invalid. I have no idea how to reproduce it but it can really break my flow when I hit it. Other than that I've been pretty pleased with it. If I didn't already have a Postico license I'd definitely be looking at TablePlus.
- petargyurov 7y ago
- _kyran 7y agoWhile we're all here talking about databases I've got a question for the DBA/well versed software engineers. What database or database design strategies to be used for an analytics dashboard that can be queries real-time similar to Google Analytics? The first real scaling issue I've come across (as a junior dev working on a side project) is a table with 38 million rows and rapidly growing. Short of just adding indexes and more RAM to the problem, is there another way to design such a table to be effectively queried for Counts and Group Bys over a date range? Iimagine things can't really be pre-comuted too easily due to the different filters?
- slindz 7y agoI'm not sure how much flexibility you have, but have you taken a look at Prometheus (for time series data) + Grafana (for dashboards) I'm certainly not a DB expert, but this is where a similar need led me a few months ago.
- kthejoker2 7y agoYou want an OLAP database, something like Clickhouse.
- resolaibohp 7y agoI currently run analytic queries on a page view dataset around ~600 million rows on postgres. You can start by looking into partitioning by date. Also take a look at your IO on your queries. IO was the biggest bottleneck for me. Make sure you have hardware that is not limiting it.
- kardianos 7y agoI use PostgreSQL and MS SQL Server. I love Postgres. There are some things that SQL Server does differently that I would love to see supported in Postgres: * Protocol: (a) no wire level named parameters support; everything must be by index. (b) Binary vs text is is not great, and binary protocol details is mostly "see source" (c) no support for inline cancellation: to cancel a query client can't signal on current TCP connection, the client must open or use a different connection to kill an existing query, this is sometimes okay, but if you are behind a load balancer or pooler this can end up being a huge pain. * Multiple Result Sets from a single query without hacks. While you can union together results with similar columns or use cursors, these are hacks and work only in specific cases. If I have two selects, return two result sets. Please. * Protocol level choosing of language. Yes, you can program in any language, but submitting an arbitrary query in any language is a pain and requires creating an anonymous function, which is a further pain if your ad-hoc query has parameters. I would love a protocol level flag that allows you to specify: "SQL", "PG/plSQL", "cool-lang-1" and have it execute the start of the text as that. I do love Postgres recently added Procs! Yay!
- jeffdavis 7y agoThat's an interesting list. Can you give more context about how you would like to use these features?
- kardianos 7y agoSure. I have two roles: database driver/client maintainer and database based application developer and maintainer. As a database Driver/Client developer: * Having two protocols/data types that two the same thing (text / binary) isn't the end of the world, but it just adds confusion. Also it adds complexity for a different server/proxy/pool implementation. Recommendation: better document binary and add an intent to remove text data types. * Not having an inline cancellation (in same TCP/IP connection) means cancellation isn't supported by many drivers, and even when it is, there are many edge cases were it stops working. Each client implementation has to work around this. As an application developer, I typically see three patterns emerge: 1. Put SQL on SQL platform: stored functions / procs. Application only calls stored functions / procs. 2. Put SQL on Application platform: simple stored SQL text, query builders. 3. Make Application dump and use simple CRUD queries and put logic fully in application. I typically find myself in category (2), though I have no beef with (1). Benefits of (2) include: (a) easier to create dynamic search criteria and (b) use a single source of schema truth for application and database. * Multiple Result Sets: - (A) The screens I make often have a dynamic column set. To accomplish this I may return three result sets: row list, column list, and field list. This works for my screens and print XLSX sheets that can auto-pivot the data in, while all the data transfer columns can be statically known. and pre-declared. This allows me to edit a pivoted table because each cell knows the origin row. - (B) Any reasonable amount of work in SQL may be long-ish (300-1000+) lines of SQL. There are often many intermediate steps and temp tables. Without Multiple Result Sets, it is difficult to efficiently return the data when there are often multiple arities and columns sets that are all relevant for analysis or work. So if I take a fully application centric view (more like application mindset (3)), you just call the database alot. But if you are more database server centric (1) or (2), this can pose a real problem as complexity of the business problem increases. (I'm aware of escape hatches, but I'm talking about strait forward development without resorting to alternative coding.) - (C) Named parameters are extremely useful for building up query parameters in the application (development model (2)). You can specify a query where-clause snippet, the parameter name and the have the system compose it for you that is impossible with ordinal queries. A driver or shim can indeed use text replacement, but that (a) adds implementation cost and (b) server computation / allocation cost, (c) mental overhead when you see trace the query on the server. Further more it composes poorly with stored functions and procs on the server (in my opinion). It is again not insurmountable, but it is another thing that adds friction. Lastly, when you have a query that takes over 30 or 50 parameters, you must use named parameters; ordinal positioning is too error prone at scale. - (D) Protocol level language selection. PostgreSQL always starts execution in plain SQL context. If you always execute functions as PL/pgSQL it is just extra overhead. In addition, running ad-hoc PL/pgSQL with named parameters isn't the most easy thing. It is possible, just not easy. This feature plus named parameters so by the time I'm writing my query, I know (a) that I have all my sent parameters available to me bound to names and (b) the first character I type is in the language I want. The combination of these features would make applications developed in model (2) go from rather hard to extremely easy. It would also make other development modes easier I would contend as well.
- throw0101a 7y agoIs there a Galera-equivalent for Postgres? There's BDR, but it's closed source for recent version Pg AFAICT.
- Beefin 7y agoTheir JSON data type is atrocious, does anybody actually use it or is it a checkbox item?
- jacques_chester 7y agoYou can join on arbitrary json fields, with indices. It is an extremely useful swiss army knife. I've used jsonb for performing ETL in-database from REST APIs. I know Concourse from 6.0 uses it for storing resource versions. They had an inputs selection algorithm that became stupidly faster because they can perform joins over JSON objects provided by 3rd-party extensions. Previously it was a nested loop in memory. https://github.com/concourse/concourse/releases/tag/v6.0.0#3602 https://github.com/concourse/concourse/releases/tag/v6.0.0#3...
- Beefin 7y agoIsn't it just a string? can you do nested queries? secondary indexes? really anything beyond just querying the key-value pairs?
- jacques_chester 7y agoIt's a real type. You can do nested queries, follow paths into the structure etc. I've performed quite sophisticated joins over jsonb records.
- kingdomcome50 7y agoPostgreSQL is a great database, but pgAdmin (or any client I have tried) is just painful compared to SSMS. And while SSMS certainly has some pain points, I have yet to find a DB client tool that even comes close in terms of usability. For this reason alone I prefer MS SQL.
- aeyes 7y agoI love Postgres as much as the next guy but there are three things that really annoy me: Postgres is still laid out for for 9 to 5 workloads, accumulating garbage during the day and running vacuum at night. Autovacuum just doesn't cut it in a 24/7 operation and it is by far what has caused the most problems in production. No query hints and no plan to ever implement it. Making the planner better only takes you so far when everything depends on table statistics which can be easily skewed. pg_hint_plan or using CTEs is not extensive enough. Partial indexes are not well supported in the query planner, thanks to no query hints I can't even force the planner to use them.
- patrec 7y agoDo you have a good Postgres DBA? Autovacuum can indeed ruin your day if you have a 24/7 DB that is delete or update heavy (the default setting are not very aggressive and once it falls behind badly it will never catch up and you'll need to do some emergency manual vacuuming). But I can assure you it's possible to run 24/7 DBs fine without running into autovacuming problems by making sure things are configured right. I've experienced both regular problems due to bad config and their complete disappearance.
- nojvek 7y agoI get the Postgres love but as someone who’s been doing mysql for a long time I found postgres really hard to use. Like even setting it up on Mac was a pain in the ass. How do I create a new db? Why is pgcli not as friendly as mysqlcli? After 2 weeks of head bashing and lots of stack overflow reading I gave up and went back to good ol MySQL. For ease of use for a new comer, mysql seems to have nicer tooling and out of box use. I’ve worked with mysql on sites with millions of daily active users and it hasn’t been a problem. Snappy 10ms queries for most things. I guess one really needs to learn ANALYZE and do the necessary index tuning. I’m just saying, use the tool you’re most comfortable with, has good ecosystem and gets the job done. For some it’s PGSQL, for some it’s MySQL, for others it could be the paid ones. MySQl8 is pretty solid nowadays. There is no universal “best”, just as there is no “best” car. All about trade offs.
- Traubenfuchs 7y ago> setting it up on Mac was a pain in the ass What was your problem? All you need to call is: docker run --rm --name pg-docker -e POSTGRES_PASSWORD=docker -d -p 5432:5432 -v $HOME/docker/volumes/postgres:/var/lib/postgresql/data postgres:latest ...and you are good to go. In depth guide: https://hackernoon.com/dont-install-postgres-docker-pull-postgres-bee20e200198 https://hackernoon.com/dont-install-postgres-docker-pull-pos...
- stickfigure 7y agoPostgres is my go-to RDBMS, but I do have one serious complaint: Connections are too expensive. This is a side-effect of the old-school one-process-per-connection architecture that Postgres uses. MySQL (ick) easily handles thousands of connections on small servers; with Postgres you will need a LOT of RAM to sustain the same, RAM that would be better served as cache. I've found (at least, for my current app) that the number of connections defines my scaling limit. I can't add more appserver instances or their combined connection pools will overflow the limit. And that limit is disappointingly small. It's not so painful that I want to reach for another RDBMS, but I'm bumping into this problem way too early in my scale curve.
- dpc_pw 7y agoIt might be possible write a small proxy in Rust, that would run next to a PG instance, accepted connections using `async` and then forwarded queries using some limited size connection pool.
- folmar 7y ago... if you are very keen on writing a pgpool replacement.
- stickfigure 7y agoThese are active connections running transactions - unfortunately there's no substitute for just having another connection.
- dpc_pw 7y agoIf there is a lot of connections actually running transactions, then I'd expect the db will become overwhelmed due to number of transaction, and per-connection overhead is not going to be a problem. How I understood the problem is thousands of connections, of which most do a query from time to time only. Judging by the other comments, it seems solutions like that are already available.
- 7y ago
- thosakwe 7y agoI use Postgres for everything. The one thing I wish they would do, though, is improve the error messages when a query fails. At the very least, if I know which position in the text is the beginning of an error, I can fix it.
- anarazel 7y ago> The one thing I wish they would do, though, is improve the error messages when a query fails. Yea, we definitely need to do better there. But it's harder than one might think initially :(. My suspicion is that we (the PG developers) will have to write our own parser generator at some point... We don't really want to go to a manually written recursive descent parser, to a large part because the parser generator ensuring that the grammar doesn't have ambiguities is very helpful. > At the very least, if I know which position in the text is the beginning of an error, I can fix it. For many syntax errors and the like we actually do output positions. E.g. postgres[1429782][1]=# SELECT * FROM FROM blarg; ERROR: 42601: syntax error at or near "FROM" LINE 1: SELECT * FROM FROM blarg; ^ (with a mono font the ^ should line up with the second FROM) If you have a more concrete example of unhelpful errors that particularly bug you, it'd be helpful. In some cases it could be an easy change.
- einpoklum 7y agoPostgreSQL is a transaction-oriented row-store. It is slow for analytics, and I mean by something like two orders of magnitude, if not more, relative to state-of-the-art systems like HyperDB or Actian Vector, on typical analytic workloads. You can extend it any which way you like, that won't change. (Ok, if you can extend it with essentially a different DBMS then that could be fast.) Security, concurrency control - those are important for transactional work facing numerous users. Not for analytic processing speed.
- csours 7y agoHow do you manage your PostgreSQL deployments?
- lazyant 7y agoPostreSQL is absolutely fantastic and for most projects, unless there's an existing legacy database or a purely Microsoft environment should be the default RDMS. My only complaints are: 1) No built-in or easy failover mechanism (and will never have one because of their estated philosophy). I'll settle for a standard one and yes I know there are different requirements and things to optimize for and there are 3rd party solutions (just not one to integrate easily). 2) Horizontal Scalability. And yes I know of some solutions and other dbs being more apt for this and asking the wrong thing.
- SonnyBlack6661 7y agoG FK j v hggf
- consultSKI 7y agoFair enough. Without Sybase XI or whatever version they would now be up to, I can see where this could be true. But when MySQL added Views in v5.0.3 as I recall, it met my requirements for an RDBMS.
- ncmncm 7y agoThat performance carries the day today is a huge improvement since 1999. Back then, there was also only one question that mattered, but the question was, "Is it Oracle?", just as the only question about network equipment was "Is it Cisco?". It is hard to explain, at this remove, what horrible perversions were performed in order to be able to say "yes" to such questions, or to avoid need to answer them. One example was a router that, at opposite ends of a long-haul network, managed data flow that was utterly insensitive to the magnitudes of latency or drop rate. They sold hardly any. It turned out the algorithm could be run in user space over UDP, and that became a roaring business, for a different company, because you could deploy without getting network IT involved. Mostly IT didn't even want bribes. They just didn't want anything around that was unfamiliar. Database products had to pretend to be an add-on to Oracle, and invisibly back up to, and restore from, an Oracle instance, because Oracle DBAs only ever wanted to back up or restore an Oracle instance. The Oracle part typically did absolutely nothing else, but Oracle collected their $500k (yes, really) regardless.
- throwawaysea 7y ago> So, you want NoSQL, Riak, REACT, Redis, Mongo, etc.? PostgreSQL does all that. Admittedly not with all the bells and whistles of all of the original products. For example, PostgreSQL doesn’t create new shards for you for any of those. That’s still a manual process. Isn't this a fairly big issue? I would think the convenience of automated scaling is a primary reason to use these other tools.
- luord 7y agoMight be a bit... Forcibly put, but I gotta say that PostgreSQL is one of my favorite pieces of software ever created. And I don't even like C all that much.
- Beefin 7y agoAny love for NoSQL here? Many have all the same capabilities as Postgres/major SQL tech but have a format that removes the need for an ORM.