anarazel

77 ms·

751.
by anarazel 8y ago
I think it's worthwhile to note that this, even before both kernel and postgres changes, really only is an issue if there are very serious storage issues, commonly causing more corruption than just forgetting the data due to be fsynced
752.
by anarazel 8y ago
> Doesn't this affect all databases? Yes, most. And several did similar changes (crash-restart -> recovery) to handle it too. It's possible to avoid the issue by using direct IO too, but often that's not the default mod
753.
by anarazel 8y ago
In a system like postgres' current heap you cannot know whether it was committed long ago, without actually looking in that side table (or modifying the page the one time you do, to set a hint bit). You pretty fundamentally need someth
754.
by anarazel 8y ago
> Any overhead is generally considered irrelevant when you’re on a small database. For example, going from 50GB in a table to 75GB is 50% overhead, but easily handled in a time where you just pay for more GB on RDS. Often disk-space is
755.
by anarazel 8y ago
You need to access that data from different connections, so it needs to be correctly locked etc. Looking purely at the tuple you need to know where to look for the tuple visibility information. Accessing data stored in some datastructure o
756.
by anarazel 8y ago
Depends on your data. If you have a link table that just contains two 4 byte integers, then yes, that's pretty significant overhead. Even if you have a few hundred bytes per row, it's still not entirely negligible.
757.
by anarazel 8y ago
We can't use transaction ids for that however, because they're necessarily assigned by the time a transaction starts to write, whereas visibility is determined by the time transactions commit. Snapshots, the datastructure that de
758.
by anarazel 8y ago
There's no chance we go for 64bit transaction ids on the tuples themselves - the space increase would be far too big. The overhead of tuple headers is already a problem, and xmin/xmax are a significant portion of that. There were
759.
by anarazel 8y ago
PA-RISCs lacking support for any meaningful support for any atomics except TAS (one with weird alignment reqs) forced me to add an atomics emulation layer to postgres. So I somewhat hate it now ;)
760.
by anarazel 8y ago
Right, and it'd be pretty crazy to trawl through all results just to come up with an accurate count; would be a huge waste of energy (even leaving costs aside). But I do think their predictions have gotten a lot worse in the last few
761.
by anarazel 8y ago
> Amazon has explained in their reinvent videos that Aurora is the storage layer of Postgres rewritten to be tightly coupled to their AWS infrastructure. So it is just regular Postgres (they upgrade to latest on a slightly slower cadenc
762.
by anarazel 8y ago
They've provided AWS credits, yes.
763.
by anarazel 8y ago
Yea, right... I don't know anything about how hard this stuff is, I'm just a postgres developer & committer, currently working on making postgres' storage pluggable.
764.
by anarazel 8y ago
The IAM stuff requires using a few hooks, at most adding minor modifications that could be upstreamed. Storage is obviously harder, but I and others are working on making that pluggable in core. Amazon's core contribution, from a few s
765.
by anarazel 8y ago
Worthwhile to note that Greeplum is being moved forward. While it initially seemed, from the outside, fairly slow-going, it seems that later versions were done more quickly. Apparently largely because of fixing some technical debt. They&#x2
766.
by anarazel 8y ago
I think Citus was the first PG fork to "unfork"... (yes yes, I'm biased, I worked my ass off making that happen)
767.
by anarazel 8y ago
Did you turn fsync off?
768.
by anarazel 8y ago
That really depends on what you're caching. Sometimes e.g. you might want to have local caches on application servers, not because the database would be too slow or couldn't take the load, but because the roundtrip time to the dat
769.
by anarazel 8y ago
That's sometimes unavoidable. If there's a clear bug somewhere, fixing it will sometimes also affect queries/plans that weren't visibly affected by the bug. That's especially the case around costing issues - some qu
770.
by anarazel 8y ago
That explicitly requires superuser permissions. If you give those out, the user has plenty ways to code execution. (And as mentioned elsewhere, that's server-side, not the server instructing the client to send files as in the tweet.)
771.
by anarazel 8y ago
I'm very doubtful that's really true for databases. A lot of memory freed this way is not short lived. E.g. the contexts for running [parts of] an analytics query can both be large and quite long lived. And it's crucial tha
772.
by anarazel 8y ago
Uh, I know of several low-mid level techies where their companies played hardball around their non-competes.
773.
by anarazel 8y ago
I'm not clear as to why that'd be faster than what we/PG has done. On a quick skim that allows for conflicts in the hashtable, which our new implementation doesn't? Ours is also a C array (albeit of offsets into one blob
774.
by anarazel 8y ago
Note it's the "raw" parsing time. If you include the semantic parse-analysis phase, i.e. looking up table names etc, it's a much smaller portion.
775.
by anarazel 8y ago
Due to the desire to have keywords that aren't fully reserved (we have keyword classes that are fully reserved, can appear as type names, can appear as column names, or can appear nearly everywhere) we/PG need to look up whether a
776.
by anarazel 8y ago
We looked at that, and the generated code/hash unfortunately isn't that good.
777.
by anarazel 8y ago
> Making 1 keyword comparison instead of log n keyword comparisons yields the 20% speedup. Worthwhile to note that that speedup is for the raw parsing, without parse-analysis (i.e. looking up the tables referenced and such), planning (fi
778.
by anarazel 8y ago
Correct.
779.
by anarazel 8y ago
ICU collations in PG are versioned, and ICU has support for accessing the different collation versions. So you might be stuck on an older collation version without explicit action, but it'll not yield wrong results.
780.
by anarazel 8y ago
No, that's not correct: Initial json support in PG was added in early 2012 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit... whereas mongodb was made public in 2009, according to wikipedia: https:&#

More ›