mslot

8 ms·

31.
by mslot 2y ago
The credentials are currently managed via the platform, so you enter them in the dashboard. We wanted to avoid specifying credentials via a SQL interface, because they can easily leak into logs and such. We'll add more authentication o
32.
by mslot 2y ago
It will do a sample, so usually much less than a table scan, but it can definitely take longer than for Parquet.
33.
by mslot 2y ago
It caches the files on locally attached NVMe drives in the background, and continues to use range requests to S3 for queries until the full download completes.
34.
by mslot 2y ago
It might be kind of nice if Postgres had a "force this statement through" option which kills any conflicting processes. We created something like that in Citus when changing a node's hostname (e.g. during a failover). While n
35.
by mslot 2y ago
Read committed mode (PostgreSQL's default) can get pretty funky. If two transactions concurrently perform a SELECT (may be in a CTE) followed by an UPDATE, then they might see and try to update the same rows. That's often undesira
36.
by mslot 3y ago
hashtext() works well
37.
by mslot 3y ago
dbt comes to mind
38.
by mslot 3y ago
You're responding to said grump ;) Andres is one of the most prolific PostgreSQL committers and his depth of understanding of systems performance is second to none. I wouldn't have guessed he would one day save the world with it,
39.
Pgcopydb: Postgres Dump and Restore on Steroids(github.com)
4 points by mslot 3y ago|1 comments
40.
by mslot 3y ago
An obvious answer would be PL/Python, though is not available on most managed services because they prefer to prevent file system access. An exception is https://docs.crunchybridge.com/extensions-and-languages/pyth
41.
by mslot 3y ago
Fwiw, pg_cron jobs run as the database user that scheduled the job and the jobs table is not writable, nor can you see jobs created by other users unless you are the admin. There is not much you can do with it that you could not via externa
42.
by mslot 3y ago
I always wonder why this pattern is not more popular: https://sive.rs/pg We try to put a clean API on everything, except the one place where it really matters...
43.
by mslot 3y ago
One challenge with running PostgreSQL in production on Kubernetes is that it does not behave nicely under high memory pressure. PostgreSQL likes to get NULL back from malloc such that it can gracefully abort transactions (overcommit_memory
44.
by mslot 3y ago
Note that cstore_fdw ≈ Citus columnar (modernized using table access method) ≈ Azure Cosmos DB for PostgreSQL (managed Citus service) ≈ Hydra (a Citus columnar fork) Fwiw, the main use case we had in mind when developing the Citus columnar
45.
by mslot 3y ago
> What's advantage over having tenant id as distribution column? Ease of use is definitely the main one. If you're willing to put in the work required to use tenant ID as a distribution column (add it to tables, primary keys, f
46.
by mslot 3y ago
The answer is mostly the same as for PostgreSQL, since Citus stores data in regular PostgreSQL tables. It's a good idea to set up archival into blob storage / S3 using a tool like WAL-G for disaster recovery purposes, and streamin
47.
by mslot 3y ago
Historically, there has been, but we have found physical replication is ultimately a lot more performant, robust, and tweakable. Citus users & platforms generally use physical replication of each node for high availability. For instance
48.
by mslot 3y ago
True, though I'd phrase it more as a middle ground. Which model is best depends on your needs. A big advantage of schemas over databases is that you can access different schemas over the same connection, which allows much higher densit
49.
by mslot 3y ago
Nice to see this on HN :) The high-level is: You enable a setting and every CREATE SCHEMA creates a new shard. All the tables in the schema will be co-located so you can have efficient joins & foreign keys between the tables. On top of
50.
by mslot 4y ago
Their lights are nice, but their motion sensor drains the battery in 2 months or so. Eventually I figured out that I could control the IKEA lights via Philips Hue, so I could switch to a Hue sensor which is already lasting a few years. In g
51.
by mslot 4y ago
Most $100B+ companies (e.g. Google, Meta, Amazon) were built primarily using relational databases for the first 10-15 years.
52.
by mslot 4y ago
Here you go! https://www.citusdata.com/blog/2022/06/17/citus-11-goes-full...
53.
by mslot 4y ago
Not right now. It's definitely something that's on our radar, but it'll take a while to get there.
54.
Pg_ivm: Incremental View Maintenance for PostgreSQL(github.com)
3 points by mslot 5y ago|0 comments
55.
by mslot 5y ago
One thing I forgot to mention in the blog post: Since all nodes in a Citus cluster now look the same, you may be missing your shards (Postgres tables that store a subset of the data in a distributed table). The shards are still there, but w
56.
Citus 11.0 beta release adds querying from any node(citusdata.com)
3 points by mslot 5y ago|1 comments
57.
by mslot 5y ago
OLAP stores are relatively fast at answering a single query on a large data set, but basically none of them can handle high throughput with subsecond response times (e.g. when the whole country checks statistics for their own postcode at 4p
58.
by mslot 5y ago
The dashboard has to deal with a complex data integration problem, with different sources with differences in completeness, accuracy, age, and granularity (at many levels), daily corrections in past data, changes in data structure and seman
59.
by mslot 5y ago
Good code explains the problem to the reader and implements the solution as a side effect. You can often express the problem quite well using a combination of if, for, comments, variable names, function names, list/set and map data str
60.
by mslot 5y ago
True, it is developed by Microsoft and available as a service on Azure. It is also open source, actively maintained and improved, and it's a PG13-compatible Postgres extension that adds both distributed database capabilities and column

More ›