jhk727

4 ms·

1.
by jhk727 5y ago
It was a nontrivial improvement to write throughput at the time. I'd imagine it could have similar impact for other write-intensive workloads.
2.
by jhk727 5y ago
Thank you for the clarification - I had heard from a few sources that the block allocator algorithm actually changes at higher utilization, but was previously unable to find anything concrete in the documentation. This helped clear up a lon
3.
by jhk727 5y ago
We use wal-g and extensively leverage its archive/point-in-time restore capabilities. I think it would be tricky to manage similar functionality with snapshots (and possibly more expensive if archival involved syncing to a remote pool)
4.
by jhk727 5y ago
We tested this extensively a few years back. We saw a compression ratio of ~1.9 with 8k recordsize/lz4, ~2.7 with 16k/lz4, and now ~5.5 with 64k/zstd.
5.
by jhk727 5y ago
Author here - it's difficult to provide a single number to summarize what we've observed re: CPU, but one data point is that average CPU utilization across our cluster increased from ~40% to ~50%. This effect is more pronounced du
6.
by jhk727 5y ago
Good callout - we use a higher blocksize than Postgres page size because it gives us a much higher compression ratio, at the cost of some read/write amplification. And yes - Postgres will automatically TOAST oversized tuples and compre
7.
by jhk727 5y ago
We aren't using cstore_fdw, though we've looked into it in the past. cstore tables don't support deletes or updates, and we still rely on updates for some key parts of our write pipeline. Additionally, we rely heavily on btre
8.
by jhk727 5y ago
Postgres is not designed for OLAP, but you can push it a lot farther than one would expect with the correct schema and indexing strategy. See https://heap.io/blog/running-10-million-postgresql-indexes-i... for a little
9.
by jhk727 5y ago
Author here - It's explained in the post but the primary driver is cost. 80%+ reduction in storage is massive when you're storing petabytes of data on ssds.
10.
by jhk727 5y ago
We are, though we started out using EBS. As you mentioned, NVMe instance storage performs much better for our workload. We work around the lack of durability through strong automation of point in time restore/swapping in of new nodes i
11.
by jhk727 5y ago
Author here - as others have noted, there's a lot of benefits to a company of our size operating infrastructure on AWS vs. managing physical hardware. A couple of the highlights for managing our primary database cluster include: - Auto
12.
by jhk727 5y ago
Author here, happy to answer any questions.
13.
by jhk727 8y ago
Hey all, author here. This project was a fun demonstration of the versatility of Redis. Happy to answer any questions.
14.
Building a Live Data Visualization in 4 Days Using Redis Pub/Sub(heapanalytics.com)
12 points by jhk727 8y ago|4 comments
15.
by jhk727 10y ago
We offer our customers the option of managing their own cluster or having us manage it for them. About 80% of our customers choose the first option. In this case, cluster sizing and queue configuration is up to them. When we manage the clus
16.
by jhk727 10y ago
Hi, author here. This post covers some of the surprising things we ran into getting started with Redshift. Happy to answer any questions.