Eikon

11 ms·

121.
by Eikon 1y ago
> why backup? Because of mistakes and malicious actors...
122.
by Eikon 2y ago
I am working on Merklemap, a CT search engine: https://www.merklemap.com/
123.
by Eikon 2y ago
> You have 5 days before the new DockerHub limits f*ck you over. How classy from a somewhat DockerHub competitor: https://www.blacksmith.sh/pricing I guess only they are allowed to make money, not Docker.
124.
by Eikon 2y ago
> The official methods for self-hosting Next.js require running the application in a stateful way, as long-running servers. While technically possible, this is very hard to operate in any real-world production environment where a single
125.
by Eikon 2y ago
That's not a server. It is a rackable desktop. You won't run general purpose server apps on this.
126.
by Eikon 2y ago
It’s mdx, with nextjs. Thanks for the kind words!
127.
by Eikon 2y ago
I tried for fun: https://calculator.aws/#/estimate?id=cfc9b9e8207961f777766e1... Seems like it would be 160k USD a month. I could not input my actual IO stats there, I was getting: Baseline IO rate can't be more t
128.
by Eikon 2y ago
No, because the custom block size is about reducing zfs r/w amplification.
129.
by Eikon 2y ago
> it's another thing entirely to write into many tables, index all of them and then query them at scale Well, that’s pretty much what I am doing.
130.
by Eikon 2y ago
There’s backlog I still need to process :)
131.
by Eikon 2y ago
> "Postgres does not scale" means that you can't just give it more machines, which is true. Well, it’s only true for writes.
132.
by Eikon 2y ago
It’s too slow at that scale, pg_squeeze works wonders though. I only “need” that because one of my table requires batch deletion, and I want to reclaim the space. I need to refactor that part. Otherwise nothing like that would be required.
133.
by Eikon 2y ago
I don’t think Postgres will be able to benefit from direct io? I might be wrong though! I use Postgres with 32K BLKSZ. I am actually using default 128K zfs recordsize, in a mixed workload, I found overall performance nicer than matching at
134.
by Eikon 2y ago
Well, that's why I said "ATM", it's not a sustained rate, all the time. And yes, there's a bunch of DELETEs too.
135.
by Eikon 2y ago
Here's mine: - autovacuum_max_workers to my number of tables (Only do so if you have enough IO capacity and CPU...). - autovacuum_naptime 10s - autovacuum_vacuum_cost_delay 1ms - autovacuum_vacuum_cost_limit 2000 You probably should re
136.
by Eikon 2y ago
I am not optimizing too much around insertion speed. I avoid GIN, GIST and hash indexes. The schema is nicely normalized. I had troubles with hash indexes requiring hundreds of gigabytes of memory to rebuild. Postgres B-Trees are painless a
137.
by Eikon 2y ago
I’d say 60-70% overall cpu usage, including database, ingest workers, web app and search. > Your replies are really valuable and informative. Thank you so much. Thank you!
138.
by Eikon 2y ago
Ahaha :)
139.
by Eikon 2y ago
Mostly, be careful with long-running transactions (hours long), and modify your autovacuum settings to be as aggressive as possible based on your workload. Be careful with the freezing threshold too. I ran into multixact "members
140.
by Eikon 2y ago
Debian!
141.
by Eikon 2y ago
No, one time + ongoing colocation costs.
142.
by Eikon 2y ago
> So run a replica? Basically, yes. https://www.postgresql.org/docs/current/runtime-config-repli...
143.
by Eikon 2y ago
> Since you’re self hosted, I’m you aren’t on AWS. How much is this setup costing you now if you don’t mind sharing. About 28K euros of hardware per replica IIRC + colo costs.
144.
by Eikon 2y ago
> were restoring to a new node Zfs send / recv or replication. > I used to run a bunch of Postgres nodes at a similar scale. The most painful parts (by far) were restoring to a new node and major version upgrades. Any tricks you
145.
by Eikon 2y ago
I'll try to get a blog post out soon! > Damn, that’s a chonky database. Have you written anything about the setup? I’d love to know more— is it running on a single machine? How many reader and writer DBs? What does the replication l
146.
by Eikon 2y ago
I run a 100 billion+ rows Postgres database [0], that is around 16TB, it's pretty painless! There are a few tricks that make it run well (PostgreSQL compiled with a non-standard block size, ZFS, careful VACUUM planning). But nothing to
147.
by Eikon 2y ago
> Take that by 6h per day x 365 You'll need to provide credible sources to support that claim. That assertion seems highly unrealistic.
148.
by Eikon 2y ago
I am indeed running PostgreSQL on bare metal with a 20TB NVMe pool, the DB itself is around 18 TB :) But that's the way to go, isn't it?
149.
by Eikon 2y ago
Same for https://www.merklemap.com/ The biggest table contains 30B records. A query that uses a B-tree index completes in a few microseconds. EXPLAIN ANALYZE SELECT * FROM table_name WHERE id = [ID_VALUE]; Index
150.
by Eikon 2y ago
I made this, thank you!

More ›