malisper

33 ms·

331.
by malisper 9y ago
If you have queries that are taking days, you'll probably be better of trying EXPLAIN ANALYZE[0] first. If you can't get enough information from EXPLAIN ANALYZE, then you may want to try using flame graphs. [0] https://
332.
by malisper 9y ago
Hi, author here. Feel free to ask any questions about the post.
333.
How Basic Performance Analysis Saved Us Millions(heap.engineering)
5 points by malisper 9y ago|1 comments
334.
by malisper 10y ago
Having millions of shards is a bit of a pain ops wise. There is also some other costs to having lots of shards. For example, the current Citus distribution planner takes time linear in the number of total shards. This isn't an issue 99
335.
by malisper 10y ago
We shard our data by customer and individual queries touch only data for one customer. We do split a customer's data into multiple shards, but the number of shards we would need to split each customer into is not worth the overhead of
336.
by malisper 10y ago
Logical replication doesn't matter as much to us. We are using CitusDB (in other words sharded Postgres), and we have our own code for handling replication.
337.
by malisper 10y ago
Besides parallel queries (specifically parallel index scans), the Postgres 10 feature I'm most looking forward to is the work Andres Freund has done on using LLVM to JIT compile expressions[0]. Based on the the layout of the database I
338.
by malisper 10y ago
I see. I thought system bytes included the mmaped block of memory (the docs for malloc_stats explicitly say it does not) and I thought the last graph was a demonstration of why the Python memory allocator was necessary.
339.
by malisper 10y ago
I don't fully understand the post, but I don't think the author found a bug anywhere. The main discovery the author made was that CPython uses its own allocator on top of memory allocated by mmap. This leads to a difference betwee
340.
by malisper 10y ago
> The problem with their algorithms.... is that all that statistics in the world can't help you when you're listening to a guy telling the truth vs an equally good liar. While that's true for sites in isolation, Google pub
341.
by malisper 10y ago
> On a related note, I have in general concerns about latency spikes due to rehashing. Is there a good, general, incremental rehashing scheme? One option is to move a few elements over from the old hash table on each operation[0]. A seco
342.
by malisper 10y ago
> I should have told her that I didn't want to be interviewed by some programmers, because I would most certainly fail. There was no need to try. I wanted to be interviewed by the person who really needs me: my future boss. That per
343.
by malisper 10y ago
It's a bit difficult to tell what exactly is going on. For future reference EXPLAIN ANALYZE provides a lot more information into the execution of the query. It tells you how how much time was spent in each part of the plan as well as h
344.
by malisper 10y ago
> I don't know any way to disable nestloop query planning only for a specific query You can do that by using SET LOCAL. Here's what your query would become: BEGIN; SET LOCAL enable_nestloop = off; <QUERY> COMMIT
345.
by malisper 10y ago
> we still needed that extra 15% or so of "ambient" performance. This doesn't make sense to me. That 15% of time is spent somewhere. It certainly isn't just "ambient".
346.
Recursive Functions of Symbolic Expressions and Their Computation by Machine(www-formal.stanford.edu)
4 points by malisper 10y ago|0 comments
347.
by malisper 10y ago
When we first switched to ZFS, we had some ingestion issues. It turned out the problems were caused by poor architecture decisions we made and was merely exaggerated by ZFS. So yes, there was a noticeable increase in latency, but ZFS was on
348.
by malisper 10y ago
For a less ad hoc approach to reducing storage costs, I suggest looking into the ZFS filesystem. Compression is completely transparent in ZFS. Once you enable compression in ZFS, all of your files will automatically be compressed when writt
349.
by malisper 10y ago
It's not a single machine. We have a cluster of machines, each of which have several TB of data. The only parameter I clearly remember changing is recordsize=8k, since postgres works with 8k pages.
350.
by malisper 10y ago
> That said, it's inadvisable to run a database on a btree file system like ZFS or btrfs if you're keeping an eye on the write performance. Our writes are actually heavily CPU bound because of how we architected the system[0].
351.
by malisper 10y ago
I'm currently managing a Postgres cluster with a petabyte of data running on ZFS on Linux on AWS. Most of the issues we've come across are around us not knowing ZFS. The first main issue was the arc_shrink_shift default being poor
352.
by malisper 10y ago
While I'm not a huge fan of functional programming, this can be done pretty easily with a fold. For example in Scala: xs.foldLeft(""){ (prev, cur) => if (prev != "a" && cur != "a") parseLine
353.
by malisper 10y ago
Elaborating on gtrubetskoy comment, ziplists are doubly linked lists (of integers and strings) encoded into an array of bytes that stores the size of both the next and previous element so you can quickly find the location of the next/p
354.
by malisper 10y ago
It is a PID loop, which I guess may not be considered to be actual reinforcement learning.
355.
by malisper 10y ago
One of my coworkers used basic reinforcement learning to automate a task someone used to have to do manually. We have two data ingestion pipelines. One that we ingest immediately, and a second for our larger customers which is throttled dur
356.
by malisper 10y ago
If you reword the title to "More than Half of 30-Year-Olds Earn More Than Their Parents" it sounds completely different.
357.
by malisper 10y ago
I have some experience with this since I went straight from high school to working in San Francisco. I don't know about the best city for someone without degree, but the best city for tech by far is San Francisco. To give you an idea o
358.
by malisper 10y ago
Hi, author here. 1) We use software called Citus[0] which sits on top of Postgres and handles all of the read time sharding logic for us. Citus could handle the write time logic, except we wrote our own logic because we are able to perform
359.
by malisper 10y ago
I'm maintaining a database cluster with 10s of millions of partial indexes, many of which contain a LIKE pattern. Each table in the cluster has several of these partial indexes. Because of all of these partial indexes, we became CPU bo
360.
by malisper 10y ago
Author here. Feel free to ask me about any questions you have about our indexing strategy.

More ›