sakras

10 ms·

61.
by sakras 2y ago
The trouble is that AI assistants have seen all of the contrived algorithmic problems before. I once interviewed a candidate for whom Copilot spat the answer out as soon as he typed the function signature. Whether these problems are a good
62.
by sakras 2y ago
This actually addresses a huge hole right now in the ecosystem. At the moment, Arrow treats things that are logically the same as different types, for example a dictionary-encoded utf8 is different from a utf8. Really there needs to be a di
63.
by sakras 2y ago
Unfortunately this is standard for TUM's database group. Their previous database, HyPer was similarly cutting-edge, but was closed source under a proprietary license. Last I heard it got sold to Tableau.
64.
by sakras 2y ago
Last I heard, the Sokoban game has a ridiculous number of puzzles on it. Can't find a source but I seem to recall hearing that it would take 400+ hours to finish it all. So.. I don't think it's entirely unreasonable it's
65.
by sakras 2y ago
Yes! We support a lot of extensions, including PostGIS. Full list here: https://neon.tech/docs/extensions/pg-extensions
66.
by sakras 2y ago
Neon engineer here. We implemented our own local cache between the pageserver and Postgres (called LFC). It compensates for the lack of a normal file system cache, and scales up with the rest of the autoscaling infrastructure.
67.
by sakras 2y ago
Sure! Each instance of Postgres runs inside a qemu VM, inside a kubernetes pod. The VM provides isolation, autoscaling, metrics, and (eventually) live migrations. These VMs share AWS Metal nodes. Source code here: https://github.
68.
by sakras 2y ago
(Neon engineer) It is a coincidence :) I had to confirm internally too.
69.
by sakras 2y ago
This is achieved through our Pageserver. The pageserver ingests streams of WAL, and so contains both snapshots and deltas. This lets us efficiently seek to any LSN by replaying page deltas on top of the nearest snapshot (we call it the GetP
70.
by sakras 2y ago
I'm an engineer at Neon, happy to answer any questions
71.
by sakras 3y ago
I've been exactly thinking about it this way for a long time. Actually once we're able to push computation down into our disk drives, I wouldn't be surprised if these "Disk Shaders" will be written in eBPF.
72.
by sakras 3y ago
A famous Super Smash Bros player, when asked for advice, would always respond with "Don't get hit". Seems this principle of not blundering has wide-reaching applications!
73.
by sakras 3y ago
Right, so I think the idea is to "unbundle" this so that you can compose your own data analytics engine.
74.
by sakras 3y ago
My general take is that while the idea of composability is good, the implementations of these things are just frankly not of high quality. Velox/Acero in particular are all plagued by what I've come to call "Java syndrome&quo
75.
by sakras 3y ago
Yes this has been an up-and-coming theme in the data science world. Arrow for the data format, Ibis for the API, Acero/Velox/DataFusion/DuckDB/Polars for execution, Substrait for the query plan representation, etc.
76.
by sakras 3y ago
The difference is that if a European social media company wanted to operate in the US, it's welcome to. With China it's a different story - they can operate in our market but we can't operate in theirs.
77.
by sakras 3y ago
I'm almost certain this will be slower than OpenMP because it uses a centralized task queue that gets locked. OpenMP uses a decentralized work-stealing task queue called the Chase-Lev Deque. There's a C implementation in this pape
78.
Ultrapacking: A Layer over Bit-Packing(save-buffer.github.io)
3 points by sakras 3y ago|0 comments
79.
by sakras 3y ago
+1 for Denis! The book does a great job explaining both “what does it mean for a program to be optimal?” And “what do I type into my terminal to check the performance?” Of course, it doesn’t cover every possible performance trick. For that,
80.
by sakras 3y ago
It sounds like your distinction of tabulation vs memoization is similar to the concept of bottom-up vs top-down DP. Bottom-up: build the table up eagerly, top-down: recurse down and fill in the table lazily.
81.
by sakras 3y ago
It's used a lot in databases for query processing. Specifically, if you're performing a hash join, probing the hash table is more expensive than probing the Bloom filter. If you are pretty sure you're not going to find very m
82.
by sakras 3y ago
There's a lot of optimizations you can do on top of the classic Bloom filter. In short, you can use a single hash to compute an offset into a table of bit patterns. SIMD lets you perform multiple lookups in parallel. I wrote a blog pos
83.
by sakras 3y ago
Yep, I've heard that actually something like 80% of the energy of the CPU is used by the front-end! Beyond decoding, you also have all the energy spent on scoreboarding data dependencies and reordering the instructions and scheduling t
84.
by sakras 3y ago
Ah yes, the legendary Guh moment. To be fair, this was only 2x leverage, not 25x like ControlTheNarrative.
85.
by sakras 3y ago
I was hoping for a more thorough survey of the compilers and their history. Notably missing is any mention of Intel’s compiler, the Cray compiler, or the competition between Borland and Watcom C compilers. There was a big span of time when
86.
Benchmarking Crimes(gernot-heiser.org)
1 points by sakras 3y ago|0 comments
87.
by sakras 3y ago
I also notice you use FMA in your AVX2 L2 distance calculation. I don't think pg_vector enables that, so SimSIMD might be slightly faster. Also interesting that it beats NumPy/SciPy by so much! I wonder what they're doing..
88.
by sakras 3y ago
As far as I'm aware pg_vector just uses the compiler's autovectorization on float32. I think specifically for Euclidean distance you won't really beat GCC (the README even admits it: "GCC handles single-precision float b
89.
Byte Positions Are Better Than Line Numbers(computerenhance.com)
8 points by sakras 3y ago|4 comments
90.
Dunkey's Guide to Streaming Services [video](youtube.com)
7 points by sakras 3y ago|0 comments

More ›