2 ms·
1. Yes. Our first attempt at storage implementation had a problem with many small file. Then the team rearchitected it around LSM trees and it got a LOT better.
by nikita 4y ago
1. Yes. Our first attempt at storage implementation had a problem with many small file. Then the team rearchitected it around LSM trees and it got a LOT better. Our benchmarks show that we are very close in performance with vanilla Postgres and Aurora. There are some "worst case" scenarios where Neon is worse than vanilla Postgres. Aurora has similar problems too.
2. It's best to custom build a storage system here. External distributed filesystems introduce complexity, cost, and bottlenecks that you don't control.
3. Purpose built. LSM trees also have a temporal dimension - LSN. You can fetch a page by pageId and LSN. This is what allows time machine and branching.
4. I call it convergence when OLTP and OLAP is one system - ultimate dream for a database systems engineer. Since I spent 10 years building it I have both scars and aspirations. I think it will come, but this will take a long time. HTAP is in a way a subset of convergence - most systems will have some HTAP. Neon will have some too, but for now it squarely focused on OLTP and helping developers build apps.