16 ms·

Databases in 2022: A Year in Review

365 points by gavinray 4y ago

XinChong 4y ago
[dead]
nikita 4y ago
Neon CEO here. There are a number of ideas in the database space that the industry is adopting across the board: - Separation of storage and compute (Neon, AlloyDB, Aurora). Every cloud database should built one. It's a big undertaking, but benefits are undeniable. - Good query processor for analytics (Snowflake, Velox, Singlestore) - Open source. Especially in OLTP open source == trust - HTAP. Can run mixed workload (Singlestore, Unistore): both OLTP (apps) and OLAP (reporting). This has always been a dream, but we still live in the world of dedicated systems: E.g. Snowflake and Postgres. - Shared nothing sharding (Vitess). This is the most controversial as you lose compatibility with the mothership (MySQL for Vitess). So it's unclear this will be the dominant architecture in the future. I think the world may get to "dynamic sharding" where storage stays separate and compute can be multinode and the user can easily and instantly change the number of nodes.
- swyx 4y ago
  since you were also a founder of Singlestore, just wondering - what will be Neon's take on HTAP workloads? trying to see how much you mean it that the industry is "adopting (HTAP) across the board"
  - nikita 4y ago
    I wouldn't say HTAP is an across the board trend. The need is mostly in the enterprise where we see high value mixed workloads. Neon will take a Postgres centric POV. I consider a few approaches, but not committing to any particular at this point. The reason to not commit is that there is a lot more opportunity for Neon in serverless, devtools, and edge. And it makes a ton more sense to partner for OLAP workloads. So what is Postgres centric POV: - Plugins like Timescale and Citus - Seamless integration with sending CDC to Snowflake, Singlestore, Databricks, Clickhouse, and Motherduck - FDW for integration with Snowflake, Singlestore, Clickhouse, Databricks, and Motherduck - We looked at putting DuckDB on Neon storage with Jordan Tigani, but it was too big of a lift as DuckDB doesn't push all data through the WAL. Maybe in the future.
    - zX41ZdbW 4y ago
      Let's commit this to ClickBench: https://github.com/neondatabase/neon/pull/2855 https://github.com/neondatabase/neon/pull/2855 So, NeonDB can be compared with other serverless Postgres offerings.
      - nikita 4y ago
        The reason we run OLAP bench is because when you separate storage full table get impacted: you fetch pages from remote storage one by one and the mitigation is prefetch. So our goal is to be on par with vanilla Postgres. We don't care about winning OLAP benchmarks.
        zX41ZdbW 4y ago
        ClickBench is still ok to compare non-OLAP databases on OLAP workloads - filter away the column-oriented DBMS, and you will be left with the comparison of non-OLAP databases. I think it makes sense to add NeonDB for the sake of completeness.
        nikita 4y ago
        I think lots and lots of benchmarks are useful internally. They highlight where the soft spots are. This is however very different from publishing benchmark results. As a dbaas vendor we should publish benchmarks that your user base really cares about which demonstrate that the system is mature and can handle the core workload your users expect you to handle. That is why publishing an OLAP benchmark results for an OLTP system makes very very little sense. (BTW OLAP systems like Clickhouse should really publish TPC-DS SF1000+ results) For OLTP systems there is TPC-C but it's actually not that telling as it is so simple. I can start seeing that people in OLTP care more and more about latencies, hence all the edge related work and optimization of the drivers. So Neon will be publishing end-to-end latencies benchmarks and also make sure it is on par with vanilla Postgres on a broad set of benchmarks. Those we will run internally and the results will simply be commits on our github.
    - swyx 4y ago
- gavinray 4y ago
  Really enjoyed the talk from your engineer Heikki at Andy's CMU "Databases!" seminar series earlier this year, great stuff =) https://www.youtube.com/watch?v=rES0yzeERns https://www.youtube.com/watch?v=rES0yzeERns Also, it's been neat to have mostly worked our replacing our old Heroku integration at Hasura with you folks.
- emptysea 4y ago
  I’m not well versed in the tech but is Spanner the sort of dynamic sharding approach your talking about?
  - gavinray 4y ago
    Scylla also uses a shared-nothing sharding approach, they have an open-source "asynchronous I/O engine" toolkit that is used inside of ScyllaDB called Seastar They have some good reading on this approach in the Seastar docs if you're curious, it's a great general introduction to Shared Nothing: https://seastar.io/shared-nothing/#:~:text=The%20Seastar%20Model%3A%20Shared%2Dnothing,not%20shared%20memory%20between%20threads https://seastar.io/shared-nothing/#:~:text=The%20Seastar%20M.... https://www.linuxfoundation.org/webinars/under-the-hood-of-a-shard-per-core-database-architecture https://www.linuxfoundation.org/webinars/under-the-hood-of-a... This is usually generalized to some variation of "Physical-OS-thread-per-CPU-core" architecture, where each CPU core gets it's own dedicated chunk of data. You use hashing or some other mechanism to route requests for data to the right core. Each CPU core can have user-land concurrency or "virtual threads" (fibers/coroutines/etc) as well, so you're not limited to sequential code necessarily. A combination of thread-per-core x userland-coroutines-per-thread is how modern query processing approaches like "Morsel-Driven Parallelism" work. CPU-1-OS-THREAD-1 COROUTINE_1 COROUTINE_2 CPU-2-OS-THREAD-2 COROUTINE_35 ... COROUTINE_102 https://db.in.tum.de/~leis/papers/morsels.pdf https://db.in.tum.de/~leis/papers/morsels.pdf
    - AtlasBarfed 4y ago
      Scylla is a C++ rewrite of cassandra, the threading has been better as well with thread-per-core. Cassandra 4.0 has some thread-per-core rearchitecture so it remains to be seen if Scylla has performance advantages it used to tout. I have not checked if Scylla ever caught up to Cassandra 3.0's feature set, last I checked it had not. It may be "going its own way" on features as well. I believe (they can correct me). My personal recommendation would be to start with Cassandra, and when operations allow it, move to scylla if performance testing shows it is better. There is also exotic stuff like Rocksandra. Cassandra and Scylla are fundamentally similar in operations (add/remove nodes, scaling, etc). The scylla folks will contend the operations are easier since they get more perf per node so you need less nodes. It's a good idea with cassandra / scylla to bootstrap with a performance testing, data loading, provisioning loop. It gets you used to operations and the tooling for backups, loads, and load testing and metrics.
- jandrewrogers 4y ago
  The separation of storage and compute is controversial, though it is definitely convenient for cloud operators. It is literally going backward in terms of the available storage bandwidth, and many modern database architectures can very effectively use all the storage bandwidth available in modern systems. There is not enough network bandwidth, and won’t be for the foreseeable future, for running storage over networks to be efficient. Database workloads are also starting to move to edge environments, which make it even worse. The resource mismatch currently leaves a lot of compute sitting idle in practice. Another issue with this model is that it tends to offer poor ratios of bandwidth to storage size. This is at odds with the desire to run mixed workloads in a single system.
  - gavinray 4y ago
    > The separation of storage and compute is controversial I don't think it's a controversial topic, it's an emerging trend. See for example this slide from the recent RocksDB meetup on their vision for the next 3 years of RocksDB's future: https://twitter.com/GavinRayDev/status/1600666127025156096 https://twitter.com/GavinRayDev/status/1600666127025156096 Disaggregated storage is becoming quite a hot topic, and there are several technologies fueling this. There are two major advances happening in terms of disk I/O right now (that aren't to do with async IO uring stuff) 1. NVMe Zones + NVMe FDP specs, which are ways for userland applications to control placement of writes on drive (like a software Flash-Translation-Layer) 2. NVMe-over-Fabrics or similar technologies (with rdma/tcp) enables the access of a remote disk with 'comparable' latency to that of a local one. E.g. if local disk access is 100us, AWS can give you 110us with NVMe-OF. These two advances support the growing trend of disaggregated storage --- Full credit to Arun George from the Samsung Memory group for explaining this to me.
    - jandrewrogers 4y ago
      The storage-over-fabric latency improvements allow it to work well for software that was not particularly optimized for storage performance, which has obvious benefits that I am not arguing. There is a good reason storage-over-fabric has been a popular database deployment model for decades, it does have advantages, but this emerging trend is nothing new nor are the issues that it has. For systems that are highly optimized for storage performance, storage bandwidth tends to be more important than storage latency and that is what is being sacrificed for the sake of disaggregating storage. I see integer factor reductions in workload throughput when deployed on disaggregated storage versus the usually NVMe JBOD, and that factor tracks real-world storage bandwidth pretty well, particularly at high storage densities. At scale, these economics start to matter quite a lot. To be clear, disaggregated storage has no effect on database design generally, seeing as how it has been widely used for decades. It does put an artificial floor on achievable throughput for many workloads due to the bandwidth bottleneck. When people realize this then the pendulum swings the other way (again), so I wouldn't want to overfit for disaggregated storage.
- zX41ZdbW 4y ago
  The most developed analytical query engine as of today is ClickHouse [1]. It is open-source, has the separation of storage and compute and the best performance. ClickHouse is a source of inspiration for many emerging database engines: Velox, Doris, DuckDB, Datafusion... [1] https://github.com/ClickHouse/ClickHouse https://github.com/ClickHouse/ClickHouse Disclaimer: I work on ClickHouse.
  - ddorian43 4y ago
    > has the separation of storage and compute and the best performance. This part is not yet production ready though. And it's unclear when it will be. Though no other open source OLAP does it either AFAIK.
    - _dark_matter_ 4y ago
      SparkSQL + delta lake definitely counts as an open source OLAP separated compute/storage.
- Ataraxy 4y ago
  I actually look forward to Neon but won't actually use it for real until pricing becomes available.
  - nikita 4y ago
    Stay tuned. It's coming very very soon.
- fulafel 4y ago
  Regarding separation of storage and compute, do those DBs you mention enable merging the application backend code and db compute node to a single process? Meaning I would deploy my application to the DB compute node and use its runtime, or have the compute node present as a component I can import to my backend code as a library. I think this is one powerful thing about Datomic (and maybe XTDB?).
deleted 4y ago
[deleted]
yingjunwu 4y ago
I'm the founder of RisingWave Labs. Glad to know that our company was mentioned in the blog. Technology wise, agreed with Nikita that storage-compute-decoupling is the trend. Severless will also be the main focus. Market wise, I observed several trends: * streaming. streamlining the data pipeline is a new trend. RisingWave and Materialize are both addresssing this market. They will probably need to figure out how to compete against established solutions like KsqlDB and Apache Flink. * real-time. There are probably 20+ vendors selling real-time OLAP databases. They all claim that they are cheaper, faster, better than Snowflake/Redshift. Just name a few: Druid, Pinot, ClickHouse, StarRocks etc. Let's see who will be the winner in 2023. * small data. The biggest winner in this domain is Motherduck, a startup commercializing DuckDB. Redshift/Snowflake are a overkill for most companies who do not own large amount of data. Motherduck may be the right choice. * hybrid. Several companies have started selling hybrid databases (mainly HTAP for now): Google AlloyDB, Snowflake Unistore, SingleStore, PingCAP TiDB+TiFlash, etc. In 2023, we will also watch how Databricks challenges Snowflake in the data warehouse domain. Best luck!
- danielvaughn 4y ago
  Druid is indeed a bit faster than Snowflake, but a previous team I worked on was able to achieve similar latencies under heavy load using some clever techniques with materialized views.
- glogla 4y ago
  > * real-time. There are probably 20+ vendors selling real-time OLAP databases. They all claim that they are cheaper, faster, better than Snowflake/Redshift. Just name a few: Druid, Pinot, ClickHouse, StarRocks etc. Let's see who will be the winner in 2023. These are faster but they are also very limited. Most of these have very limited support for joins, can't do proper SQL (e.g. no CTEs in ClickHouse) and require a lot of careful design for your data (e.g. StarRocks can do smart pre-aggregation but you have to tell it which columns are dimensions and which are values). That is fine for limited use cases - like "dedicated engineering team is building specific application" - but doesn't really work for general purpose data platforms. Snowflake I think is successful because it is pretty good at handling everything you throw at it, even if it costs a lot of money.
friedman23 4y ago
Happy to find this blog, wish there was more talk about NewSQL databases. I've personally seen the amount of cost that goes into migrating off of traditional RDBMs systems to sharded RDBMs systems and was wondering why people don't go with these scalable systems from the start now that they exist.
- likeabbas 4y ago
  +1. Recently, Yugabyte has seemed intriguing to me. (It seems like) You can launch it as a standalone instance and use it just like Postgres early in your company, and then when you need to distribute, you can do that.
  - brnewd 4y ago
    I think many love the promise of a distributed database, but in practice there is a real speed and feature set compromise to make.
    - likeabbas 4y ago
      I found that having a NoSQL distributed database for prod that stream Change Data Captures to a read-only SQL DB to be a great compromise. The SQL data was slightly stale but that wasn't an issue for most situations (like graphing data or certain error metrics)
- pm90 4y ago
  Probably because there is a lot more operational understanding of “traditional” RDBMs and they’re sufficient in early stages. Once your product takes off it can be difficult to scale but by that time you probably have enough money to attract engineers that can do the migration.
  - friedman23 4y ago
    Yes, that's the common refrain, just migrate when you hit scale. Just pay people to do the migration. My experience at multiple, billion dollar companies has been that it's never that easy. People become afraid of doing schema migrations and tools like dynamodb become the default datastore choice for all new feature development. Every single startup I've worked at, technical cruft that existed from company founding never disappears despite the founders intent that it was never supposed to stay that way and it slows down feature development to a crawl. If NewSQL databases exist that support 90% of the features of postgres but give support for cross shard transactions then I'd rather use them than technology that has been proven to not scale. Unfortunately I'm not an expert on distributed databases and there is not much written about these databases in practice outside of the startups creating the databases themselves.
lioeters 4y ago
Seems like Supabase deserves a mention, with their contributions to the Postgres ecosystem. https://supabase.com/ https://supabase.com/ pg_graphql - https://supabase.com/blog/pg-graphql-v1 https://supabase.com/blog/pg-graphql-v1 pg_jsonschema - https://supabase.com/blog/pg-jsonschema-a-postgres-extension-for-json-validation https://supabase.com/blog/pg-jsonschema-a-postgres-extension... Postgres WASM - https://supabase.com/blog/postgres-wasm https://supabase.com/blog/postgres-wasm ..And other open-source features.
- apavlo 4y ago
  We want to write a paper on the vast extensions ecosystem for Postgres at some point. It's on my TODO list for 2023. Having so many plug-ins/add-ons available is unique to Postgres and due to Stonebraker's original vision of including extensibility as a first-class design principle in the DBMS.
  - dang 4y ago
    I didn't know that and it would make for an interesting historical article in its own right.
  - whitepoplar 4y ago
    In doing so, it would be great if you could provide details around extension safety. I've heard that some extensions may not play nicely in combination with others, and it would be really nice to know how one is to vet extensions for use in production.
- danielvaughn 4y ago
  Also Planetscale edit: I'm okay with being downvoted, but I'm honestly asking why, not complaining. Is Planetscale not an interesting database?
  - mritchie712 4y ago
    fwiw, I don't get the downvotes either
marsupialtail_2 4y ago
I am writing a Python based SQL query engine: https://github.com/marsupialtail/quokka https://github.com/marsupialtail/quokka. My personal goal is to get Andy Pavlo to mention it in his year-end blogs. I agree with many of the points made in the blog by Andy. Writing a distributed database has become way easier due to open source components like Ray, Arrow, Velox, DuckDB, SQLGlot etc. I personally believe we will see a switch from JVM based technologies to Rust/C based with Python wrapper
- apavlo 4y ago
  I saw your engine a week or so ago. Good stuff. I wanted to add it to dbdb.io but I couldn't find an SVG version of this: https://github.com/marsupialtail/quokka/blob/master/docs/docs/quokka2.png https://github.com/marsupialtail/quokka/blob/master/docs/doc... Can you email it to me?
  - marsupialtail_2 4y ago
    Just did.
- password4321 4y ago
  discussed 3 days ago: https://news.ycombinator.com/item?id=34189422 https://news.ycombinator.com/item?id=34189422
zX41ZdbW 4y ago
> he claimed that because MonetDB focuses on read-only OLAP workloads, then MMAP was good enough ClickHouse has switchable IO engines: - read; - pread; - mmap; - pread_threadpool; The best performance is achieved with the combined method, named 'pread_threadpool'. It uses the `preadv2` syscall to check if the file is in the page cache. More details here: https://github.com/ClickHouse/ClickHouse/pull/26791 https://github.com/ClickHouse/ClickHouse/pull/26791 There was also big research of performance on HDD, SSD, and Optane, comparing various methods: https://clickhouse.com/blog/a-journey-to-io_uring-aio-and-modern-storage-devices https://clickhouse.com/blog/a-journey-to-io_uring-aio-and-mo... And there is a pull request adding io_uring: https://github.com/ClickHouse/ClickHouse/pull/38456 https://github.com/ClickHouse/ClickHouse/pull/38456 - Unfortunately, it's unfinished and cannot be merged because the CI has found bugs. Nevertheless, the advantages of io_uring for analytical databases are negligible. The `mmap` method is still useful. For example, for data import. See here: https://github.com/ClickHouse/ClickHouse/pull/43927 https://github.com/ClickHouse/ClickHouse/pull/43927
- jlokier 4y ago
  > The best performance is achieved with the combined method, named 'pread_threadpool'. I found the same when testing my own database engine. `pread` in a suitably well-designed threadpool outperformend every other option for NVMe random-access 4k reads not in cache. Variations on how many and which types of locks, work queuing and ordering, and when to start and stop threads and control their number made a considerable difference as well. In certain system configs, `CLONE_IO` made a difference. I use tiny (smaller than 1 page) userspace stacks with `clone`-based threads, and dynamic auto-tuning of the number of blocked and executing threads. > It uses the `preadv2` syscall to check if the file is in the page cache. More details here: https://github.com/ClickHouse/ClickHouse/pull/26791 https://github.com/ClickHouse/ClickHouse/pull/26791 That's `preadv2` with the `RWF_NOWAIT` flag. It proved slower when I tried it in my db engine, which was a bit of a surprise. `RWF_NOWAIT` is used to read data synchronously from cache, before passing it to the thread pool to read asynchronously if not in cache. I expected an average speedup when there are many cache hits, so I was surprised and disappointed to find the `preadv2(..,RWF_NOWAIT)` syscall to be slow enough that it was usually a performance loss overall to use it, at least on the kernel versions and hardware I tested on (a reasonably fast software-RAID NVMe). A nicer way to look at that is that the auto-tuning thread pool was sufficiently sleek and fast that the asynchronous read was fast enough to make the benefit of a synchronous path too small to be worth it. One not in your list is mmap_threadpool. I found for many workloads, that was faster than pread_threadpool, and of course it does a better job of sharing memory with the kernel. Unlike synchronous mmap, it is effectively an asynchronous read, where the thread does the page fault instead of syscall, so the main thread is not blocked and device I/O queues are kept full enough. > And there is a pull request adding io_uring: https://github.com/ClickHouse/ClickHouse/pull/38456 https://github.com/ClickHouse/ClickHouse/pull/38456 [..] Nevertheless, the advantages of `io_uring` for analytical databases are negligible. Compared with `pread_threadpool` equivalent (in my db engine), I found `io_uring` was sometimes similar, sometimes slower, never better, so not the preferred default. It makes sense that it could almost reach the devices' I/O capability, though with less control over queue depths than doing it directly in threads. But I was surprised that the "zero-syscall" queues of `io_uring` didn't provide a noticable improvement over `pread` syscalls, given that I measure a considerable baseline syscall overhead of all syscalls like `pread`, `preadv2` and `futex`, with that overhead having a significant throughput effect in `pread_threadpool` equivalent (because the NVMe devices were fast enough for syscall overhead to affect throughput). > - Unfortunately, it's [io_uring] unfinished and cannot be merged because the CI has found bugs. I found what I think is a subtle memory barrier bug in `liburing`. If the ClickHouse implementation is using `liburing` or copying its methods, it's conceivable that may be the cause of the hangs seen in CI. There are also kernel versions where `io_uring` was buggy, evidenced by changes in later kernels to fix bugs.
aanfhn 4y ago
What are people's and the market take on Starburst? I haven't kept up with databases and the general market, but recently heard about them. Is there a sizeable demand and market for them? Any other big players to watch for?
saurabhnanda 4y ago
What an amazingly written piece. I was learning interesting stuff and chuckling alongside, as well. Didn't get the closing remarks about Larry Ellison, though. Any explanation?
- friedman23 4y ago
  The Larry Ellison remarks are written with a lot of sarcasm.
  - thayne 4y ago
    It definitely seemed like sarcasm to me, but it felt a little out of place. Were other parts also sarcastic and I didn't pick up on them?
- SPBS 4y ago
  Oh great, it seems like this year he made the sarcasm about Larry Ellison a lot easier to pick up. Frankly I was still unsure how he really felt about Larry after his review from last year[1]. After reading this, it confirms to me that he doesn’t like Larry :). [1] https://news.ycombinator.com/item?id=29733729 https://news.ycombinator.com/item?id=29733729
mildbyte 4y ago
I mentioned it recently[0], but this looks like a very good topic to plug our new database, Seafowl, that we released last year [1]. It also uses Apache DataFusion (like IOx) and separates storage and compute (like Neon, Snowflake etc) but is designed for client-side Web apps to run analytical SQL queries over HTTP (using semantics that make the query results cacheable by browser caches and CDNs). This makes it really useful for things like interactive visualizations or dashboards. We're currently doing a lot of work at Splitgraph to reposition the product around this "analytics at the edge" use case, with usage-based billing, eventually moving our query execution from PostgreSQL to Seafowl. [0] https://news.ycombinator.com/item?id=34175545 https://news.ycombinator.com/item?id=34175545 [1] https://seafowl.io https://seafowl.io
- ddorian43 4y ago
  Your first link is not the correct one (click the timestamp of the comment to get the exact link)
  - mildbyte 4y ago
    Thanks for the catch! Fixed.
- zX41ZdbW 4y ago
  You can use ClickHouse as a stateless query engine to process externally hosted datasets. This way you just publish a MergeTree table to S3 bucket or a set of static files on an HTTP server, and then plugging it on an empty ClickHouse server. ClickHouse also wins in the comparison of serverless engines: https://datastudio.google.com/u/0/reporting/c870737c-e8b6-467f-9860-8cd60c751f81/page/Mn94C https://datastudio.google.com/u/0/reporting/c870737c-e8b6-46... Discussed here: https://github.com/ClickHouse/ClickHouse/issues/43589#issuecomment-1335105554 https://github.com/ClickHouse/ClickHouse/issues/43589#issuec...
  - deleted 4y ago
    [deleted]
gavinray 4y ago
Can we also start a general "Andy Pavlo" appreciation comment chain? Please comment below if Andy has in any way positively improved your life or helped you further your education/career. I can start: - Early in 2022, I decided I wanted to learn how databases like Postgres work and build one from scratch. If it wasn't for the CMU lectures available to the public on YouTube, I don't think I would have been able to give myself anywhere near a decent education. - This year, Andy even made the homework-grading platform available to the public, so you could submit solutions to the database coursework and get graded as if you were a student attending CMU. - Not only does Andy take the time to record, edit, and upload his lectures to the public, but he hosts a yearly seminar series where he interviews speakers from all over the database industry (https://db.cs.cmu.edu/seminar2022 https://db.cs.cmu.edu/seminar2022) - Andy does all he does (Professorship) on top of running OtterTune, I have no idea how he functions. He truly is the realest of all gangsters, and a secret member of the Wu-Tang Clan. Andy makes databases fun.
- deleted 4y ago
  [deleted]
- polskibus 4y ago
  Can you point me to the homework-grading platform ? Can one use it outside of standard course schedule ? It would be great to do the task in my own pace.
  - gavinray 4y ago
    The homework solutions are submitted to a platform called "Gradescope", see this tweet and the tweets above it in the thread for more details: https://twitter.com/andy_pavlo/status/1564305764730167303 https://twitter.com/andy_pavlo/status/1564305764730167303 Also, I should mention there is a very large (unofficial) Discord server (+1,000 people) where some of the TA's from this DB course and myself hang out if anyone is interested: https://discord.gg/7HPufafdsW https://discord.gg/7HPufafdsW And if you find database engineering interesting in general, I might also invite you to come hang out in Phil Eaton's hacker-space Discord, which is quite good: https://discord.gg/SAqUyrPkY7 https://discord.gg/SAqUyrPkY7
Alifatisk 4y ago
Is there a site that lists every db in every category? Like Timescale, Clickhouse for time series & Mongodb for documents etc
- deleted 4y ago
  [deleted]
- jsumrall 4y ago
  Check out https://db-engines.com/en/ranking https://db-engines.com/en/ranking Note that you can select categories on the left side to limit rankings to that category.
- boberoni 4y ago
  Dbdb.io has a catalog of database tech. It is maintained by the CMU database research group. https://dbdb.io https://dbdb.io
  - apavlo 4y ago
    It's maintained by me (also the author of this article).
mythhouse 4y ago
> Timescale’s $110m series C, Voltron Data’s $110m seed + series A, and Dbt Labs’s $222m series D. weird that dbt is mentioned in 'big database' funding. Also suprised to see the huge investment in dbt.
- rorymalcolm 4y ago
  I think it makes sense in the context of the poential of plain old SQL (and additions such as dbt) winning out vs Python or other general purpose programming solutions as the analytics workbench (although this is controversial, there's a school of thought that backs it).
- sa46 4y ago
  Slightly off topic, would someone mind providing a down to earth summary of the benefits (or disadvantages) of dbt? To ape the infamous iPod and Dropbox middlebrow dismissals: dbt looks like a control loop with a nice UI to make a database schema match templated SQL files. More seriously, at what level of maturity would a company find dbt useful and which teams benefit the most?
  - password4321 4y ago
    A couple anecdotes from a week ago: https://news.ycombinator.com/item?id=34159699#34160408 https://news.ycombinator.com/item?id=34159699#34160408
eximius 4y ago
What I want is a new database that defines tables in terms of rich structures like proto, like Spanner. If that doesn't exist in a couple years, maybe I'll try to build one.
- lukeramsden 4y ago
  > What I want is a new database that defines tables in terms of rich structures like proto, like Spanner. Do you mean like https://www.edgedb.com/ https://www.edgedb.com/? Or something more... rich?
  - eximius 4y ago
    Well, that's certainly something. I haven't stumbled across this before. It's very interesting, but one reason I want it to be proto is so you can re-use that for bindings with your applications.
    - lukeramsden 4y ago
      that's actually a very interesting idea. you could probably generate a proto schema from your edgedb schema...
      - RyanHamilton 4y ago
        translation = ORM? Which is what you wanted to avoid?
        eximius 4y ago
        Oh, it's not so bad. Having and independent schema file being one to one with the proto files is not quite as nice, but then you still can get bindings in each language to the proto. The goal is to avoid defining the table schema in each language, like ORMs, where things can be out of date. There may even be some benefits to this, as you might have some lifecycle hooks around migrations to keep the proto up to date with the current schema. You could do this with just protoc plugins to hide fields in the language bindings or you could do it turing the edgedb schema to proto translation.
getcrunk 4y ago
I see the author links to his youtube playlist for his CMU Databases class. https://www.youtube.com/playlist?list=PLSE8ODhjZXjaKScG3l0nuOiDTTqpfnWFf https://www.youtube.com/playlist?list=PLSE8ODhjZXjaKScG3l0nu... Are there any good recommendations for a lecture or series of lectures that go over Databases from a theoretical perspective but also do a practical survey of the various DB systems and technologies out there?
- whitepoplar 4y ago
  Not a lecture, but Martin Kleppmann's "Designing Data-Intensive Applications" book is excellent for this.
bullen 4y ago
I came back to MariaDB after leaving MySQL about a decade ago. The JDBC driver is 10x as big and times-out after less than 12 hours default. Things are not going in the right direction, but on the upside I now learned how to install MariaDB manually so installers can't brick my OS like MySQL did back in 2014 (MySQL bricked itself and no way to unbrick it except reinstalling Windows).
- KronisLV 4y ago
  > Things are not going in the right direction, but on the upside I now learned how to install MariaDB manually so installers can't brick my OS like MySQL did back in 2014 (MySQL bricked itself and no way to unbrick it except reinstalling Windows). For local development and testing, running a database in a container is generally a good idea: https://hub.docker.com/_/mariadb/ https://hub.docker.com/_/mariadb/ You know, if you ever want to be able to create various throwaway environments with resource limits or port mappings easily, for any number of versions you might need to run in each of your projects. This works similarly well for something like PostgreSQL, too! And if you need persistence, you can use either volumes or bind mounts.
- zX41ZdbW 4y ago
  What I found strange about MariaDB is that it is ~10 times slower than MySQL on OLAP queries[1]. It looks entirely wrong and most likely related to an issue with Debian packages. [1] https://pastila.nl/?00697bd3/ef6bf8858182a18150cc646c0c73e655.link https://pastila.nl/?00697bd3/ef6bf8858182a18150cc646c0c73e65...
- unixhero 4y ago
  Just postgres your life
  - bullen 4y ago
    I would, I even made my own ORM to be able to switch db's without friction, but the particular project I'm reviving was made before my ORM was ported to Postgres and Oracle. So much to do, so little time...
lvca 4y ago
What about ArcadeDB? https://arcadedb.com https://arcadedb.com (Apache2)
jerrygenser 4y ago
While the author seems to be an expert in production or web scale databases, I feel like this was a big year for SQLite also! Many people including myself are looking at it for server side use cases.
jimmyed 4y ago
Andy's obsession with mmap is interesting. I don't think it's the worst idea in some cases, like Prometheus. They use mmap to simplify the code and only keep the relevant (hot) parts of the index/data in memory. I guess the fact that the data is immutable helps, but that's the point; it's a good idea for some usecases and I don't agree with a blanket "mmap bad"
- nikita 4y ago
  We used mmap extensively and it took a gigantic effort to purge it out of the codebase in SingleStore. Lots of issues the biggest one is that munmap has a mutex and you incur a lot of contention. The second one is that mmap prevents you from a tight control of memory in various pipelining scenarios when you want to have your memory overhead to be constant and not sizeof(data).
  - menaerus 4y ago
    > Lots of issues the biggest one is that munmap has a mutex and you incur a lot of contention. But the alternative we have suffers from the same issue, doesn't it? malloc is implemented in terms of sbrk and mmap/munmap, both of which are guarded by locks. And malloc will also have its own user-space locks on top of the ones from the kernel. I checked glibc and jemalloc implementation. > mmap prevents you from a tight control of memory ... when you want to have your memory overhead to be constant and not sizeof(data). I am not sure I understand this argument. It's trivial to implement the user-space custom MMAP allocator that gives you no less control over the memory than other "types" of allocators. Actually, it can give you more power because you can avoid the locks from user-space malloc implementations if you know that the use-case you're crafting it for is single-threaded. I used this technique in the past for short-living objects and in certain workloads it improved the performance by a factor.
    - sakras 4y ago
      When Andy rages about databases using mmap, he means using it to replace the buffer pool. In theory, it provides exactly what the database needs by paging the data in and out on demand. In practice this scales poorly to large number of threads due to TLB shootdowns, and as Nikita said, you give up control of memory usage (since it’s the OS managing it). Nobody is arguing that you shouldn’t call mmap to get yourself a few anonymous virtual pages to do your work in.
carlineng 4y ago
> I had a student storm out of my class [1] because I was teaching relational databases instead of web3. Wow... that exchange is surreal. I wonder if it was some kind of fraternity prank or troll job? [1] https://www.youtube.com/watch?t=1490&v=uikbtpVZS2s https://www.youtube.com/watch?t=1490&v=uikbtpVZS2s
- softwaredoug 4y ago
  I don’t know, but it was pretty peak tech bro self parody. Tell a knowledgeable expert you know better and then leave the class.
- hinkley 4y ago
  Basically stormed out because it wasn't Web Scale.
deleted 4y ago
[deleted]
mekster 4y ago
I hope SurrealDB becomes mainstream. RDBMS haven't evolved at all for a very long time but only doing minor incremental improvements and consumers have to turn to complex queries as the principle never changes. It's time we get something more modern. - It has web UI to handle data. - It can write logic with JS (The more logic in DB the better.) - Can write a lot like SQL but way more powerful. - Receive real time updates. - Incremental backups. https://surrealdb.com/ https://surrealdb.com/ (Many examples on the landing page.) https://github.com/surrealdb/surrealdb https://github.com/surrealdb/surrealdb https://news.ycombinator.com/item?id=32550543 https://news.ycombinator.com/item?id=32550543
- valenterry 4y ago
  > It can write logic with JS You mean Javascript? If we are looking to make progress, then we shouldn't use those outdated (despite widely distributed) languages.
  - mekster 4y ago
    It is JavaScript. I'd say less learning curve the better adoption. What do you suggest as a good language to use here?
    - valenterry 4y ago
      With that attitude we would stick to assembly forever though. I don't know the exact requirements, but I have a hard time imaging that Javascript is a great choice, except for how many people know it.
- KRAKRISMOTT 4y ago
  They are gonna eat xata.io's lunch, xata is dreadfully overpriced.
- Omie6541 4y ago
  > It can write logic with JS (The more logic in DB the better.) why the more logic in DB the better? It's not visible to all devs, we can't put it in git, we can't write unit tests for the same!
alfor 4y ago
[flagged]
- fortituded0002 4y ago
  Which part are you referring to? I can't get through his highly opinionated and dismissive writing. Plus he refers to his daughter as his "biological daughter" which is super awkward and weird. I keep getting distracted by it. It comes across like it's supposed to be a passive-aggressive comment about his step-daughter or adopted daughter.
  - all_factz 4y ago
    If you clicked on the “biological daughter” link, you’d see it links out to a Twitter post where Pavlo shares a paternity test he had to take for some reason. So it’s just a joke, not a commentary on other children he may or may not have.
    - fortituded0002 4y ago
      Thanks for the context. Sounds like he's being passive-aggressive towards the parties involved that required the test for some reason. Now it's not just weird, but comes across as oddly petty for someone that is an adult that's teaching a bunch of younger people. Not sure what it is, but the way the guy is operating is a big turn off in a lot of ways.
- deleted 4y ago
  [deleted]
morningman-cmy 4y ago
(Apache Doris PMC) Apache Doris is an analytical database system that became an Apache top-level project in June this year. - https://github.com/apache/doris https://github.com/apache/doris In fact, a large part of the work of many emerging database systems, including Doris, is to supplement the missing functions compared with the traditional databases(Like Oracle), so as to help users smoothly migrate from the original system to the new system. Such as the CTE mentioned by zX41ZdbW. At the same time, most of projects are emphasizing their performance and cost-effectiveness, because this is the first factor for user selection. For example, Doris and SelectDB (Doris Commercial Company) also participated in Clickbench to reflect their performance advantages. But the implementations of various projects are actually converging (Doris' execution engine also refers to Clickhouse, thanks for open source), so we can only look for advantages in specific scenarios, such as high concurrency, time series, etc. On the other hand, performance and cost-effectiveness do not really solve users' problems, so we are also thinking about what users really need besides technology. Another interesting point is the trend of "de-distribution". Like Clickhouse and DuckDB, they can run well on a single machine with a multi-core, and can also meet the needs of many users. The reliability, availability, and scalability problems solved by "distribution" can all be solved in the PaaS of cloud vendors.
infogulch 4y ago
My takes for where databases are going in 2023 and beyond: Separation of storage and compute is clearly the right direction for cloud databases. Pay for what you use. A cache between object storage and compute will paper over the differences and enable supporting workloads that match yesterday's and today's typical application architectures. Still waiting for a database architecture that embraces the reality of distributed systems / concurrent mutation and exposes the right set of tools to reliably hit the theoretically minimum tradeoffs across a wide range of applications. CRDTs for example are making progress here but have not been generalized enough yet. At some point transactional workload volume will grow to what we call "big data" today, and we'll be forced to use more storage-efficient columnar data representations for OLTP. I'm curious to see where 'build your own database kit'-DataFusion and efficient dataframe tools like polars/arrow2 can take us.
- jlokier 4y ago
  > At some point transactional workload volume will grow to what we call "big data" today, and we'll be forced to use more storage-efficient columnar data representations for OLTP. I wonder. I built some proof of concept storage using row representations with a novel compression scheme designed to compete with compressed columnar format for space while being fast. The result is very storage efficient compared with competing alternatives that I started from. It has fewer writes per stored row than columnar storage if you don't have large batches of rows to write (often the case with OLTP), and fewer reads per full row when doing lots of random access point queries (sometimes the case with OLTP and certainly the case for the application I did the tests for). I didn't compare with a compressed columnar representation, but in theory (there was some compression theory) it should take same or less space in many cases as well as using less I/O. (It also was neither B-tree nor LSM-tree but had some desirable performance characteristics of both, but that's another story. Not a fractal tree / B^ε tree either, as sometimes people ask that). Presumably, well-funded groups are out there doing a much better job than I did in my spare time, so I wouldn't be surprised if "big data" OLTP doesn't end up requiring columnar representations after all, or if a scheme such as the one I used ends up competing with columnar formats in the end for space and performance reasons.
  - 4y ago
jwr 4y ago
Hmm. An article about modern databases that doesn't even mention FoundationDB? (which BTW Snowflake uses)
readonlybarbie 4y ago
[dead]
Dowwie 4y ago
Did you ever finish your paper about why no one should ever use MMAP in their database?
- digity 4y ago
  Are You Sure You Want to Use MMAP in Your Database Management System? https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf
  - deleted 4y ago
    [deleted]
nchohan 4y ago
Why are we using databases? Why not the blockchain? /s
einpoklum 4y ago
Andy certainly has a flair for rhetorics :-) From his 2021 DBMS year in review: > Too much blood has been shed in the database community in previous benchmark turf wars ... I even broke up with a girlfriend once because of sloppy benchmark results. Well, better over that than over east const vs const west.
vgt 4y ago
MotherDuck cofounder checking in. Great post, and thanks for the call although i am skeptical of commodotization of OLAP systems thanks to the likes of substrait and velox. It'll take a lot of unwilling participants.

365 points by gavinray 4y ago

[dead]

Neon CEO here. There are a number of ideas in the database space that the industry is adopting across the board: - Separation of storage and compute (Neon, AlloyDB, Aurora). Every cloud database should built one. It's a big undertaking, but benefits are undeniable. - Good query processor for analytics (Snowflake, Velox, Singlestore) - Open source. Especially in OLTP open source == trust - HTAP. Can run mixed workload (Singlestore, Unistore): both OLTP (apps) and OLAP (reporting). This has always been a dream, but we still live in the world of dedicated systems: E.g. Snowflake and Postgres. - Shared nothing sharding (Vitess). This is the most controversial as you lose compatibility with the mothership (MySQL for Vitess). So it's unclear this will be the dominant architecture in the future. I think the world may get to "dynamic sharding" where storage stays separate and compute can be multinode and the user can easily and instantly change the number of nodes.

swyx 4y ago
since you were also a founder of Singlestore, just wondering - what will be Neon's take on HTAP workloads? trying to see how much you mean it that the industry is "adopting (HTAP) across the board"
- nikita 4y ago
  I wouldn't say HTAP is an across the board trend. The need is mostly in the enterprise where we see high value mixed workloads. Neon will take a Postgres centric POV. I consider a few approaches, but not committing to any particular at this point. The reason to not commit is that there is a lot more opportunity for Neon in serverless, devtools, and edge. And it makes a ton more sense to partner for OLAP workloads. So what is Postgres centric POV: - Plugins like Timescale and Citus - Seamless integration with sending CDC to Snowflake, Singlestore, Databricks, Clickhouse, and Motherduck - FDW for integration with Snowflake, Singlestore, Clickhouse, Databricks, and Motherduck - We looked at putting DuckDB on Neon storage with Jordan Tigani, but it was too big of a lift as DuckDB doesn't push all data through the WAL. Maybe in the future.
  - zX41ZdbW 4y ago
    Let's commit this to ClickBench: https://github.com/neondatabase/neon/pull/2855 https://github.com/neondatabase/neon/pull/2855 So, NeonDB can be compared with other serverless Postgres offerings.
    - nikita 4y ago
      The reason we run OLAP bench is because when you separate storage full table get impacted: you fetch pages from remote storage one by one and the mitigation is prefetch. So our goal is to be on par with vanilla Postgres. We don't care about winning OLAP benchmarks.
      - zX41ZdbW 4y ago
        ClickBench is still ok to compare non-OLAP databases on OLAP workloads - filter away the column-oriented DBMS, and you will be left with the comparison of non-OLAP databases. I think it makes sense to add NeonDB for the sake of completeness.
        nikita 4y ago
        I think lots and lots of benchmarks are useful internally. They highlight where the soft spots are. This is however very different from publishing benchmark results. As a dbaas vendor we should publish benchmarks that your user base really cares about which demonstrate that the system is mature and can handle the core workload your users expect you to handle. That is why publishing an OLAP benchmark results for an OLTP system makes very very little sense. (BTW OLAP systems like Clickhouse should really publish TPC-DS SF1000+ results) For OLTP systems there is TPC-C but it's actually not that telling as it is so simple. I can start seeing that people in OLTP care more and more about latencies, hence all the edge related work and optimization of the drivers. So Neon will be publishing end-to-end latencies benchmarks and also make sure it is on par with vanilla Postgres on a broad set of benchmarks. Those we will run internally and the results will simply be commits on our github.
  - swyx 4y ago
gavinray 4y ago
Really enjoyed the talk from your engineer Heikki at Andy's CMU "Databases!" seminar series earlier this year, great stuff =) https://www.youtube.com/watch?v=rES0yzeERns https://www.youtube.com/watch?v=rES0yzeERns Also, it's been neat to have mostly worked our replacing our old Heroku integration at Hasura with you folks.
emptysea 4y ago
I’m not well versed in the tech but is Spanner the sort of dynamic sharding approach your talking about?
- gavinray 4y ago
  Scylla also uses a shared-nothing sharding approach, they have an open-source "asynchronous I/O engine" toolkit that is used inside of ScyllaDB called Seastar They have some good reading on this approach in the Seastar docs if you're curious, it's a great general introduction to Shared Nothing: https://seastar.io/shared-nothing/#:~:text=The%20Seastar%20Model%3A%20Shared%2Dnothing,not%20shared%20memory%20between%20threads https://seastar.io/shared-nothing/#:~:text=The%20Seastar%20M.... https://www.linuxfoundation.org/webinars/under-the-hood-of-a-shard-per-core-database-architecture https://www.linuxfoundation.org/webinars/under-the-hood-of-a... This is usually generalized to some variation of "Physical-OS-thread-per-CPU-core" architecture, where each CPU core gets it's own dedicated chunk of data. You use hashing or some other mechanism to route requests for data to the right core. Each CPU core can have user-land concurrency or "virtual threads" (fibers/coroutines/etc) as well, so you're not limited to sequential code necessarily. A combination of thread-per-core x userland-coroutines-per-thread is how modern query processing approaches like "Morsel-Driven Parallelism" work. CPU-1-OS-THREAD-1 COROUTINE_1 COROUTINE_2 CPU-2-OS-THREAD-2 COROUTINE_35 ... COROUTINE_102 https://db.in.tum.de/~leis/papers/morsels.pdf https://db.in.tum.de/~leis/papers/morsels.pdf
  - AtlasBarfed 4y ago
    Scylla is a C++ rewrite of cassandra, the threading has been better as well with thread-per-core. Cassandra 4.0 has some thread-per-core rearchitecture so it remains to be seen if Scylla has performance advantages it used to tout. I have not checked if Scylla ever caught up to Cassandra 3.0's feature set, last I checked it had not. It may be "going its own way" on features as well. I believe (they can correct me). My personal recommendation would be to start with Cassandra, and when operations allow it, move to scylla if performance testing shows it is better. There is also exotic stuff like Rocksandra. Cassandra and Scylla are fundamentally similar in operations (add/remove nodes, scaling, etc). The scylla folks will contend the operations are easier since they get more perf per node so you need less nodes. It's a good idea with cassandra / scylla to bootstrap with a performance testing, data loading, provisioning loop. It gets you used to operations and the tooling for backups, loads, and load testing and metrics.
jandrewrogers 4y ago
The separation of storage and compute is controversial, though it is definitely convenient for cloud operators. It is literally going backward in terms of the available storage bandwidth, and many modern database architectures can very effectively use all the storage bandwidth available in modern systems. There is not enough network bandwidth, and won’t be for the foreseeable future, for running storage over networks to be efficient. Database workloads are also starting to move to edge environments, which make it even worse. The resource mismatch currently leaves a lot of compute sitting idle in practice. Another issue with this model is that it tends to offer poor ratios of bandwidth to storage size. This is at odds with the desire to run mixed workloads in a single system.
- gavinray 4y ago
  > The separation of storage and compute is controversial I don't think it's a controversial topic, it's an emerging trend. See for example this slide from the recent RocksDB meetup on their vision for the next 3 years of RocksDB's future: https://twitter.com/GavinRayDev/status/1600666127025156096 https://twitter.com/GavinRayDev/status/1600666127025156096 Disaggregated storage is becoming quite a hot topic, and there are several technologies fueling this. There are two major advances happening in terms of disk I/O right now (that aren't to do with async IO uring stuff) 1. NVMe Zones + NVMe FDP specs, which are ways for userland applications to control placement of writes on drive (like a software Flash-Translation-Layer) 2. NVMe-over-Fabrics or similar technologies (with rdma/tcp) enables the access of a remote disk with 'comparable' latency to that of a local one. E.g. if local disk access is 100us, AWS can give you 110us with NVMe-OF. These two advances support the growing trend of disaggregated storage --- Full credit to Arun George from the Samsung Memory group for explaining this to me.
  - jandrewrogers 4y ago
    The storage-over-fabric latency improvements allow it to work well for software that was not particularly optimized for storage performance, which has obvious benefits that I am not arguing. There is a good reason storage-over-fabric has been a popular database deployment model for decades, it does have advantages, but this emerging trend is nothing new nor are the issues that it has. For systems that are highly optimized for storage performance, storage bandwidth tends to be more important than storage latency and that is what is being sacrificed for the sake of disaggregating storage. I see integer factor reductions in workload throughput when deployed on disaggregated storage versus the usually NVMe JBOD, and that factor tracks real-world storage bandwidth pretty well, particularly at high storage densities. At scale, these economics start to matter quite a lot. To be clear, disaggregated storage has no effect on database design generally, seeing as how it has been widely used for decades. It does put an artificial floor on achievable throughput for many workloads due to the bandwidth bottleneck. When people realize this then the pendulum swings the other way (again), so I wouldn't want to overfit for disaggregated storage.
zX41ZdbW 4y ago
The most developed analytical query engine as of today is ClickHouse [1]. It is open-source, has the separation of storage and compute and the best performance. ClickHouse is a source of inspiration for many emerging database engines: Velox, Doris, DuckDB, Datafusion... [1] https://github.com/ClickHouse/ClickHouse https://github.com/ClickHouse/ClickHouse Disclaimer: I work on ClickHouse.
- ddorian43 4y ago
  > has the separation of storage and compute and the best performance. This part is not yet production ready though. And it's unclear when it will be. Though no other open source OLAP does it either AFAIK.
  - _dark_matter_ 4y ago
    SparkSQL + delta lake definitely counts as an open source OLAP separated compute/storage.
Ataraxy 4y ago
I actually look forward to Neon but won't actually use it for real until pricing becomes available.
- nikita 4y ago
  Stay tuned. It's coming very very soon.
fulafel 4y ago
Regarding separation of storage and compute, do those DBs you mention enable merging the application backend code and db compute node to a single process? Meaning I would deploy my application to the DB compute node and use its runtime, or have the compute node present as a component I can import to my backend code as a library. I think this is one powerful thing about Datomic (and maybe XTDB?).

deleted 4y ago

[deleted]

yingjunwu 4y ago

I'm the founder of RisingWave Labs. Glad to know that our company was mentioned in the blog. Technology wise, agreed with Nikita that storage-compute-decoupling is the trend. Severless will also be the main focus. Market wise, I observed several trends: * streaming. streamlining the data pipeline is a new trend. RisingWave and Materialize are both addresssing this market. They will probably need to figure out how to compete against established solutions like KsqlDB and Apache Flink. * real-time. There are probably 20+ vendors selling real-time OLAP databases. They all claim that they are cheaper, faster, better than Snowflake/Redshift. Just name a few: Druid, Pinot, ClickHouse, StarRocks etc. Let's see who will be the winner in 2023. * small data. The biggest winner in this domain is Motherduck, a startup commercializing DuckDB. Redshift/Snowflake are a overkill for most companies who do not own large amount of data. Motherduck may be the right choice. * hybrid. Several companies have started selling hybrid databases (mainly HTAP for now): Google AlloyDB, Snowflake Unistore, SingleStore, PingCAP TiDB+TiFlash, etc. In 2023, we will also watch how Databricks challenges Snowflake in the data warehouse domain. Best luck!

danielvaughn 4y ago
Druid is indeed a bit faster than Snowflake, but a previous team I worked on was able to achieve similar latencies under heavy load using some clever techniques with materialized views.
glogla 4y ago
> * real-time. There are probably 20+ vendors selling real-time OLAP databases. They all claim that they are cheaper, faster, better than Snowflake/Redshift. Just name a few: Druid, Pinot, ClickHouse, StarRocks etc. Let's see who will be the winner in 2023. These are faster but they are also very limited. Most of these have very limited support for joins, can't do proper SQL (e.g. no CTEs in ClickHouse) and require a lot of careful design for your data (e.g. StarRocks can do smart pre-aggregation but you have to tell it which columns are dimensions and which are values). That is fine for limited use cases - like "dedicated engineering team is building specific application" - but doesn't really work for general purpose data platforms. Snowflake I think is successful because it is pretty good at handling everything you throw at it, even if it costs a lot of money.

friedman23 4y ago

Happy to find this blog, wish there was more talk about NewSQL databases. I've personally seen the amount of cost that goes into migrating off of traditional RDBMs systems to sharded RDBMs systems and was wondering why people don't go with these scalable systems from the start now that they exist.

likeabbas 4y ago
+1. Recently, Yugabyte has seemed intriguing to me. (It seems like) You can launch it as a standalone instance and use it just like Postgres early in your company, and then when you need to distribute, you can do that.
- brnewd 4y ago
  I think many love the promise of a distributed database, but in practice there is a real speed and feature set compromise to make.
  - likeabbas 4y ago
    I found that having a NoSQL distributed database for prod that stream Change Data Captures to a read-only SQL DB to be a great compromise. The SQL data was slightly stale but that wasn't an issue for most situations (like graphing data or certain error metrics)
pm90 4y ago
Probably because there is a lot more operational understanding of “traditional” RDBMs and they’re sufficient in early stages. Once your product takes off it can be difficult to scale but by that time you probably have enough money to attract engineers that can do the migration.
- friedman23 4y ago
  Yes, that's the common refrain, just migrate when you hit scale. Just pay people to do the migration. My experience at multiple, billion dollar companies has been that it's never that easy. People become afraid of doing schema migrations and tools like dynamodb become the default datastore choice for all new feature development. Every single startup I've worked at, technical cruft that existed from company founding never disappears despite the founders intent that it was never supposed to stay that way and it slows down feature development to a crawl. If NewSQL databases exist that support 90% of the features of postgres but give support for cross shard transactions then I'd rather use them than technology that has been proven to not scale. Unfortunately I'm not an expert on distributed databases and there is not much written about these databases in practice outside of the startups creating the databases themselves.

lioeters 4y ago

Seems like Supabase deserves a mention, with their contributions to the Postgres ecosystem. https://supabase.com/ https://supabase.com/ pg_graphql - https://supabase.com/blog/pg-graphql-v1 https://supabase.com/blog/pg-graphql-v1 pg_jsonschema - https://supabase.com/blog/pg-jsonschema-a-postgres-extension-for-json-validation https://supabase.com/blog/pg-jsonschema-a-postgres-extension... Postgres WASM - https://supabase.com/blog/postgres-wasm https://supabase.com/blog/postgres-wasm ..And other open-source features.

apavlo 4y ago
We want to write a paper on the vast extensions ecosystem for Postgres at some point. It's on my TODO list for 2023. Having so many plug-ins/add-ons available is unique to Postgres and due to Stonebraker's original vision of including extensibility as a first-class design principle in the DBMS.
- dang 4y ago
  I didn't know that and it would make for an interesting historical article in its own right.
- whitepoplar 4y ago
  In doing so, it would be great if you could provide details around extension safety. I've heard that some extensions may not play nicely in combination with others, and it would be really nice to know how one is to vet extensions for use in production.
danielvaughn 4y ago
Also Planetscale edit: I'm okay with being downvoted, but I'm honestly asking why, not complaining. Is Planetscale not an interesting database?
- mritchie712 4y ago
  fwiw, I don't get the downvotes either

marsupialtail_2 4y ago

I am writing a Python based SQL query engine: https://github.com/marsupialtail/quokka https://github.com/marsupialtail/quokka. My personal goal is to get Andy Pavlo to mention it in his year-end blogs. I agree with many of the points made in the blog by Andy. Writing a distributed database has become way easier due to open source components like Ray, Arrow, Velox, DuckDB, SQLGlot etc. I personally believe we will see a switch from JVM based technologies to Rust/C based with Python wrapper

apavlo 4y ago
I saw your engine a week or so ago. Good stuff. I wanted to add it to dbdb.io but I couldn't find an SVG version of this: https://github.com/marsupialtail/quokka/blob/master/docs/docs/quokka2.png https://github.com/marsupialtail/quokka/blob/master/docs/doc... Can you email it to me?
- marsupialtail_2 4y ago
  Just did.
password4321 4y ago
discussed 3 days ago: https://news.ycombinator.com/item?id=34189422 https://news.ycombinator.com/item?id=34189422

zX41ZdbW 4y ago

> he claimed that because MonetDB focuses on read-only OLAP workloads, then MMAP was good enough ClickHouse has switchable IO engines: - read; - pread; - mmap; - pread_threadpool; The best performance is achieved with the combined method, named 'pread_threadpool'. It uses the `preadv2` syscall to check if the file is in the page cache. More details here: https://github.com/ClickHouse/ClickHouse/pull/26791 https://github.com/ClickHouse/ClickHouse/pull/26791 There was also big research of performance on HDD, SSD, and Optane, comparing various methods: https://clickhouse.com/blog/a-journey-to-io_uring-aio-and-modern-storage-devices https://clickhouse.com/blog/a-journey-to-io_uring-aio-and-mo... And there is a pull request adding io_uring: https://github.com/ClickHouse/ClickHouse/pull/38456 https://github.com/ClickHouse/ClickHouse/pull/38456 - Unfortunately, it's unfinished and cannot be merged because the CI has found bugs. Nevertheless, the advantages of io_uring for analytical databases are negligible. The `mmap` method is still useful. For example, for data import. See here: https://github.com/ClickHouse/ClickHouse/pull/43927 https://github.com/ClickHouse/ClickHouse/pull/43927

jlokier 4y ago
> The best performance is achieved with the combined method, named 'pread_threadpool'. I found the same when testing my own database engine. `pread` in a suitably well-designed threadpool outperformend every other option for NVMe random-access 4k reads not in cache. Variations on how many and which types of locks, work queuing and ordering, and when to start and stop threads and control their number made a considerable difference as well. In certain system configs, `CLONE_IO` made a difference. I use tiny (smaller than 1 page) userspace stacks with `clone`-based threads, and dynamic auto-tuning of the number of blocked and executing threads. > It uses the `preadv2` syscall to check if the file is in the page cache. More details here: https://github.com/ClickHouse/ClickHouse/pull/26791 https://github.com/ClickHouse/ClickHouse/pull/26791 That's `preadv2` with the `RWF_NOWAIT` flag. It proved slower when I tried it in my db engine, which was a bit of a surprise. `RWF_NOWAIT` is used to read data synchronously from cache, before passing it to the thread pool to read asynchronously if not in cache. I expected an average speedup when there are many cache hits, so I was surprised and disappointed to find the `preadv2(..,RWF_NOWAIT)` syscall to be slow enough that it was usually a performance loss overall to use it, at least on the kernel versions and hardware I tested on (a reasonably fast software-RAID NVMe). A nicer way to look at that is that the auto-tuning thread pool was sufficiently sleek and fast that the asynchronous read was fast enough to make the benefit of a synchronous path too small to be worth it. One not in your list is mmap_threadpool. I found for many workloads, that was faster than pread_threadpool, and of course it does a better job of sharing memory with the kernel. Unlike synchronous mmap, it is effectively an asynchronous read, where the thread does the page fault instead of syscall, so the main thread is not blocked and device I/O queues are kept full enough. > And there is a pull request adding io_uring: https://github.com/ClickHouse/ClickHouse/pull/38456 https://github.com/ClickHouse/ClickHouse/pull/38456 [..] Nevertheless, the advantages of `io_uring` for analytical databases are negligible. Compared with `pread_threadpool` equivalent (in my db engine), I found `io_uring` was sometimes similar, sometimes slower, never better, so not the preferred default. It makes sense that it could almost reach the devices' I/O capability, though with less control over queue depths than doing it directly in threads. But I was surprised that the "zero-syscall" queues of `io_uring` didn't provide a noticable improvement over `pread` syscalls, given that I measure a considerable baseline syscall overhead of all syscalls like `pread`, `preadv2` and `futex`, with that overhead having a significant throughput effect in `pread_threadpool` equivalent (because the NVMe devices were fast enough for syscall overhead to affect throughput). > - Unfortunately, it's [io_uring] unfinished and cannot be merged because the CI has found bugs. I found what I think is a subtle memory barrier bug in `liburing`. If the ClickHouse implementation is using `liburing` or copying its methods, it's conceivable that may be the cause of the hangs seen in CI. There are also kernel versions where `io_uring` was buggy, evidenced by changes in later kernels to fix bugs.

aanfhn 4y ago

What are people's and the market take on Starburst? I haven't kept up with databases and the general market, but recently heard about them. Is there a sizeable demand and market for them? Any other big players to watch for?

saurabhnanda 4y ago

What an amazingly written piece. I was learning interesting stuff and chuckling alongside, as well. Didn't get the closing remarks about Larry Ellison, though. Any explanation?

friedman23 4y ago
The Larry Ellison remarks are written with a lot of sarcasm.
- thayne 4y ago
  It definitely seemed like sarcasm to me, but it felt a little out of place. Were other parts also sarcastic and I didn't pick up on them?
SPBS 4y ago
Oh great, it seems like this year he made the sarcasm about Larry Ellison a lot easier to pick up. Frankly I was still unsure how he really felt about Larry after his review from last year[1]. After reading this, it confirms to me that he doesn’t like Larry :). [1] https://news.ycombinator.com/item?id=29733729 https://news.ycombinator.com/item?id=29733729

mildbyte 4y ago

I mentioned it recently[0], but this looks like a very good topic to plug our new database, Seafowl, that we released last year [1]. It also uses Apache DataFusion (like IOx) and separates storage and compute (like Neon, Snowflake etc) but is designed for client-side Web apps to run analytical SQL queries over HTTP (using semantics that make the query results cacheable by browser caches and CDNs). This makes it really useful for things like interactive visualizations or dashboards. We're currently doing a lot of work at Splitgraph to reposition the product around this "analytics at the edge" use case, with usage-based billing, eventually moving our query execution from PostgreSQL to Seafowl. [0] https://news.ycombinator.com/item?id=34175545 https://news.ycombinator.com/item?id=34175545 [1] https://seafowl.io https://seafowl.io

ddorian43 4y ago
Your first link is not the correct one (click the timestamp of the comment to get the exact link)
- mildbyte 4y ago
  Thanks for the catch! Fixed.
zX41ZdbW 4y ago
You can use ClickHouse as a stateless query engine to process externally hosted datasets. This way you just publish a MergeTree table to S3 bucket or a set of static files on an HTTP server, and then plugging it on an empty ClickHouse server. ClickHouse also wins in the comparison of serverless engines: https://datastudio.google.com/u/0/reporting/c870737c-e8b6-467f-9860-8cd60c751f81/page/Mn94C https://datastudio.google.com/u/0/reporting/c870737c-e8b6-46... Discussed here: https://github.com/ClickHouse/ClickHouse/issues/43589#issuecomment-1335105554 https://github.com/ClickHouse/ClickHouse/issues/43589#issuec...
- deleted 4y ago
  [deleted]

gavinray 4y ago

Can we also start a general "Andy Pavlo" appreciation comment chain? Please comment below if Andy has in any way positively improved your life or helped you further your education/career. I can start: - Early in 2022, I decided I wanted to learn how databases like Postgres work and build one from scratch. If it wasn't for the CMU lectures available to the public on YouTube, I don't think I would have been able to give myself anywhere near a decent education. - This year, Andy even made the homework-grading platform available to the public, so you could submit solutions to the database coursework and get graded as if you were a student attending CMU. - Not only does Andy take the time to record, edit, and upload his lectures to the public, but he hosts a yearly seminar series where he interviews speakers from all over the database industry (https://db.cs.cmu.edu/seminar2022 https://db.cs.cmu.edu/seminar2022) - Andy does all he does (Professorship) on top of running OtterTune, I have no idea how he functions. He truly is the realest of all gangsters, and a secret member of the Wu-Tang Clan. Andy makes databases fun.

deleted 4y ago
[deleted]
polskibus 4y ago
Can you point me to the homework-grading platform ? Can one use it outside of standard course schedule ? It would be great to do the task in my own pace.
- gavinray 4y ago
  The homework solutions are submitted to a platform called "Gradescope", see this tweet and the tweets above it in the thread for more details: https://twitter.com/andy_pavlo/status/1564305764730167303 https://twitter.com/andy_pavlo/status/1564305764730167303 Also, I should mention there is a very large (unofficial) Discord server (+1,000 people) where some of the TA's from this DB course and myself hang out if anyone is interested: https://discord.gg/7HPufafdsW https://discord.gg/7HPufafdsW And if you find database engineering interesting in general, I might also invite you to come hang out in Phil Eaton's hacker-space Discord, which is quite good: https://discord.gg/SAqUyrPkY7 https://discord.gg/SAqUyrPkY7

Alifatisk 4y ago

Is there a site that lists every db in every category? Like Timescale, Clickhouse for time series & Mongodb for documents etc

deleted 4y ago
[deleted]
jsumrall 4y ago
Check out https://db-engines.com/en/ranking https://db-engines.com/en/ranking Note that you can select categories on the left side to limit rankings to that category.
boberoni 4y ago
Dbdb.io has a catalog of database tech. It is maintained by the CMU database research group. https://dbdb.io https://dbdb.io
- apavlo 4y ago
  It's maintained by me (also the author of this article).

mythhouse 4y ago

> Timescale’s $110m series C, Voltron Data’s $110m seed + series A, and Dbt Labs’s $222m series D. weird that dbt is mentioned in 'big database' funding. Also suprised to see the huge investment in dbt.

rorymalcolm 4y ago
I think it makes sense in the context of the poential of plain old SQL (and additions such as dbt) winning out vs Python or other general purpose programming solutions as the analytics workbench (although this is controversial, there's a school of thought that backs it).
sa46 4y ago
Slightly off topic, would someone mind providing a down to earth summary of the benefits (or disadvantages) of dbt? To ape the infamous iPod and Dropbox middlebrow dismissals: dbt looks like a control loop with a nice UI to make a database schema match templated SQL files. More seriously, at what level of maturity would a company find dbt useful and which teams benefit the most?
- password4321 4y ago
  A couple anecdotes from a week ago: https://news.ycombinator.com/item?id=34159699#34160408 https://news.ycombinator.com/item?id=34159699#34160408

eximius 4y ago

What I want is a new database that defines tables in terms of rich structures like proto, like Spanner. If that doesn't exist in a couple years, maybe I'll try to build one.

lukeramsden 4y ago
> What I want is a new database that defines tables in terms of rich structures like proto, like Spanner. Do you mean like https://www.edgedb.com/ https://www.edgedb.com/? Or something more... rich?
- eximius 4y ago
  Well, that's certainly something. I haven't stumbled across this before. It's very interesting, but one reason I want it to be proto is so you can re-use that for bindings with your applications.
  - lukeramsden 4y ago
    that's actually a very interesting idea. you could probably generate a proto schema from your edgedb schema...
    - RyanHamilton 4y ago
      translation = ORM? Which is what you wanted to avoid?
      - eximius 4y ago
        Oh, it's not so bad. Having and independent schema file being one to one with the proto files is not quite as nice, but then you still can get bindings in each language to the proto. The goal is to avoid defining the table schema in each language, like ORMs, where things can be out of date. There may even be some benefits to this, as you might have some lifecycle hooks around migrations to keep the proto up to date with the current schema. You could do this with just protoc plugins to hide fields in the language bindings or you could do it turing the edgedb schema to proto translation.

getcrunk 4y ago

I see the author links to his youtube playlist for his CMU Databases class. https://www.youtube.com/playlist?list=PLSE8ODhjZXjaKScG3l0nuOiDTTqpfnWFf https://www.youtube.com/playlist?list=PLSE8ODhjZXjaKScG3l0nu... Are there any good recommendations for a lecture or series of lectures that go over Databases from a theoretical perspective but also do a practical survey of the various DB systems and technologies out there?

whitepoplar 4y ago
Not a lecture, but Martin Kleppmann's "Designing Data-Intensive Applications" book is excellent for this.

bullen 4y ago

I came back to MariaDB after leaving MySQL about a decade ago. The JDBC driver is 10x as big and times-out after less than 12 hours default. Things are not going in the right direction, but on the upside I now learned how to install MariaDB manually so installers can't brick my OS like MySQL did back in 2014 (MySQL bricked itself and no way to unbrick it except reinstalling Windows).

KronisLV 4y ago
> Things are not going in the right direction, but on the upside I now learned how to install MariaDB manually so installers can't brick my OS like MySQL did back in 2014 (MySQL bricked itself and no way to unbrick it except reinstalling Windows). For local development and testing, running a database in a container is generally a good idea: https://hub.docker.com/_/mariadb/ https://hub.docker.com/_/mariadb/ You know, if you ever want to be able to create various throwaway environments with resource limits or port mappings easily, for any number of versions you might need to run in each of your projects. This works similarly well for something like PostgreSQL, too! And if you need persistence, you can use either volumes or bind mounts.
zX41ZdbW 4y ago
What I found strange about MariaDB is that it is ~10 times slower than MySQL on OLAP queries[1]. It looks entirely wrong and most likely related to an issue with Debian packages. [1] https://pastila.nl/?00697bd3/ef6bf8858182a18150cc646c0c73e655.link https://pastila.nl/?00697bd3/ef6bf8858182a18150cc646c0c73e65...
unixhero 4y ago
Just postgres your life
- bullen 4y ago
  I would, I even made my own ORM to be able to switch db's without friction, but the particular project I'm reviving was made before my ORM was ported to Postgres and Oracle. So much to do, so little time...

lvca 4y ago

What about ArcadeDB? https://arcadedb.com https://arcadedb.com (Apache2)

jerrygenser 4y ago

While the author seems to be an expert in production or web scale databases, I feel like this was a big year for SQLite also! Many people including myself are looking at it for server side use cases.

jimmyed 4y ago

Andy's obsession with mmap is interesting. I don't think it's the worst idea in some cases, like Prometheus. They use mmap to simplify the code and only keep the relevant (hot) parts of the index/data in memory. I guess the fact that the data is immutable helps, but that's the point; it's a good idea for some usecases and I don't agree with a blanket "mmap bad"

nikita 4y ago
We used mmap extensively and it took a gigantic effort to purge it out of the codebase in SingleStore. Lots of issues the biggest one is that munmap has a mutex and you incur a lot of contention. The second one is that mmap prevents you from a tight control of memory in various pipelining scenarios when you want to have your memory overhead to be constant and not sizeof(data).
- menaerus 4y ago
  > Lots of issues the biggest one is that munmap has a mutex and you incur a lot of contention. But the alternative we have suffers from the same issue, doesn't it? malloc is implemented in terms of sbrk and mmap/munmap, both of which are guarded by locks. And malloc will also have its own user-space locks on top of the ones from the kernel. I checked glibc and jemalloc implementation. > mmap prevents you from a tight control of memory ... when you want to have your memory overhead to be constant and not sizeof(data). I am not sure I understand this argument. It's trivial to implement the user-space custom MMAP allocator that gives you no less control over the memory than other "types" of allocators. Actually, it can give you more power because you can avoid the locks from user-space malloc implementations if you know that the use-case you're crafting it for is single-threaded. I used this technique in the past for short-living objects and in certain workloads it improved the performance by a factor.
  - sakras 4y ago
    When Andy rages about databases using mmap, he means using it to replace the buffer pool. In theory, it provides exactly what the database needs by paging the data in and out on demand. In practice this scales poorly to large number of threads due to TLB shootdowns, and as Nikita said, you give up control of memory usage (since it’s the OS managing it). Nobody is arguing that you shouldn’t call mmap to get yourself a few anonymous virtual pages to do your work in.

carlineng 4y ago

> I had a student storm out of my class [1] because I was teaching relational databases instead of web3. Wow... that exchange is surreal. I wonder if it was some kind of fraternity prank or troll job? [1] https://www.youtube.com/watch?t=1490&v=uikbtpVZS2s https://www.youtube.com/watch?t=1490&v=uikbtpVZS2s

softwaredoug 4y ago
I don’t know, but it was pretty peak tech bro self parody. Tell a knowledgeable expert you know better and then leave the class.
hinkley 4y ago
Basically stormed out because it wasn't Web Scale.

deleted 4y ago

[deleted]

mekster 4y ago

I hope SurrealDB becomes mainstream. RDBMS haven't evolved at all for a very long time but only doing minor incremental improvements and consumers have to turn to complex queries as the principle never changes. It's time we get something more modern. - It has web UI to handle data. - It can write logic with JS (The more logic in DB the better.) - Can write a lot like SQL but way more powerful. - Receive real time updates. - Incremental backups. https://surrealdb.com/ https://surrealdb.com/ (Many examples on the landing page.) https://github.com/surrealdb/surrealdb https://github.com/surrealdb/surrealdb https://news.ycombinator.com/item?id=32550543 https://news.ycombinator.com/item?id=32550543

valenterry 4y ago
> It can write logic with JS You mean Javascript? If we are looking to make progress, then we shouldn't use those outdated (despite widely distributed) languages.
- mekster 4y ago
  It is JavaScript. I'd say less learning curve the better adoption. What do you suggest as a good language to use here?
  - valenterry 4y ago
    With that attitude we would stick to assembly forever though. I don't know the exact requirements, but I have a hard time imaging that Javascript is a great choice, except for how many people know it.
KRAKRISMOTT 4y ago
They are gonna eat xata.io's lunch, xata is dreadfully overpriced.
Omie6541 4y ago
> It can write logic with JS (The more logic in DB the better.) why the more logic in DB the better? It's not visible to all devs, we can't put it in git, we can't write unit tests for the same!

alfor 4y ago

[flagged]

fortituded0002 4y ago
Which part are you referring to? I can't get through his highly opinionated and dismissive writing. Plus he refers to his daughter as his "biological daughter" which is super awkward and weird. I keep getting distracted by it. It comes across like it's supposed to be a passive-aggressive comment about his step-daughter or adopted daughter.
- all_factz 4y ago
  If you clicked on the “biological daughter” link, you’d see it links out to a Twitter post where Pavlo shares a paternity test he had to take for some reason. So it’s just a joke, not a commentary on other children he may or may not have.
  - fortituded0002 4y ago
    Thanks for the context. Sounds like he's being passive-aggressive towards the parties involved that required the test for some reason. Now it's not just weird, but comes across as oddly petty for someone that is an adult that's teaching a bunch of younger people. Not sure what it is, but the way the guy is operating is a big turn off in a lot of ways.
deleted 4y ago
[deleted]

morningman-cmy 4y ago

(Apache Doris PMC) Apache Doris is an analytical database system that became an Apache top-level project in June this year. - https://github.com/apache/doris https://github.com/apache/doris In fact, a large part of the work of many emerging database systems, including Doris, is to supplement the missing functions compared with the traditional databases(Like Oracle), so as to help users smoothly migrate from the original system to the new system. Such as the CTE mentioned by zX41ZdbW. At the same time, most of projects are emphasizing their performance and cost-effectiveness, because this is the first factor for user selection. For example, Doris and SelectDB (Doris Commercial Company) also participated in Clickbench to reflect their performance advantages. But the implementations of various projects are actually converging (Doris' execution engine also refers to Clickhouse, thanks for open source), so we can only look for advantages in specific scenarios, such as high concurrency, time series, etc. On the other hand, performance and cost-effectiveness do not really solve users' problems, so we are also thinking about what users really need besides technology. Another interesting point is the trend of "de-distribution". Like Clickhouse and DuckDB, they can run well on a single machine with a multi-core, and can also meet the needs of many users. The reliability, availability, and scalability problems solved by "distribution" can all be solved in the PaaS of cloud vendors.

infogulch 4y ago

My takes for where databases are going in 2023 and beyond: Separation of storage and compute is clearly the right direction for cloud databases. Pay for what you use. A cache between object storage and compute will paper over the differences and enable supporting workloads that match yesterday's and today's typical application architectures. Still waiting for a database architecture that embraces the reality of distributed systems / concurrent mutation and exposes the right set of tools to reliably hit the theoretically minimum tradeoffs across a wide range of applications. CRDTs for example are making progress here but have not been generalized enough yet. At some point transactional workload volume will grow to what we call "big data" today, and we'll be forced to use more storage-efficient columnar data representations for OLTP. I'm curious to see where 'build your own database kit'-DataFusion and efficient dataframe tools like polars/arrow2 can take us.

jlokier 4y ago
> At some point transactional workload volume will grow to what we call "big data" today, and we'll be forced to use more storage-efficient columnar data representations for OLTP. I wonder. I built some proof of concept storage using row representations with a novel compression scheme designed to compete with compressed columnar format for space while being fast. The result is very storage efficient compared with competing alternatives that I started from. It has fewer writes per stored row than columnar storage if you don't have large batches of rows to write (often the case with OLTP), and fewer reads per full row when doing lots of random access point queries (sometimes the case with OLTP and certainly the case for the application I did the tests for). I didn't compare with a compressed columnar representation, but in theory (there was some compression theory) it should take same or less space in many cases as well as using less I/O. (It also was neither B-tree nor LSM-tree but had some desirable performance characteristics of both, but that's another story. Not a fractal tree / B^ε tree either, as sometimes people ask that). Presumably, well-funded groups are out there doing a much better job than I did in my spare time, so I wouldn't be surprised if "big data" OLTP doesn't end up requiring columnar representations after all, or if a scheme such as the one I used ends up competing with columnar formats in the end for space and performance reasons.
- 4y ago

jwr 4y ago

Hmm. An article about modern databases that doesn't even mention FoundationDB? (which BTW Snowflake uses)

readonlybarbie 4y ago

[dead]

Dowwie 4y ago

Did you ever finish your paper about why no one should ever use MMAP in their database?

digity 4y ago
Are You Sure You Want to Use MMAP in Your Database Management System? https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf
- deleted 4y ago
  [deleted]

nchohan 4y ago

Why are we using databases? Why not the blockchain? /s

einpoklum 4y ago

Andy certainly has a flair for rhetorics :-) From his 2021 DBMS year in review: > Too much blood has been shed in the database community in previous benchmark turf wars ... I even broke up with a girlfriend once because of sloppy benchmark results. Well, better over that than over east const vs const west.

vgt 4y ago

MotherDuck cofounder checking in. Great post, and thanks for the call although i am skeptical of commodotization of OLAP systems thanks to the likes of substrait and velox. It'll take a lot of unwilling participants.