16 ms·

Databases in 2024: A Year in Review

625 points by avinassh 2y ago

mihirrd 2y ago
Quite informative
badindentation 2y ago
The section on Larry Ellison is amusing.
- epolanski 2y ago
  I couldn't understand if it was satire or something. Do they believe the guy marries a 30 years old cause she loves him? In any case, who cares, how was that relevant..
  - masklinn 2y ago
    It's definitely satire, how would you take this sentence as serious: > I told Larry this especially means a lot to me because my former #1 ranked Ph.D. student is now a professor in Michigan's Computer Science department with their famous Database Group.
  - CRConrad 2y ago
    > I couldn't understand if it was satire or something. The quotes from last year's bit on Ellison in sibling comment https://news.ycombinator.com/item?id=42567484 https://news.ycombinator.com/item?id=42567484 might help you make up your mind.
- leeoniya 2y ago
  "Do not fall into the trap of anthropomorphising Larry Ellison." https://m.youtube.com/watch?t=33m1s&v=-zRN7XLCRhc&feature=youtu.be https://m.youtube.com/watch?t=33m1s&v=-zRN7XLCRhc&feature=yo...
- avinassh 2y ago
  Larry always makes appearances in his reviews, lol > But the real big news in 2023 was how Elon Musk personally helped reset Larry's Twitter password after he invested $1b in Musk's takeover of the social media company. And with this $1b password reset, we were graced in October 2023 with Larry's second-ever tweet and his first new one in over a decade. https://www.cs.cmu.edu/~pavlo/blog/2024/01/2023-databases-retrospective.html https://www.cs.cmu.edu/~pavlo/blog/2024/01/2023-databases-re... > These journalists made it sound like Larry was doing something nefarious or indecent, like the time he made his pregnant third wife sign a prenup two hours before their wedding. I can assure you that Larry was only trying to use his vast wealth as the 7th richest person in the world to help his country. His participation in this call is admirable and should be lauded. Free and fair elections are not a trivial affair, like a boat race where sometimes shenanigans are okay as long as you win. Larry has done other great things with his money that are overlooked, like spending $370m on anti-aging research so that he can live forever https://www.cs.cmu.edu/~pavlo/blog/2022/12/2022-databases-retrospective.html https://www.cs.cmu.edu/~pavlo/blog/2022/12/2022-databases-re...
samanthasu 2y ago
would love to see what Andy's take on GreptimeDB https://github.com/GreptimeTeam/greptimedb https://github.com/GreptimeTeam/greptimedb
- leeoniya 2y ago
  made same comment recently in https://news.ycombinator.com/item?id=42330055#42331927 https://news.ycombinator.com/item?id=42330055#42331927
  - samanthasu 2y ago
    looking forward to the bonus content!
  - DennisZ_89 2y ago
    Glad to see people start trying on GreptimeDB. We're committed to building a fully open-source and cost-effective, unified time series database for metrics, logs, and events. To this point, it may still be too small to be included in the year database summary. But hope we'll grow fast in 2025 :)
rozenmd 2y ago
I loved ottertune, it's a shame it died the way it did.
memhole 2y ago
Love the style! CMU making databases cool. Sorry to hear about OtterTune.
Beefin 2y ago
TL;DR SQL is king
m_ke 2y ago
Andy is a treasure, if only we had more professors like him
antirez 2y ago
Wow, the reasons why Redis commands API suck in Andy's video (linked in the post) are the weakest ever. It is possible to make a case against the Redis API (I would not agree of course but... it's totally legitimate), but you gotta have stronger arguments than those, particularly if you are a teacher of some kind. Especially: you need to be somewhat fluent in Redis and how developers use Redis in order to understand why so many people like it, and then elaborate what it's wrong about it (if you believe there is something wrong). The video shows a general feeling of "I don't really use / know this, but I don't like how NON-SQL it is".
- nojito 2y ago
  SQL is king and history has shown non-sql languages are not good which causes many non-sql DBMS's to adopt sql eventually.
  - antirez 2y ago
    Many non-SQL DBs had query languages that were broken Javascript-ish versions of SQL. Of course, this is wrong, and people will eventually adopt SQL instead. But if your data model isn't anything like relational DBs, non-SQL makes a ton of sense. OP seems to miss exactly this, that the Redis query language is shaped on the Redis data model, that is basically alien to the relational model. The idea behind Redis data model is that "describe data" then "query those data in random ways" is conceptually nice but practically will not model very well many use cases. SQL databases plagued tech with performance issues for decades because of that. So Redis says instead: you need to put your data thinking about fundamental things like data structures and access times and the way you'll need those data back. And the API reflects this. You don't have to automatically agree with that. But you have to understand that, then provide your "I'm against" arguments. Especially if you are in front of young people listening to you.
    - im_down_w_otp 2y ago
      Agreed. Many noSQL-boom-era databases eventually bolted on a SQL-esque layer, but that was also because they were mostly also all targeting "enterprise database" use cases and customers who both expected that and whose use cases largely fit with it. So, there was a lot of pressure to conform to norms when the advantage of not doing so wasn't immediately self-evident. We have a database [1] and query language [2] that's tailored to storing & querying trace/telemetry data produced by different layers and components of cyber-physical systems for systems engineers to analyze, verify, and validate what a complex system is doing. It's not quite a traditional relational problem. It's not quite a traditional time series problem. It's not quite a traditional graph problem. Addressing the way that systems engineers think about their domain in an effective way required coming up with something different. Are there caveats and rough edges? Sure. But, they're a lot less pernicious and onerous than the alternative of trying to leverage a bunch of ill-fitting menageries of different solutions. Redis is fit-for-purpose. So, it makes sense that its query interface would also express that. [1] https://docs.auxon.io/modality/ https://docs.auxon.io/modality/ [2] https://docs.auxon.io/speqtr/ https://docs.auxon.io/speqtr/
codeulike 2y ago
Weird how SQL Server and its Azure variants gets no mention. It dominates in certain sectors. DBEngines ranks it third most popular overall https://db-engines.com/en/ranking https://db-engines.com/en/ranking
- patja 2y ago
  The fawning over Larry Ellison is also weird.
  - cactusfrog 2y ago
    The joke is that his greed/ unwillingness to squeeze margins has made the entire database company ecosystem possible.
- RadiozRadioz 2y ago
  Lots of people deliberately avoid Microsoft technologies and their whole ecosystem. There's of course interesting stuff happening there, but not enough for those outside the ecosystem to care. It's more a cultural thing than anything else. HN for example largely leans away from MS. It's quite interesting how little overlap there is between the two worlds sometimes. Speaking as one of those people, it's just not my thing, so it's not on my radar at all. There's enough stuff happening outside MS to keep me busy forever.
- teej 2y ago
  Are people choosing SQL Server independently of the Microsoft ecosystem? My understating is that you typically use it because you’re forced to choose a MS product.
  - rawgabbit 2y ago
    SQL Server is a terrific product. And I detest most things Microsoft.
    - breadwinner 2y ago
sigbottle 2y ago
These year in review posts are really neat, I liked the AI in review posts really well. Maybe algorithms review or TCS review or some specific math topic review next?
CT4u8798 2y ago
I love SQL. I'm not a full-time developer but always use SQL over other abstractions, which I find extremely confusing and way more complicated that plain SQL.
- skeeter2020 2y ago
  I'm now firmly into management but the one skill I use very regularly is SQL. By far the best investment I made in my entire career was a little bit of relational algebra, some casual study of DBMS internals and a lot of hands-on SQL. The quasi-standards have also made it the easiest transfer across specific DBs and their flavours over the years. PSA: Hi kids, here's a dinosaur with yet more free advice: put the tiniest bit of effort into SQL early on and watch the compound interest add up.
- threeseed 2y ago
  That's just because it is what you are comfortable with. Many developers will jump straight for ORMs when given the chance.
  - Tostino 2y ago
    Which for certain types of applications ORMs absolutely have their use.
  - dominicrose 2y ago
    Some ORMs have weird design issues. Eloquent for example allows you to pull relationships lazily on single objects, so if you're in a loop that'll create a lot of queries. So much for laziness! I'm OK with this ability, but the API shouldn't encourage it by making it trivial.
wahnfrieden 2y ago
2024 was also the year that Realm died
travisgriggs 2y ago
The funding section had me thinking “one of these is not like the others”. Both the amount and count of successive rounds.
ksec 2y ago
>Six years after MySQL v8 went GA, the team turned v9 out on the streets. ......Oracle is putting all its time and energy into its proprietary MySQL Heatwave service. Oracle actually released 9.1 already in 2024. [1] And expect another release this month, and every quarter. So I think MySQL continues to get some new features bug fix and support like it used to. Contrary to most people think it is all going to Heatwave. I just hope Vector will be open source later as official to MySQL rather than behind Heatwaves. [1] https://dev.mysql.com/doc/relnotes/mysql/9.1/en/news-9-1-0.html https://dev.mysql.com/doc/relnotes/mysql/9.1/en/news-9-1-0.h...
the_arun 2y ago
This person started with news on DB - reviewing all prominent DBs & finally ended talking about love of Larry Ellison. A perfect human in the days of LLMs. Amazing write up.
RedShift1 2y ago
I've been using plain postgres for over 5 years now, reading this I feel like I'm in the eye of a storm...
mebcitto 2y ago
A couple of spicy things: > OtterTune. Dana, Bohan, and I worked on this research project and startup for almost a decade. And now it is dead. I am disappointed at how a particular company treated us at the end, so they are forever banned from recruiting CMU-DB students. They know who they are and what they did. Ouch. > Lastly, I want to give a shout-out to ByteBase for their article Database Tools in 2024: A Year in Review. In previous years, they emailed me asking for permission to translate my end-of-year database articles into Chinese for their blog. This year, they could not wait for me to finish writing this one, so they jocked my flow and wrote their own off-brand article with the same title and premise. Also sounds like he's preparing a new company: > I hope to announce our next start-up soon (hint: it’s about databases).
- spprashant 2y ago
  Anyone know what company he may be talking about?
  - C-programmer 2y ago
    Inspect element on https://web.archive.org/web/20240827031455/https://ottertune.com/ https://web.archive.org/web/20240827031455/https://ottertune... For more context: > I'm to sad to announce that @OtterTuneAI is officially dead. Our service is shutdown and we let everyone go today (1mo notice). I can't got into details of what happened but we got screwed over by a PE Postgres company on an acquisition offer. https://x.com/andy_pavlo/status/1801687420330770841 https://x.com/andy_pavlo/status/1801687420330770841
    - yencabulator 2y ago
      Because that was a little too subtle and I was sufficiently curious: view-source:https://web.archive.org/web/20240827031455/https://ottertune.com/ https://web.archive.org/web/20240827031455/https://ottertune... scroll until you see ASCII art
      - swyx 2y ago
mrtimo 2y ago
Enjoyed his roundup in the "Shoving Ducks into Everything" section. DuckDB is a great tool. In April 2020, the creator of DuckDB gave a talk at CMU. In the beginning he makes a convincing argument (in 5 minutes) why data scientists don't use RDBMS and how this was the genesis of DuckDB. Here is a video that starts 3 minutes into the talk (where is argument starts): https://youtu.be/PFUZlNQIndo?si=ql9n2QuBlAEuGIqo&t=204 https://youtu.be/PFUZlNQIndo?si=ql9n2QuBlAEuGIqo&t=204
dig1 2y ago
> There was no major effort to fork off MongoDB, Neo4j, Kafka, or CockroachDB when they announced their license changes. AFAIK people didn't take MongoDB seriously from the start, especially with the "web scale database" joke circulating. The Neo4j Community version has been under GPLv3 for quite some time, while the Enterprise version has always been somewhat closed, regardless of whether the source code was available on GitHub (the mentioned license change affected the Enterprise version). Regarding CockroachDB, I must admit that I've only heard about it on HN and don't know anyone who seriously uses it. As for Kafka, there are two versions: Apache Kafka, the open-source version that almost everyone uses (under the Apache license), and Confluent Kafka, which is Apache Kafka enhanced with many additional features from Confluent, and the license change affected Confluent Kafka. In short, maybe the majority simply didn't care about these projects very much, so there is no major fork. > It cannot be because the Redis and Elasticsearch install base is so much larger than these other systems, and therefore, there were more people upset by the change since the number of MongoDB and Kafka installations was equally as large when they switched their licenses. I can’t speak for MongoDB, but the Confluent Kafka install base is significantly smaller than that of Apache Kafka, Redis and ES. > Dana, Bohan, and I worked on this research project and startup for almost a decade. And now it is dead. I am disappointed at how a particular company treated us at the end, so they are forever banned from recruiting CMU-DB students. They know who they are and what they did. Call me a skeptic, but I can't see this as a fair approach. If your company fails for whatever reasons, you should not recruit the university department/group/students against your peers (I can't find that CMU-DB was one of the founders of Ottertune). Wrt Andy, here are [1] somehow interesting views from (presumably) previous employees. [1] https://www.reddit.com/r/Database/comments/1dgaazw/comment/ldw90qj/ https://www.reddit.com/r/Database/comments/1dgaazw/comment/l...
- paulddraper 2y ago
  There are production uses of MongoDB (Stripe comes to mind). But it is certainly not a popular choice there.
  - threeseed 2y ago
    MongoDB does over $2b in revenue (and growing by 20%) each year. There are a lot of production uses.
maeil 2y ago
Good read! > Postgres' support for extensions and plugins is impressive. One of the original design goals of Postgres from the 1980s was to be extensible. The intention was to easily support new access methods and new data types and operations on those data types (i.e., object-relational). Since 2006, Postgres' "hook" API. Our research shows that Postgres has the most expansive and diverse extension ecosystem compared to every other DBMS. Greenhorn developers don't even know that there are non-Postgres databases which have extensions too - such is the gap! I wouldn't be surprised if Postgres had as many as all others combined.
softwaredoug 2y ago
On the “Amazon can just offer your DB as a service” Yes this can happen. But a lot of people don’t want a AWS managed service. They're like 30% cheaper for 30% less value. They can develop a bad reputation and feel like weird forks (kinesis vs Kafka) that have weird undocumented gotchas and edge cases that never get fixed. Many teams want to host on k8s anyway, and you’ll probably have better k8s support from the main project. Another example is the success of Flink over hosted Google Dataflow. Seems eventually the teams I know trend to the most mainstream OSS implementation over time, maybe after early prototyping on a managed system. IMO it might not be the highest growth market anymore. Those who want to pay for a managed service will. But many are just figuring out a k8s based solution to their infra needs as k8s knowledge becomes more ubiquitous.
- senderista 2y ago
  Kinesis is not Kafka (at all), nor is Google Dataflow Flink (perhaps you're thinking of Apache Beam?).
wslh 2y ago
Great heads up. I wonder about graph databases. He mentioned <https://umbra-db.com/ https://umbra-db.com/> and <https://cedardb.com/ https://cedardb.com/> both include the graph use case and I wonder how they compare to <https://neo4j.com/ https://neo4j.com/>.
- Tanjreeve 2y ago
  Umbra and cedar are both still relational databases. Afaik the jury is now out on if graph databases are better than modern relational databases for most graph queries especially ones with good query planners/compilers. The only time graph DBs seem to be consistently better is for very specialist many degrees traversals for small amounts of data.
  - wslh 2y ago
    My point is that both Umbra and Cedar mentioned graph support, and I don't believe this was a coincidence. Umbra highlights: "Groupjoins enable efficient computation of aggregates, worst-case optimal joins handle complex queries on graph structured data, and range joins efficiently evaluate queries with conditions on location or time intervals." while Cedar includes in the hero: "CedarDB is a relational-first database system that delivers best-in-class performance for all your workloads, from transactional to analytical to graph,..."
    - Tanjreeve 2y ago
      I think that phrase of "relational first" explains it best. Anything that supports joins can support a graph. Whether it'll perform satisfactorily is going to depend on the compiler and use case/size.
atombender 2y ago
The article mentions Greenplum, but it's worth noting that when the code was closed, several of the original developers created an open-source fork, Cloudberry, which seems to be thriving. Cloudberry was accepted into the Apache project this year, and has synced with Postgres 14, whereas the closed-source Greenplum is still stuck on Postgres 12. The architecture is quite ancient at this point, but I'm not sure it's completely outdated. It's single-master shared-nothing, with shards distributed among replicas, similar to Citus. But the GPORCA query planner is probably the most advanced distributed query planner in the open source world at this point. From what I know, Greenplum/Cloudberry can be significantly faster than Citus thanks to the planner being smarter about splitting the work across shards.
- ledgerdev 2y ago
  Thanks for the cloudberry mention, wasn't aware of it.
- Icathian 2y ago
  This is mostly correct, but it's worth mentioning that cloudberry substantially predates Greenplum going closed source. It just got quite a boost from that change happening. Different dev team too, afaik none of the original Greenplum team was involved with Cloudberry until very recently. Also, Greenplum 7 tracks postgres 14. Which is still old at this point, but not so bad as 12.... I also don't think I'd call the architecture ancient. Just very tightly coupled to postgres' own (as a fork of postgres that tries to ingest new versions from upstream every year or two) and paying the overhead of that choice in the modern landscape. Source: former member of the Greenplum Kernel team.
  - atombender 2y ago
    Thanks for the context. In what way would you say Cloudberry lags behind Greenplum technology-wise? I see newer Greenplum versions have a lot of planner improvements. Greenplum 7 is listed as tracking Postgres 12 in the release announcement [1], and the release notes for later 7.x versions don't mention anything. Is there a newer release with higher compatibility? When I say ancient, I mean that it's a "classical" shared-nothing design where the database is partitioned and hosted as parallel, self-contained replica servers, where each node runs as a shard that could, in theory, by queried independently of the master database. This is in contrast to newer architectures where data is sharded at the heap level (e.g. Yugabyte, CockroachDB) and/or compute is separated from data (e.g. Aurora, ClickHouse, Neon, TiDB). [1] https://greenplum.org/partition-in-greenplum-7-whats-new/ https://greenplum.org/partition-in-greenplum-7-whats-new/
PeterZaitsev 2y ago
I think one thing Andy misses about why people were pissed about Elastic and Redis but not as many for MongoDB and some other is their license and size of Contributors Community. When original license is as restricted as AGPL it is unlikely there is much of embedded use... so less people are impacted in truly catastrophic way Also if there is no contributor community to speak of... who is going to do the fork ? I put some thoughts about it in my post about ScyllaDB https://peterzaitsev.com/thoughts-on-scylladb-license-change/ https://peterzaitsev.com/thoughts-on-scylladb-license-change...
- tayo42 2y ago
  > In the case of Redis, I can only think that people perceive Redis Ltd. as unfairly profiting off others' work since the company's founders were not the system's original creators. An analysis of Redis' source code repository also shows that a sizable percentage of contributions to the DBMS comes from outside the company He mentions this in "Andy’s Take" section btw
  - PeterZaitsev 2y ago
    Yes. Not the license though.
- rockwotj 2y ago
  RE ScyllaDB, there will absolutely be no fork and it’s very unlikely they have ever had a meaningful contribution to Scylla OSS (which is not changing, just going to bit rot and the enterprise version which was closed source was moved to source available). The reasoning being the bar is very high for contributions. It’s C++ 20/23 without virtual memory and a userland cooperative thread per core scheduler (this is the underlying seastar framework). The skills needed to do anything meaningful here is very high and I don’t see the motivation behind it as I would expect people having feature requests would be customers, have an easier time extending Cassandra or the feature would be available in the Enterprise version.
  - menaerus 2y ago
kwillets 2y ago
I spent the past year puzzling over the DB market as well, but I don't feel like I'm much closer to understanding it. It appears that a lot of attention is now directed at the folks doing 100 MB queries, and the high end has moved past everybody's radar. My idea of an exciting product is Ocient, who have skipped over Cloud and gone for hyperscale on-prem hardware. Yellowbrick is also a contender here. I have a lot of experience with Vertica, and they seem to have gotten stuck in this niche as well, with sales tilted towards big accounts, but less traction in smaller shops, and a difficult road to get a SaaS or similar easy-start offering. There's a crossover point where self-managed is cheaper than cloud, but nobody seems to have any idea where it is. Snowflake will gladly tell you that your sub-$1M Vertica cluster should be replaced by $10M of sluggish SaaS, and that you are saving money by doing so. These decisions seem more in the realm of psychology or political science. DHH's cloud exit was a refreshing take on the expense issue, even if it wasn't strictly in the database space -- the cost per VCPU and so forth that he documented is a good start for estimating savings, and he debunked a lot of the "hidden costs" that cloud maximalists claim. In the business/financial space the biggest news to me was the correction in Snowflake's stock price, which seemed to indicate that investors were finally noticing metrics like price-performance, but they added a little more AI and went back into irrationality. I'm heavily in favor of DuckDB, Hudi, Iceberg, S3 tables, and the like. Mixing high-end and low-end tools seems like the best strategy (although settling on one high-end DWH has also worked IME), and the low end is getting better and cheaper, squeezing out the mid-range SaaS vendors. In research I found Goetz Graefe's work in offset-value coding exciting -- he's wired it into query operators in a way that saves a lot of CPU on sorting and joins/aggregation. This is a technique that I've applied favorably in string sorting, and it was discovered in the DB community decades ago but largely forgotten. (This work precedes 2024, but I'm a slow study.)
- downsplat 2y ago
  > There's a crossover point where self-managed is cheaper than cloud Single data point here: before cloud managed dbs were a thing our smallish startup was running mysql on virtual servers by installing it from the linux package manager. Always worked great, runs without needing manual attention for years at a time once set up, so I've never felt the need to change. So at least in some cases the crossover point is "right from the start".
bionhoward 2y ago
Redis is slow?
- kermatt 2y ago
  I wish there was more context around that statement in his post. Redis while not having some of the features he mentions in [1] (i.e. SQL), when used for what it excels at is usually not considered "slow". As an in-memory data structure server, a common use case is to use it for where some operations in a typical RDBMS are slow. [1] https://youtu.be/fZbwD1gzjLk?t=2018 https://youtu.be/fZbwD1gzjLk?t=2018
- senderista 2y ago
  Yes. 1. It is single-threaded, which severely limits throughput for a single instance. 2. All communication must go over a socket, which severely impacts latency for use cases where it could otherwise run in-process.
  - breadwinner 2y ago
    What is the better alternative?
    - bdangubic 2y ago
      depends on your use case. also, for most redis is fast enough which is why it is wildly popular
    - yla92 2y ago
      Do checkout https://www.dragonflydb.io https://www.dragonflydb.io . It's also mentioned in the review!
      - CRConrad 2y ago
        I'm so tired of these Web pages that look like iPhone screens.
- 2y ago
ak_111 2y ago
Wow his database startup that raised 12M died this year after only three years. If anything this shows how insanely difficult it must be to succeed as a database startup (when was the most recent startup success in this space?), as the founding team is stellar. On the other hand I am surprised it died this quick and interested to know if they did a proper postmortem. Not only did they raise way more than is needed to survive for three years but the idea is about utilising AI to improve DB performance and I find it hard to imagine they couldn't find more investors to lend them a lifeline with all the AI hype.
- Tanjreeve 2y ago
  No idea about internal workings but as a "DB optimisation" startup you're competing with - most people don't need it - People who do need it having DBAs/Operations people - or consultancies - Database vendors that have automatic optimisation as a feature Ok "AI" in the name but I think for something as specific as DB optimisation AI jazz hands probably don't work as well. Writing it out it almost seems harder than being an actual DB vendor.
- didgetmaster 2y ago
  I thought the same thing. I have a personal project that started out as a file system replacement (object store), but also does some amazing DB operations and is useful for analytics. I have often thought about trying to turn it into a profitable business; but the thought of needing to raise insane amounts of capital just to get a few years of runway, seems daunting.
- hipadev23 2y ago
  Andy's ego is far too big to actually run a company and his focus is entirely on vanity metrics. You can see it in these database reviews where he spends substantial time pining over how much money CompanyX raised and shitting on the bread and butter tools used by those of us who don't have VC money to blow on whatever DBaaS is the current hype.
- cmrdporcupine 2y ago
  I mean it's also arguably the most difficult time for startups (in general) in the last 15-20 years. Esp if you got your initial investment and valuation before the end of ZIRP.
lvl155 2y ago
I highly recommend their Youtube series on databases. They have great guest speakers.
- syspec 2y ago
  Link?
  - nojito 2y ago
    https://www.youtube.com/c/CMUDatabaseGroup https://www.youtube.com/c/CMUDatabaseGroup
deleted 2y ago
[deleted]
based2 2y ago
https://dbos-project.github.io https://dbos-project.github.io news
- refset 2y ago
  More specifically, DBOS Inc. raised a $8.5 million seed round [0] and is backed by Michael Stonebraker (the creator of Postgres). I initially assumed Andy was alluding to this when he wrote "the most famous database octogenarian splashing cash" :) [0] https://techcrunch.com/2024/03/12/new-startup-from-postgres-creator-puts-the-database-at-heart-of-software-stack/ https://techcrunch.com/2024/03/12/new-startup-from-postgres-...
roark_howard 2y ago
DuckDB dominating over DataFusion could fuel the ongoing language war with a great half baked argument!
- yencabulator 2y ago
  Every time I've tried to use DuckDB it has segfaulted on me, so yeah I'm betting on DataFusion..
  - 1egg0myegg0 2y ago
    That is very surprising to hear! If you can reproduce it, could you please file a bug report on GitHub? That would be a huge help!!
    - riku_iki 2y ago
      I personally filed bunch of bugs, they mostly were autoclosed because of no activity after 3 months. This is very discouraging, and I rather invest effort in looking for workarounds in the future.
polishdude20 2y ago
After I interviewing at OtterTune a while back and being bombarded with multiple rounds of leetcode questions, I somehow knew OtterTune wouldn't make it
gigatexal 2y ago
This take screams more than a technical criticism but of something personal. “I'll be blunt: I don't care for Redis. It is slow, it has fake transactions, and its query syntax is a freakshow. Our experiments at CMU found Dragonfly to have much more impressive performance numbers (even with a single CPU core). In my database course, I use the Redis query language as an example of what not to do.” (From the article) Of course it’s not to be used as a general purpose DB it’s keys and values. Used for caches and things like that. In my experience in real world scenarios and loads vanilla single threaded Redis is stable, fast, and nigh bulletproof.
- deleted 2y ago
  [deleted]
rednafi 2y ago
Loved the overview. Hated the shade toward Redis. Redis has arguably the best key-value query syntax, and there’s a reason so many people swear by it. True, the decision-makers at Redis Ltd are absolute pieces of trash, but Redis itself is a delightful piece of engineering artifact. I don’t care about the billion-dollar drama behind a piece of tech, but Redis defined the key-value query API for many similar databases. Trashing it just because it isn’t SQL-like feels unjustified.
quotemstr 2y ago
There's one QOL extension that I haven't seen anyone else implement: dimensional analysis. I can declare a column is an integer. Why not an integer that expresses feet? Why shouldn't I be able to write SELECT 1inch + 1cm and get a correctly computed length? Why can't the query parser help me avoid nonsense like SELECT 1kg + 1hr? All this stuff is pretty straightforward to add and would help avoid avoidable mistakes.
zhousun 2y ago
It's such an honor our https://pgmooncake.com/ https://pgmooncake.com/ is covered in the review! A little sad Andy didn't share more of his thoughts on the intersection between Data and AI, and how that's going to evolve.
Upvoter33 2y ago
Pretty funny. One factual issue: "The university had previously announced that this player was transferring from Louisiana State to Michigan." This is not true. Underwood had committed to LSU but then switched his commitment to Michigan. He was still in high school at the time, and has never attended LSU. But, do you really expect a funny database prof to know much about football?
- mpbart 2y ago
  I never thought I’d see a discussion about the Underwood NIL drama on a databases blog post but here we are.
funkysapien 2y ago
[dead]
bcoates 2y ago
"I've never met anybody that used Alteryx" I have! It's a pretty good no-code/minimal-code graphical ELT+Analytics in one tool. It's one of those alternate-universe tools that has it's own way of doing things from everything else in the industry, but it’s pragmatic and the people who use it tend to love it. The one thing that makes it viable is that is has/had (pre-acquisition) very aggressive compatibility with anything else that can hold data, so you can use it as a bolt-on to whatever other databases or files your company has. Despite what the PE press release about the acquisition says, it has virtually nothing to do with AI, at lease in the modern big NN sense. If you're looking to fix your giant pile of alteryx workbooks or migrate them to something else, hmu
osigurdson 2y ago
More like: "Database license drama - a year in review".
pbrunoster 2y ago
oracle, sqlserver , db2 , informix , teradata for example, does not exist ... ok
pbrunoster 2y ago
oracle, sqlserver , db2 , informix , teradata for example, does not exist ... ok ....
nwatson 2y ago
Interesting seeing the death of blockchain-based AWS QLDB mentioned. I worked at a company for a while that used QLDB as the primary system of record. The idea is great but the problem is that due to performance and other QLDB limitations all data had to be mirrored to an RDBMS via a streaming/queuing system, and there always were programmatic errors in interpreting data arriving for import into the RDBMS ... text field too long for RDBMS field; wrong data type or overflowing integer; invalid text encoding; ... Etc. These errors had to be noticed, debugged, fixed, and data had to be re-streamed. In the meantime official transactions were missing from the RDBMS side, which was used for reporting, driving the UI, deriving monetary obligations, etc. it was not worth the trouble. (I was lucky to not be involved in that design or implementation.)
kopirgan 2y ago
Wow so much to learn reading this . Thanks
hdesh 2y ago
> The upcoming year is going to be the test of strength for many database startups. Nobody wants to be the next MariaDB Corporation, The link for "MariaDB corporation" points to an empty image with white colour background. Can anyone explain the context here?
dandan7 2y ago
Thanks for the article. On the topic of redis: 3 executives from redis built FalkorDB (succeeded redisgraph) raising 3m to build a graphdb for better rag (ref:https://github.com/FalkorDB/GraphRAG-SDK https://github.com/FalkorDB/GraphRAG-SDK)
uncomplexity_ 2y ago
thank you for your work andy! for a moment i got reminded of the rap music in your courses im glad that tigerbeetle got here, really impressive team they have. there are a lot of other missing alien technologies i've discovered recently too like quickwit which is like elasticsearch but s3-compatible, and typesense which is like elasticsearch but memory-based
swyx 2y ago
> I need to figure out to juice my stats because in September 2024, Wikipedia removed the article about me over not having enough citations. guys, what are we doing here. this is ridiculous. andy pavlo cannot get an article on wikipedia? have you seen his work?
phartenfeller 2y ago
There is a lot about Larry Ellison, but not a single word about this year's rather big Oracle release (23ai)?
deleted 2y ago
[deleted]
christinewarr 2y ago
[dead]
christinewarr 2y ago
[dead]
tiffanyava 2y ago
[dead]
Lavigne3746 2y ago
[dead]

625 points by avinassh 2y ago

mihirrd 2y ago

Quite informative

badindentation 2y ago

The section on Larry Ellison is amusing.

epolanski 2y ago
I couldn't understand if it was satire or something. Do they believe the guy marries a 30 years old cause she loves him? In any case, who cares, how was that relevant..
- masklinn 2y ago
  It's definitely satire, how would you take this sentence as serious: > I told Larry this especially means a lot to me because my former #1 ranked Ph.D. student is now a professor in Michigan's Computer Science department with their famous Database Group.
- CRConrad 2y ago
  > I couldn't understand if it was satire or something. The quotes from last year's bit on Ellison in sibling comment https://news.ycombinator.com/item?id=42567484 https://news.ycombinator.com/item?id=42567484 might help you make up your mind.
leeoniya 2y ago
"Do not fall into the trap of anthropomorphising Larry Ellison." https://m.youtube.com/watch?t=33m1s&v=-zRN7XLCRhc&feature=youtu.be https://m.youtube.com/watch?t=33m1s&v=-zRN7XLCRhc&feature=yo...
avinassh 2y ago
Larry always makes appearances in his reviews, lol > But the real big news in 2023 was how Elon Musk personally helped reset Larry's Twitter password after he invested $1b in Musk's takeover of the social media company. And with this $1b password reset, we were graced in October 2023 with Larry's second-ever tweet and his first new one in over a decade. https://www.cs.cmu.edu/~pavlo/blog/2024/01/2023-databases-retrospective.html https://www.cs.cmu.edu/~pavlo/blog/2024/01/2023-databases-re... > These journalists made it sound like Larry was doing something nefarious or indecent, like the time he made his pregnant third wife sign a prenup two hours before their wedding. I can assure you that Larry was only trying to use his vast wealth as the 7th richest person in the world to help his country. His participation in this call is admirable and should be lauded. Free and fair elections are not a trivial affair, like a boat race where sometimes shenanigans are okay as long as you win. Larry has done other great things with his money that are overlooked, like spending $370m on anti-aging research so that he can live forever https://www.cs.cmu.edu/~pavlo/blog/2022/12/2022-databases-retrospective.html https://www.cs.cmu.edu/~pavlo/blog/2022/12/2022-databases-re...

samanthasu 2y ago

would love to see what Andy's take on GreptimeDB https://github.com/GreptimeTeam/greptimedb https://github.com/GreptimeTeam/greptimedb

leeoniya 2y ago
made same comment recently in https://news.ycombinator.com/item?id=42330055#42331927 https://news.ycombinator.com/item?id=42330055#42331927
- samanthasu 2y ago
  looking forward to the bonus content!
- DennisZ_89 2y ago
  Glad to see people start trying on GreptimeDB. We're committed to building a fully open-source and cost-effective, unified time series database for metrics, logs, and events. To this point, it may still be too small to be included in the year database summary. But hope we'll grow fast in 2025 :)

rozenmd 2y ago

I loved ottertune, it's a shame it died the way it did.

memhole 2y ago

Love the style! CMU making databases cool. Sorry to hear about OtterTune.

Beefin 2y ago

TL;DR SQL is king

m_ke 2y ago

Andy is a treasure, if only we had more professors like him

antirez 2y ago

Wow, the reasons why Redis commands API suck in Andy's video (linked in the post) are the weakest ever. It is possible to make a case against the Redis API (I would not agree of course but... it's totally legitimate), but you gotta have stronger arguments than those, particularly if you are a teacher of some kind. Especially: you need to be somewhat fluent in Redis and how developers use Redis in order to understand why so many people like it, and then elaborate what it's wrong about it (if you believe there is something wrong). The video shows a general feeling of "I don't really use / know this, but I don't like how NON-SQL it is".

nojito 2y ago
SQL is king and history has shown non-sql languages are not good which causes many non-sql DBMS's to adopt sql eventually.
- antirez 2y ago
  Many non-SQL DBs had query languages that were broken Javascript-ish versions of SQL. Of course, this is wrong, and people will eventually adopt SQL instead. But if your data model isn't anything like relational DBs, non-SQL makes a ton of sense. OP seems to miss exactly this, that the Redis query language is shaped on the Redis data model, that is basically alien to the relational model. The idea behind Redis data model is that "describe data" then "query those data in random ways" is conceptually nice but practically will not model very well many use cases. SQL databases plagued tech with performance issues for decades because of that. So Redis says instead: you need to put your data thinking about fundamental things like data structures and access times and the way you'll need those data back. And the API reflects this. You don't have to automatically agree with that. But you have to understand that, then provide your "I'm against" arguments. Especially if you are in front of young people listening to you.
  - im_down_w_otp 2y ago
    Agreed. Many noSQL-boom-era databases eventually bolted on a SQL-esque layer, but that was also because they were mostly also all targeting "enterprise database" use cases and customers who both expected that and whose use cases largely fit with it. So, there was a lot of pressure to conform to norms when the advantage of not doing so wasn't immediately self-evident. We have a database [1] and query language [2] that's tailored to storing & querying trace/telemetry data produced by different layers and components of cyber-physical systems for systems engineers to analyze, verify, and validate what a complex system is doing. It's not quite a traditional relational problem. It's not quite a traditional time series problem. It's not quite a traditional graph problem. Addressing the way that systems engineers think about their domain in an effective way required coming up with something different. Are there caveats and rough edges? Sure. But, they're a lot less pernicious and onerous than the alternative of trying to leverage a bunch of ill-fitting menageries of different solutions. Redis is fit-for-purpose. So, it makes sense that its query interface would also express that. [1] https://docs.auxon.io/modality/ https://docs.auxon.io/modality/ [2] https://docs.auxon.io/speqtr/ https://docs.auxon.io/speqtr/

codeulike 2y ago

Weird how SQL Server and its Azure variants gets no mention. It dominates in certain sectors. DBEngines ranks it third most popular overall https://db-engines.com/en/ranking https://db-engines.com/en/ranking

patja 2y ago
The fawning over Larry Ellison is also weird.
- cactusfrog 2y ago
  The joke is that his greed/ unwillingness to squeeze margins has made the entire database company ecosystem possible.
RadiozRadioz 2y ago
Lots of people deliberately avoid Microsoft technologies and their whole ecosystem. There's of course interesting stuff happening there, but not enough for those outside the ecosystem to care. It's more a cultural thing than anything else. HN for example largely leans away from MS. It's quite interesting how little overlap there is between the two worlds sometimes. Speaking as one of those people, it's just not my thing, so it's not on my radar at all. There's enough stuff happening outside MS to keep me busy forever.
teej 2y ago
Are people choosing SQL Server independently of the Microsoft ecosystem? My understating is that you typically use it because you’re forced to choose a MS product.
- rawgabbit 2y ago
  SQL Server is a terrific product. And I detest most things Microsoft.
  - breadwinner 2y ago

sigbottle 2y ago

These year in review posts are really neat, I liked the AI in review posts really well. Maybe algorithms review or TCS review or some specific math topic review next?

CT4u8798 2y ago

I love SQL. I'm not a full-time developer but always use SQL over other abstractions, which I find extremely confusing and way more complicated that plain SQL.

skeeter2020 2y ago
I'm now firmly into management but the one skill I use very regularly is SQL. By far the best investment I made in my entire career was a little bit of relational algebra, some casual study of DBMS internals and a lot of hands-on SQL. The quasi-standards have also made it the easiest transfer across specific DBs and their flavours over the years. PSA: Hi kids, here's a dinosaur with yet more free advice: put the tiniest bit of effort into SQL early on and watch the compound interest add up.
threeseed 2y ago
That's just because it is what you are comfortable with. Many developers will jump straight for ORMs when given the chance.
- Tostino 2y ago
  Which for certain types of applications ORMs absolutely have their use.
- dominicrose 2y ago
  Some ORMs have weird design issues. Eloquent for example allows you to pull relationships lazily on single objects, so if you're in a loop that'll create a lot of queries. So much for laziness! I'm OK with this ability, but the API shouldn't encourage it by making it trivial.

wahnfrieden 2y ago

2024 was also the year that Realm died

travisgriggs 2y ago

The funding section had me thinking “one of these is not like the others”. Both the amount and count of successive rounds.

ksec 2y ago

>Six years after MySQL v8 went GA, the team turned v9 out on the streets. ......Oracle is putting all its time and energy into its proprietary MySQL Heatwave service. Oracle actually released 9.1 already in 2024. [1] And expect another release this month, and every quarter. So I think MySQL continues to get some new features bug fix and support like it used to. Contrary to most people think it is all going to Heatwave. I just hope Vector will be open source later as official to MySQL rather than behind Heatwaves. [1] https://dev.mysql.com/doc/relnotes/mysql/9.1/en/news-9-1-0.html https://dev.mysql.com/doc/relnotes/mysql/9.1/en/news-9-1-0.h...

the_arun 2y ago

This person started with news on DB - reviewing all prominent DBs & finally ended talking about love of Larry Ellison. A perfect human in the days of LLMs. Amazing write up.

RedShift1 2y ago

I've been using plain postgres for over 5 years now, reading this I feel like I'm in the eye of a storm...

mebcitto 2y ago

A couple of spicy things: > OtterTune. Dana, Bohan, and I worked on this research project and startup for almost a decade. And now it is dead. I am disappointed at how a particular company treated us at the end, so they are forever banned from recruiting CMU-DB students. They know who they are and what they did. Ouch. > Lastly, I want to give a shout-out to ByteBase for their article Database Tools in 2024: A Year in Review. In previous years, they emailed me asking for permission to translate my end-of-year database articles into Chinese for their blog. This year, they could not wait for me to finish writing this one, so they jocked my flow and wrote their own off-brand article with the same title and premise. Also sounds like he's preparing a new company: > I hope to announce our next start-up soon (hint: it’s about databases).

spprashant 2y ago
Anyone know what company he may be talking about?
- C-programmer 2y ago
  Inspect element on https://web.archive.org/web/20240827031455/https://ottertune.com/ https://web.archive.org/web/20240827031455/https://ottertune... For more context: > I'm to sad to announce that @OtterTuneAI is officially dead. Our service is shutdown and we let everyone go today (1mo notice). I can't got into details of what happened but we got screwed over by a PE Postgres company on an acquisition offer. https://x.com/andy_pavlo/status/1801687420330770841 https://x.com/andy_pavlo/status/1801687420330770841
  - yencabulator 2y ago
    Because that was a little too subtle and I was sufficiently curious: view-source:https://web.archive.org/web/20240827031455/https://ottertune.com/ https://web.archive.org/web/20240827031455/https://ottertune... scroll until you see ASCII art
    - swyx 2y ago

mrtimo 2y ago

Enjoyed his roundup in the "Shoving Ducks into Everything" section. DuckDB is a great tool. In April 2020, the creator of DuckDB gave a talk at CMU. In the beginning he makes a convincing argument (in 5 minutes) why data scientists don't use RDBMS and how this was the genesis of DuckDB. Here is a video that starts 3 minutes into the talk (where is argument starts): https://youtu.be/PFUZlNQIndo?si=ql9n2QuBlAEuGIqo&t=204 https://youtu.be/PFUZlNQIndo?si=ql9n2QuBlAEuGIqo&t=204

dig1 2y ago

> There was no major effort to fork off MongoDB, Neo4j, Kafka, or CockroachDB when they announced their license changes. AFAIK people didn't take MongoDB seriously from the start, especially with the "web scale database" joke circulating. The Neo4j Community version has been under GPLv3 for quite some time, while the Enterprise version has always been somewhat closed, regardless of whether the source code was available on GitHub (the mentioned license change affected the Enterprise version). Regarding CockroachDB, I must admit that I've only heard about it on HN and don't know anyone who seriously uses it. As for Kafka, there are two versions: Apache Kafka, the open-source version that almost everyone uses (under the Apache license), and Confluent Kafka, which is Apache Kafka enhanced with many additional features from Confluent, and the license change affected Confluent Kafka. In short, maybe the majority simply didn't care about these projects very much, so there is no major fork. > It cannot be because the Redis and Elasticsearch install base is so much larger than these other systems, and therefore, there were more people upset by the change since the number of MongoDB and Kafka installations was equally as large when they switched their licenses. I can’t speak for MongoDB, but the Confluent Kafka install base is significantly smaller than that of Apache Kafka, Redis and ES. > Dana, Bohan, and I worked on this research project and startup for almost a decade. And now it is dead. I am disappointed at how a particular company treated us at the end, so they are forever banned from recruiting CMU-DB students. They know who they are and what they did. Call me a skeptic, but I can't see this as a fair approach. If your company fails for whatever reasons, you should not recruit the university department/group/students against your peers (I can't find that CMU-DB was one of the founders of Ottertune). Wrt Andy, here are [1] somehow interesting views from (presumably) previous employees. [1] https://www.reddit.com/r/Database/comments/1dgaazw/comment/ldw90qj/ https://www.reddit.com/r/Database/comments/1dgaazw/comment/l...

paulddraper 2y ago
There are production uses of MongoDB (Stripe comes to mind). But it is certainly not a popular choice there.
- threeseed 2y ago
  MongoDB does over $2b in revenue (and growing by 20%) each year. There are a lot of production uses.

maeil 2y ago

Good read! > Postgres' support for extensions and plugins is impressive. One of the original design goals of Postgres from the 1980s was to be extensible. The intention was to easily support new access methods and new data types and operations on those data types (i.e., object-relational). Since 2006, Postgres' "hook" API. Our research shows that Postgres has the most expansive and diverse extension ecosystem compared to every other DBMS. Greenhorn developers don't even know that there are non-Postgres databases which have extensions too - such is the gap! I wouldn't be surprised if Postgres had as many as all others combined.

softwaredoug 2y ago

On the “Amazon can just offer your DB as a service” Yes this can happen. But a lot of people don’t want a AWS managed service. They're like 30% cheaper for 30% less value. They can develop a bad reputation and feel like weird forks (kinesis vs Kafka) that have weird undocumented gotchas and edge cases that never get fixed. Many teams want to host on k8s anyway, and you’ll probably have better k8s support from the main project. Another example is the success of Flink over hosted Google Dataflow. Seems eventually the teams I know trend to the most mainstream OSS implementation over time, maybe after early prototyping on a managed system. IMO it might not be the highest growth market anymore. Those who want to pay for a managed service will. But many are just figuring out a k8s based solution to their infra needs as k8s knowledge becomes more ubiquitous.

senderista 2y ago
Kinesis is not Kafka (at all), nor is Google Dataflow Flink (perhaps you're thinking of Apache Beam?).

wslh 2y ago

Great heads up. I wonder about graph databases. He mentioned <https://umbra-db.com/ https://umbra-db.com/> and <https://cedardb.com/ https://cedardb.com/> both include the graph use case and I wonder how they compare to <https://neo4j.com/ https://neo4j.com/>.

Tanjreeve 2y ago
Umbra and cedar are both still relational databases. Afaik the jury is now out on if graph databases are better than modern relational databases for most graph queries especially ones with good query planners/compilers. The only time graph DBs seem to be consistently better is for very specialist many degrees traversals for small amounts of data.
- wslh 2y ago
  My point is that both Umbra and Cedar mentioned graph support, and I don't believe this was a coincidence. Umbra highlights: "Groupjoins enable efficient computation of aggregates, worst-case optimal joins handle complex queries on graph structured data, and range joins efficiently evaluate queries with conditions on location or time intervals." while Cedar includes in the hero: "CedarDB is a relational-first database system that delivers best-in-class performance for all your workloads, from transactional to analytical to graph,..."
  - Tanjreeve 2y ago
    I think that phrase of "relational first" explains it best. Anything that supports joins can support a graph. Whether it'll perform satisfactorily is going to depend on the compiler and use case/size.

atombender 2y ago

The article mentions Greenplum, but it's worth noting that when the code was closed, several of the original developers created an open-source fork, Cloudberry, which seems to be thriving. Cloudberry was accepted into the Apache project this year, and has synced with Postgres 14, whereas the closed-source Greenplum is still stuck on Postgres 12. The architecture is quite ancient at this point, but I'm not sure it's completely outdated. It's single-master shared-nothing, with shards distributed among replicas, similar to Citus. But the GPORCA query planner is probably the most advanced distributed query planner in the open source world at this point. From what I know, Greenplum/Cloudberry can be significantly faster than Citus thanks to the planner being smarter about splitting the work across shards.

ledgerdev 2y ago
Thanks for the cloudberry mention, wasn't aware of it.
Icathian 2y ago
This is mostly correct, but it's worth mentioning that cloudberry substantially predates Greenplum going closed source. It just got quite a boost from that change happening. Different dev team too, afaik none of the original Greenplum team was involved with Cloudberry until very recently. Also, Greenplum 7 tracks postgres 14. Which is still old at this point, but not so bad as 12.... I also don't think I'd call the architecture ancient. Just very tightly coupled to postgres' own (as a fork of postgres that tries to ingest new versions from upstream every year or two) and paying the overhead of that choice in the modern landscape. Source: former member of the Greenplum Kernel team.
- atombender 2y ago
  Thanks for the context. In what way would you say Cloudberry lags behind Greenplum technology-wise? I see newer Greenplum versions have a lot of planner improvements. Greenplum 7 is listed as tracking Postgres 12 in the release announcement [1], and the release notes for later 7.x versions don't mention anything. Is there a newer release with higher compatibility? When I say ancient, I mean that it's a "classical" shared-nothing design where the database is partitioned and hosted as parallel, self-contained replica servers, where each node runs as a shard that could, in theory, by queried independently of the master database. This is in contrast to newer architectures where data is sharded at the heap level (e.g. Yugabyte, CockroachDB) and/or compute is separated from data (e.g. Aurora, ClickHouse, Neon, TiDB). [1] https://greenplum.org/partition-in-greenplum-7-whats-new/ https://greenplum.org/partition-in-greenplum-7-whats-new/

PeterZaitsev 2y ago

I think one thing Andy misses about why people were pissed about Elastic and Redis but not as many for MongoDB and some other is their license and size of Contributors Community. When original license is as restricted as AGPL it is unlikely there is much of embedded use... so less people are impacted in truly catastrophic way Also if there is no contributor community to speak of... who is going to do the fork ? I put some thoughts about it in my post about ScyllaDB https://peterzaitsev.com/thoughts-on-scylladb-license-change/ https://peterzaitsev.com/thoughts-on-scylladb-license-change...

tayo42 2y ago
> In the case of Redis, I can only think that people perceive Redis Ltd. as unfairly profiting off others' work since the company's founders were not the system's original creators. An analysis of Redis' source code repository also shows that a sizable percentage of contributions to the DBMS comes from outside the company He mentions this in "Andy’s Take" section btw
- PeterZaitsev 2y ago
  Yes. Not the license though.
rockwotj 2y ago
RE ScyllaDB, there will absolutely be no fork and it’s very unlikely they have ever had a meaningful contribution to Scylla OSS (which is not changing, just going to bit rot and the enterprise version which was closed source was moved to source available). The reasoning being the bar is very high for contributions. It’s C++ 20/23 without virtual memory and a userland cooperative thread per core scheduler (this is the underlying seastar framework). The skills needed to do anything meaningful here is very high and I don’t see the motivation behind it as I would expect people having feature requests would be customers, have an easier time extending Cassandra or the feature would be available in the Enterprise version.
- menaerus 2y ago

kwillets 2y ago

I spent the past year puzzling over the DB market as well, but I don't feel like I'm much closer to understanding it. It appears that a lot of attention is now directed at the folks doing 100 MB queries, and the high end has moved past everybody's radar. My idea of an exciting product is Ocient, who have skipped over Cloud and gone for hyperscale on-prem hardware. Yellowbrick is also a contender here. I have a lot of experience with Vertica, and they seem to have gotten stuck in this niche as well, with sales tilted towards big accounts, but less traction in smaller shops, and a difficult road to get a SaaS or similar easy-start offering. There's a crossover point where self-managed is cheaper than cloud, but nobody seems to have any idea where it is. Snowflake will gladly tell you that your sub-$1M Vertica cluster should be replaced by $10M of sluggish SaaS, and that you are saving money by doing so. These decisions seem more in the realm of psychology or political science. DHH's cloud exit was a refreshing take on the expense issue, even if it wasn't strictly in the database space -- the cost per VCPU and so forth that he documented is a good start for estimating savings, and he debunked a lot of the "hidden costs" that cloud maximalists claim. In the business/financial space the biggest news to me was the correction in Snowflake's stock price, which seemed to indicate that investors were finally noticing metrics like price-performance, but they added a little more AI and went back into irrationality. I'm heavily in favor of DuckDB, Hudi, Iceberg, S3 tables, and the like. Mixing high-end and low-end tools seems like the best strategy (although settling on one high-end DWH has also worked IME), and the low end is getting better and cheaper, squeezing out the mid-range SaaS vendors. In research I found Goetz Graefe's work in offset-value coding exciting -- he's wired it into query operators in a way that saves a lot of CPU on sorting and joins/aggregation. This is a technique that I've applied favorably in string sorting, and it was discovered in the DB community decades ago but largely forgotten. (This work precedes 2024, but I'm a slow study.)

downsplat 2y ago
> There's a crossover point where self-managed is cheaper than cloud Single data point here: before cloud managed dbs were a thing our smallish startup was running mysql on virtual servers by installing it from the linux package manager. Always worked great, runs without needing manual attention for years at a time once set up, so I've never felt the need to change. So at least in some cases the crossover point is "right from the start".

bionhoward 2y ago

Redis is slow?

kermatt 2y ago
I wish there was more context around that statement in his post. Redis while not having some of the features he mentions in [1] (i.e. SQL), when used for what it excels at is usually not considered "slow". As an in-memory data structure server, a common use case is to use it for where some operations in a typical RDBMS are slow. [1] https://youtu.be/fZbwD1gzjLk?t=2018 https://youtu.be/fZbwD1gzjLk?t=2018
senderista 2y ago
Yes. 1. It is single-threaded, which severely limits throughput for a single instance. 2. All communication must go over a socket, which severely impacts latency for use cases where it could otherwise run in-process.
- breadwinner 2y ago
  What is the better alternative?
  - bdangubic 2y ago
    depends on your use case. also, for most redis is fast enough which is why it is wildly popular
  - yla92 2y ago
    Do checkout https://www.dragonflydb.io https://www.dragonflydb.io . It's also mentioned in the review!
    - CRConrad 2y ago
      I'm so tired of these Web pages that look like iPhone screens.
2y ago

ak_111 2y ago

Wow his database startup that raised 12M died this year after only three years. If anything this shows how insanely difficult it must be to succeed as a database startup (when was the most recent startup success in this space?), as the founding team is stellar. On the other hand I am surprised it died this quick and interested to know if they did a proper postmortem. Not only did they raise way more than is needed to survive for three years but the idea is about utilising AI to improve DB performance and I find it hard to imagine they couldn't find more investors to lend them a lifeline with all the AI hype.

Tanjreeve 2y ago
No idea about internal workings but as a "DB optimisation" startup you're competing with - most people don't need it - People who do need it having DBAs/Operations people - or consultancies - Database vendors that have automatic optimisation as a feature Ok "AI" in the name but I think for something as specific as DB optimisation AI jazz hands probably don't work as well. Writing it out it almost seems harder than being an actual DB vendor.
didgetmaster 2y ago
I thought the same thing. I have a personal project that started out as a file system replacement (object store), but also does some amazing DB operations and is useful for analytics. I have often thought about trying to turn it into a profitable business; but the thought of needing to raise insane amounts of capital just to get a few years of runway, seems daunting.
hipadev23 2y ago
Andy's ego is far too big to actually run a company and his focus is entirely on vanity metrics. You can see it in these database reviews where he spends substantial time pining over how much money CompanyX raised and shitting on the bread and butter tools used by those of us who don't have VC money to blow on whatever DBaaS is the current hype.
cmrdporcupine 2y ago
I mean it's also arguably the most difficult time for startups (in general) in the last 15-20 years. Esp if you got your initial investment and valuation before the end of ZIRP.

lvl155 2y ago

I highly recommend their Youtube series on databases. They have great guest speakers.

syspec 2y ago
Link?
- nojito 2y ago
  https://www.youtube.com/c/CMUDatabaseGroup https://www.youtube.com/c/CMUDatabaseGroup

deleted 2y ago

[deleted]

based2 2y ago

https://dbos-project.github.io https://dbos-project.github.io news

refset 2y ago
More specifically, DBOS Inc. raised a $8.5 million seed round [0] and is backed by Michael Stonebraker (the creator of Postgres). I initially assumed Andy was alluding to this when he wrote "the most famous database octogenarian splashing cash" :) [0] https://techcrunch.com/2024/03/12/new-startup-from-postgres-creator-puts-the-database-at-heart-of-software-stack/ https://techcrunch.com/2024/03/12/new-startup-from-postgres-...

roark_howard 2y ago

DuckDB dominating over DataFusion could fuel the ongoing language war with a great half baked argument!

yencabulator 2y ago
Every time I've tried to use DuckDB it has segfaulted on me, so yeah I'm betting on DataFusion..
- 1egg0myegg0 2y ago
  That is very surprising to hear! If you can reproduce it, could you please file a bug report on GitHub? That would be a huge help!!
  - riku_iki 2y ago
    I personally filed bunch of bugs, they mostly were autoclosed because of no activity after 3 months. This is very discouraging, and I rather invest effort in looking for workarounds in the future.

polishdude20 2y ago

After I interviewing at OtterTune a while back and being bombarded with multiple rounds of leetcode questions, I somehow knew OtterTune wouldn't make it

gigatexal 2y ago

This take screams more than a technical criticism but of something personal. “I'll be blunt: I don't care for Redis. It is slow, it has fake transactions, and its query syntax is a freakshow. Our experiments at CMU found Dragonfly to have much more impressive performance numbers (even with a single CPU core). In my database course, I use the Redis query language as an example of what not to do.” (From the article) Of course it’s not to be used as a general purpose DB it’s keys and values. Used for caches and things like that. In my experience in real world scenarios and loads vanilla single threaded Redis is stable, fast, and nigh bulletproof.

deleted 2y ago
[deleted]

rednafi 2y ago

Loved the overview. Hated the shade toward Redis. Redis has arguably the best key-value query syntax, and there’s a reason so many people swear by it. True, the decision-makers at Redis Ltd are absolute pieces of trash, but Redis itself is a delightful piece of engineering artifact. I don’t care about the billion-dollar drama behind a piece of tech, but Redis defined the key-value query API for many similar databases. Trashing it just because it isn’t SQL-like feels unjustified.

quotemstr 2y ago

There's one QOL extension that I haven't seen anyone else implement: dimensional analysis. I can declare a column is an integer. Why not an integer that expresses feet? Why shouldn't I be able to write SELECT 1inch + 1cm and get a correctly computed length? Why can't the query parser help me avoid nonsense like SELECT 1kg + 1hr? All this stuff is pretty straightforward to add and would help avoid avoidable mistakes.

zhousun 2y ago

It's such an honor our https://pgmooncake.com/ https://pgmooncake.com/ is covered in the review! A little sad Andy didn't share more of his thoughts on the intersection between Data and AI, and how that's going to evolve.

Upvoter33 2y ago

Pretty funny. One factual issue: "The university had previously announced that this player was transferring from Louisiana State to Michigan." This is not true. Underwood had committed to LSU but then switched his commitment to Michigan. He was still in high school at the time, and has never attended LSU. But, do you really expect a funny database prof to know much about football?

mpbart 2y ago
I never thought I’d see a discussion about the Underwood NIL drama on a databases blog post but here we are.

funkysapien 2y ago

[dead]

bcoates 2y ago

"I've never met anybody that used Alteryx" I have! It's a pretty good no-code/minimal-code graphical ELT+Analytics in one tool. It's one of those alternate-universe tools that has it's own way of doing things from everything else in the industry, but it’s pragmatic and the people who use it tend to love it. The one thing that makes it viable is that is has/had (pre-acquisition) very aggressive compatibility with anything else that can hold data, so you can use it as a bolt-on to whatever other databases or files your company has. Despite what the PE press release about the acquisition says, it has virtually nothing to do with AI, at lease in the modern big NN sense. If you're looking to fix your giant pile of alteryx workbooks or migrate them to something else, hmu

osigurdson 2y ago

More like: "Database license drama - a year in review".

pbrunoster 2y ago

oracle, sqlserver , db2 , informix , teradata for example, does not exist ... ok

pbrunoster 2y ago

oracle, sqlserver , db2 , informix , teradata for example, does not exist ... ok ....

nwatson 2y ago

Interesting seeing the death of blockchain-based AWS QLDB mentioned. I worked at a company for a while that used QLDB as the primary system of record. The idea is great but the problem is that due to performance and other QLDB limitations all data had to be mirrored to an RDBMS via a streaming/queuing system, and there always were programmatic errors in interpreting data arriving for import into the RDBMS ... text field too long for RDBMS field; wrong data type or overflowing integer; invalid text encoding; ... Etc. These errors had to be noticed, debugged, fixed, and data had to be re-streamed. In the meantime official transactions were missing from the RDBMS side, which was used for reporting, driving the UI, deriving monetary obligations, etc. it was not worth the trouble. (I was lucky to not be involved in that design or implementation.)

kopirgan 2y ago

Wow so much to learn reading this . Thanks

hdesh 2y ago

> The upcoming year is going to be the test of strength for many database startups. Nobody wants to be the next MariaDB Corporation, The link for "MariaDB corporation" points to an empty image with white colour background. Can anyone explain the context here?

dandan7 2y ago

Thanks for the article. On the topic of redis: 3 executives from redis built FalkorDB (succeeded redisgraph) raising 3m to build a graphdb for better rag (ref:https://github.com/FalkorDB/GraphRAG-SDK https://github.com/FalkorDB/GraphRAG-SDK)

uncomplexity_ 2y ago

thank you for your work andy! for a moment i got reminded of the rap music in your courses im glad that tigerbeetle got here, really impressive team they have. there are a lot of other missing alien technologies i've discovered recently too like quickwit which is like elasticsearch but s3-compatible, and typesense which is like elasticsearch but memory-based

swyx 2y ago

> I need to figure out to juice my stats because in September 2024, Wikipedia removed the article about me over not having enough citations. guys, what are we doing here. this is ridiculous. andy pavlo cannot get an article on wikipedia? have you seen his work?

phartenfeller 2y ago

There is a lot about Larry Ellison, but not a single word about this year's rather big Oracle release (23ai)?

deleted 2y ago

[deleted]

christinewarr 2y ago

[dead]

christinewarr 2y ago

[dead]

tiffanyava 2y ago

[dead]

Lavigne3746 2y ago

[dead]