4 ms·
This is really surprising for a commercial company. Not about the choice of language itself, but the fact that they prioritize such rewrites over features, simi
by d3w4s9 3y ago
This is really surprising for a commercial company. Not about the choice of language itself, but the fact that they prioritize such rewrites over features, similar to the concerns from other commenters. They mention "performance" and "garbage collector" and "error handling" which are almost technical details.
A company can make good money for quite a while as long as the performance is "good enough", and usually focuses on adding features instead of worrying about any rewriting/re-architect until there is a bottleneck or issues start to significantly slow down development. How "successful" such a rewrite still needs to be seen, but this is risky for most companies/products.
- voiper1 3y agoThe first comment is from the Cofounder and he lists the _features_ that motivated a rewrite: >Then there's the question of why we did a rewrite at all. We wanted to get at some important requirements: >Unlimited cardinality >Analytics queries against time series at the performance of a columnar DB >Use object store as the durability layer for historical data (i.e. separate compute from storage) >SQL and broader ecosystem compatibility >All of that stuff taken together meant that we'd be rewriting most of the core of the database. ...
- d3w4s9 3y agoSure, the write definitely could have helped releasing these new features, but the language transition itself takes non-trivial effort and could have been used for adding additional features instead. I haven't seen any evidence that such a language transition is desperately needed. Also I said deprioritize adding features, not stop adding features, mind you. The core question here is what kind of benefit we are seeing by changing the language. If you are a fan of Rust and want to rewrite all your just because of that and you are the CTO, whatever, what can I say. But the title here is a bit clickbait-y and I really don't see much reflection on the languages themselves and how to balance the business needs for such a transition.
- pauldix 3y agoIn this case, the features we kept getting asked for by our customers necessitated a change in underlying database architecture. I talk about that quite a bit in the reddit thread. I totally agree that a rewrite is risky. It's not something I'd choose to do again, but at the time we didn't really see any way around rewriting the bulk of the database (even if we kept it implemented in Go). Using Rust and the Arrow ecosystem of projects (Parquet, DataFusion, Flight) meant that there were a ton of things we didn't have to do from scratch. One of our staff engineers, Andrew Lamb, has called it a toolkit for building databases. Thanks in part to his contributions, I think he's right.
- say_it_as_it_is 3y agoWhat is meant by separating compute from storage? This keeps being mentioned as if it were some new paradigm shift so I assume there's a non-obvious situation.
- pauldix 3y agoA traditional monolithic database assumes that you have locally attached storage. All of your ingest, indexing and query processing happens on the machine with that storage (i.e. your compute and storage live together). The cloud kind of complicated things with EBS and high IOPS network storage, but generally, those work kind of the same way. The volume is mounted on a single machine that uses it. When people talk about separating compute from storage, they mean pulling compute heavy tasks like query, ingest, indexing, and compaction apart and using a shared storage tier that many systems can talk to. Usually this is object storage paired with some sort of catalog (kept in either object storage or some other store or API). Snowflake popularized this approach in the data warehousing and OLAP space with great success. Their papers submitted to VLDB are great reads on the topoic.