4 ms·
> Postgres and Apache Iceberg are both mature systems Apache Iceberg as mature? I mean, there's a lot of activity around it, but I remember a year ago the rust
by ajd555 1y ago
> Postgres and Apache Iceberg are both mature systems
Apache Iceberg as mature? I mean, there's a lot of activity around it, but I remember a year ago the rust library didn't even have write capabilities. And it's not like the library is a client and there's an iceberg server - the library literally is the whole product, interacting with the files in s3
- ajd555 1y agoI suppose, in fairness, the Java library has been around for much longer
- _ea1k 1y agoA lot of people will spend dozens of hours and tens of thousands of their company's money to avoid learning Java. I'm not even sure if I'm joking. :)
- pat2man 1y agoI mean RisingWave, the solution mentioned in the article, is a complete startup rewriting things in Rust mostly to avoid the larger Java solutions like Flink and Spark...
- datadrivenangel 1y agoFlink and Spark are painful for streaming. They do work, but they have some sharp and rough edges.
- fifilura 1y agoI think 90% of the use cases for streaming data is because managers have nothing better to do than to reload their profits dashboard. They are rarely supported by real business value. Save the data as soon as it comes in and transform it in batch mode.
- kwillets 1y agoThis is data engineering, where people spend thousands of dollars of their company's money to avoid learning SQL. The place with no Java is across the street (old Soviet joke, originally for meat/fish stores).
- icedchai 1y agoSad but true. Or they learn "something" about SQL but not about indexes, data types, joins, or even aggregate functions. I've seen some python horror shows that would select * entire tables into lists of dicts, only to do the equivalent of a where clause and a couple of sums.
- _ea1k 1y agoOuch. I was really hoping this wasn't so common, but I guess it is. I'm sure they heard somewhere that "joins are expensive" or something along those lines, so they solved it!
- solid_fuel 1y agoPrisma built a whole ORM around that, complete with the client-side joins.
- icedchai 1y agoThere are also other "lets pretend we have big data" patterns I've seen. Running something like AWS Athena or Hadoop over and over, for less than a couple gigs of data that would fit into memory on a laptop from 15 years ago.
- joseda-hg 1y agoClassic, I remember seeing something like this SELECT * FROM X, into a C# list, Filter it with LINQ, and then use C# to do calculations "Why not EF?" "Someone told me it was slower"
- nxm 1y agoMost tools the data ecosystem support Iceberg as first class citizen, including major analytical query engines. Lots of development in this space has happened over the past year in particular.