4 ms·

I was curious how long it would take with Polars (for scale), apparently 33s: https://github.com/Butch78/1BillionRowChallenge/tree/main https://github.com/Butch

I was curious how long it would take with Polars (for scale), apparently 33s: https://github.com/Butch78/1BillionRowChallenge/tree/main https://github.com/Butch78/1BillionRowChallenge/tree/main I’m kind of interested in the opposite problem, what is the simplest solution using a well known library/db that approaches the fastest hand optimized solution to this problem?

sharno 3y ago
That’s the question worth asking imo. I was wondering how fast is the idiomatic Java solution
- lucianbr 3y ago
  71 seconds https://questdb.io/blog/billion-row-challenge-step-by-step/ https://questdb.io/blog/billion-row-challenge-step-by-step/
  - kaba0 3y ago
    Is that the same hardware? Otherwise it doesn’t say much.
- KingOfCoders 3y ago
  Java is often slightly faster than Go, has similar (perhaps, older, better optimized Map) constructs, perhaps better GC (older, more optimized), though I don't think the GC is a challenge, has slower startup times - so I'd say roughly the same as the idiomatic Go version?
  - kaba0 3y ago
    Java’s GC is just incomparably better, literally every research on the topic is written for Java. Though that’s true that without value types, Java does rely slightly more on it.
    - geodel 3y ago
      It could be matter of fact comment if incomparably better and slight reliance are brought to the middle.
  - Xeoncross 3y ago
    Java can be faster than Go, but it comes at the same cost as most "faster" things in software: Java uses significantly more memory. Java is also more mature, which means you are entering a massive package-bloat setup that has evolved over the years to work for everyones wild and varied needs. By the time you have your database, cache, http/other handlers, tests, fixtures, metrics, logging, tracing, etc... setup you're looking at a scary pile of dependencies spanning thousands of classes that would make even NPM jealous.
- benhoyt 3y ago
  On my machine the simple/idiomatic "baseline" Java solution (https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_baseline.java https://github.com/gunnarmorling/1brc/blob/main/src/main/jav...) takes 2m0s, compared to 1m45s for my simple Go version.
gigatexal 3y ago
Where’s the source data I’d like to attempt ingesting this and processing it with DuckDb.
- nhinck3 3y ago
  In the original 1BRC, it's a python script that generates the data.
- schu 3y ago
  Instructions on how to create it can be found here: https://github.com/gunnarmorling/1brc?tab=readme-ov-file#running-the-challenge https://github.com/gunnarmorling/1brc?tab=readme-ov-file#run... The Python version: https://github.com/gunnarmorling/1brc/blob/main/src/main/python/create_measurements.py https://github.com/gunnarmorling/1brc/blob/main/src/main/pyt...
- llimllib 3y ago
  Here’s a thread on results with duckdb, I don’t mean to discourage you taking a shot at all though: https://github.com/gunnarmorling/1brc/discussions/39 https://github.com/gunnarmorling/1brc/discussions/39
geysersam 3y ago
Sounds very reasonable. In the blog post about 20s were shaved off by assuming we don't need complicated string parsing. An of the shelf library can't make that assumption so they will always have to pay the extra cost.
- 392 3y ago
  True in general but some (especially libs aimed at larger datasets processed in batch) are taking advantage of benchmarks like this to do things like: Try the fast way, if it works great Try the slow way, if above fails This makes the slow path 2x slower at worst (and you can advise to always use the slow way with optional params) but the fast path can be 10x faster
jsmith99 3y ago
I’m surprised at the poor performance of python here. For reference there are several very brief R examples which are just 2-3 seconds. Eg http://blog.schochastics.net/posts/2024-01-08_one-billion-rows-challenge/ http://blog.schochastics.net/posts/2024-01-08_one-billion-ro...
- cmdlineluser 3y ago
  Are you talking about the 2nd table in the Benchmark section? It seems they are not running against the full dataset: > Moving on to the 100 million file to see if size makes a difference. ggplot2::autoplot(reorderMicrobenchmarkResults(bench1e8)) One would also have to run both approaches on the same hardware for a meaningful comparison?

sharno 3y ago

That’s the question worth asking imo. I was wondering how fast is the idiomatic Java solution

lucianbr 3y ago
71 seconds https://questdb.io/blog/billion-row-challenge-step-by-step/ https://questdb.io/blog/billion-row-challenge-step-by-step/
- kaba0 3y ago
  Is that the same hardware? Otherwise it doesn’t say much.
KingOfCoders 3y ago
Java is often slightly faster than Go, has similar (perhaps, older, better optimized Map) constructs, perhaps better GC (older, more optimized), though I don't think the GC is a challenge, has slower startup times - so I'd say roughly the same as the idiomatic Go version?
- kaba0 3y ago
  Java’s GC is just incomparably better, literally every research on the topic is written for Java. Though that’s true that without value types, Java does rely slightly more on it.
  - geodel 3y ago
    It could be matter of fact comment if incomparably better and slight reliance are brought to the middle.
- Xeoncross 3y ago
  Java can be faster than Go, but it comes at the same cost as most "faster" things in software: Java uses significantly more memory. Java is also more mature, which means you are entering a massive package-bloat setup that has evolved over the years to work for everyones wild and varied needs. By the time you have your database, cache, http/other handlers, tests, fixtures, metrics, logging, tracing, etc... setup you're looking at a scary pile of dependencies spanning thousands of classes that would make even NPM jealous.
benhoyt 3y ago
On my machine the simple/idiomatic "baseline" Java solution (https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_baseline.java https://github.com/gunnarmorling/1brc/blob/main/src/main/jav...) takes 2m0s, compared to 1m45s for my simple Go version.

gigatexal 3y ago

Where’s the source data I’d like to attempt ingesting this and processing it with DuckDb.

nhinck3 3y ago
In the original 1BRC, it's a python script that generates the data.
schu 3y ago
Instructions on how to create it can be found here: https://github.com/gunnarmorling/1brc?tab=readme-ov-file#running-the-challenge https://github.com/gunnarmorling/1brc?tab=readme-ov-file#run... The Python version: https://github.com/gunnarmorling/1brc/blob/main/src/main/python/create_measurements.py https://github.com/gunnarmorling/1brc/blob/main/src/main/pyt...
llimllib 3y ago
Here’s a thread on results with duckdb, I don’t mean to discourage you taking a shot at all though: https://github.com/gunnarmorling/1brc/discussions/39 https://github.com/gunnarmorling/1brc/discussions/39

geysersam 3y ago

Sounds very reasonable. In the blog post about 20s were shaved off by assuming we don't need complicated string parsing. An of the shelf library can't make that assumption so they will always have to pay the extra cost.

392 3y ago
True in general but some (especially libs aimed at larger datasets processed in batch) are taking advantage of benchmarks like this to do things like: Try the fast way, if it works great Try the slow way, if above fails This makes the slow path 2x slower at worst (and you can advise to always use the slow way with optional params) but the fast path can be 10x faster

jsmith99 3y ago

I’m surprised at the poor performance of python here. For reference there are several very brief R examples which are just 2-3 seconds. Eg http://blog.schochastics.net/posts/2024-01-08_one-billion-rows-challenge/ http://blog.schochastics.net/posts/2024-01-08_one-billion-ro...

cmdlineluser 3y ago
Are you talking about the 2nd table in the Benchmark section? It seems they are not running against the full dataset: > Moving on to the 100 million file to see if size makes a difference. ggplot2::autoplot(reorderMicrobenchmarkResults(bench1e8)) One would also have to run both approaches on the same hardware for a meaningful comparison?