4 ms·

This is a staggering ~3x improvement in just under 2 years since Sep was introduced June, 2023. You can't claim this when you also do a huge hardware jump

WD-42 1y ago
Yea wtf is that chart, it literally skips 4 cpu generations where it shows “massive performance gain”. Straight to the trash with this post.
- g-mork 1y ago
  It also appears to be reporting whole-CPU vs. single thread, 1.3 GB/sec is not impressive for single thread perf
  - iamleppert 1y ago
    Agreed. How hard is it to keep hardware fixed, load the data into memory, and use a single core for your benchmarks? When I see a chart like that I think, "What else are they hiding?" Folks should check out https://github.com/dathere/qsv https://github.com/dathere/qsv if they need an actually fast CSV parser.
  - Remnant44 1y ago
    I mean... A single 9950x core is going to struggle to do more than 16 GB/second of direct mem copy bandwidth. So being within an order of magnitude of that seems reasonable
- ziml77 1y ago
  But it repeats the 0.9.0 test on the new hardware. So the first big jump is a hardware change, but the second jump is the software changes.
- matja 1y ago
  4 generations? 5950x is Zen 3 9950x is Zen 5
  - chupasaurus 1y ago
    Sine Zen 2 (3000) the mobile CPUs are up by a thousand respectively to their desktop counterparts. edit: Or Nx2000 where N is from Zen N.
  - hinkley 1y ago
    And even with 2, CPU generations aren't what they used to be back when a candy bar cost less than a dollar.
freeone3000 1y ago
They claim a 3GB/s improvement versus previous version of sep on equal hardware — and unlike “marketing” benchmarks, include the actual speed achieved and the hardware used.
- stabbles 1y ago
  Do note that this speed even before the 3GB/s improvement exceeds the bandwidth of most disks, so the bottleneck is loading data in memory. I don't know of many applications where CSV is produced and consumed in memory, so I wonder what the use is.
  - freeone3000 1y ago
    Slower than network! In-memory processing of OLAP tables, streaming splitters, large data set division… but also the faster the parser, the less time you spend parsing and the more you spend doing actual work
    - tetha 1y ago
      This is honestly something that caught me off-guard a bit. If you have good internal network connectivity, small queries and your relational database has the data in memory, it can be faster to fetch data from the DB via the network than reading it from disk. Like, sure, I can give you an application server with faster disks and more memory and you or me are certainly capable of implementing an application server that could load the data from disk faster than all of that. And then we build caching to keep the hot data in memory, because that's faster. But then we've spent very advanced development resources to build a relational database with some application code at the edge. This can make sense in some high frequency trading situations, but in many more mundane web-backends, a chunky database and someone capable of optimizing stupid queries enable and simplify the work of a much bigger number of developers.
      - bee_rider 1y ago
        You can also get this with Infiniband, although it is less surprising, and basically what you’d expect to see. I did once use a system where the network bandwidth was in the same ballpark as the memory bandwidth, which might not be surprising for some of the real HPC-heads here but it surprised me!
jbverschoor 1y ago
They also included 0.9.0 vs 0.10.0. on the new hardware. (21385 vs 18203), so the jump because of software is 17%. Then if we take 0.9.0 on previous hardware (13088) and add the 17%, it's 15375. Version 0.1.0 was 7335. So... 15375/7335 -> a staggering 2.1x improvement in just under 2 years
perching_aix 1y ago
> You can't claim this when you also do a huge hardware jump Well, they did. Personally, I find it an interesting way of looking at it, it's a lens for the "real performance" one could get using this software year over year. (Not saying it isn't a misleading or fallacious claim though.)
bawolff 1y ago
Perhaps, but i think we are well past the moore's law era where a 3x speed up is to be expected just from hardware. Its still a pretty impressive feat in the modern era.

WD-42 1y ago

Yea wtf is that chart, it literally skips 4 cpu generations where it shows “massive performance gain”. Straight to the trash with this post.

g-mork 1y ago
It also appears to be reporting whole-CPU vs. single thread, 1.3 GB/sec is not impressive for single thread perf
- iamleppert 1y ago
  Agreed. How hard is it to keep hardware fixed, load the data into memory, and use a single core for your benchmarks? When I see a chart like that I think, "What else are they hiding?" Folks should check out https://github.com/dathere/qsv https://github.com/dathere/qsv if they need an actually fast CSV parser.
- Remnant44 1y ago
  I mean... A single 9950x core is going to struggle to do more than 16 GB/second of direct mem copy bandwidth. So being within an order of magnitude of that seems reasonable
ziml77 1y ago
But it repeats the 0.9.0 test on the new hardware. So the first big jump is a hardware change, but the second jump is the software changes.
matja 1y ago
4 generations? 5950x is Zen 3 9950x is Zen 5
- chupasaurus 1y ago
  Sine Zen 2 (3000) the mobile CPUs are up by a thousand respectively to their desktop counterparts. edit: Or Nx2000 where N is from Zen N.
- hinkley 1y ago
  And even with 2, CPU generations aren't what they used to be back when a candy bar cost less than a dollar.

freeone3000 1y ago

They claim a 3GB/s improvement versus previous version of sep on equal hardware — and unlike “marketing” benchmarks, include the actual speed achieved and the hardware used.

stabbles 1y ago
Do note that this speed even before the 3GB/s improvement exceeds the bandwidth of most disks, so the bottleneck is loading data in memory. I don't know of many applications where CSV is produced and consumed in memory, so I wonder what the use is.
- freeone3000 1y ago
  Slower than network! In-memory processing of OLAP tables, streaming splitters, large data set division… but also the faster the parser, the less time you spend parsing and the more you spend doing actual work
  - tetha 1y ago
    This is honestly something that caught me off-guard a bit. If you have good internal network connectivity, small queries and your relational database has the data in memory, it can be faster to fetch data from the DB via the network than reading it from disk. Like, sure, I can give you an application server with faster disks and more memory and you or me are certainly capable of implementing an application server that could load the data from disk faster than all of that. And then we build caching to keep the hot data in memory, because that's faster. But then we've spent very advanced development resources to build a relational database with some application code at the edge. This can make sense in some high frequency trading situations, but in many more mundane web-backends, a chunky database and someone capable of optimizing stupid queries enable and simplify the work of a much bigger number of developers.
    - bee_rider 1y ago
      You can also get this with Infiniband, although it is less surprising, and basically what you’d expect to see. I did once use a system where the network bandwidth was in the same ballpark as the memory bandwidth, which might not be surprising for some of the real HPC-heads here but it surprised me!

jbverschoor 1y ago

They also included 0.9.0 vs 0.10.0. on the new hardware. (21385 vs 18203), so the jump because of software is 17%. Then if we take 0.9.0 on previous hardware (13088) and add the 17%, it's 15375. Version 0.1.0 was 7335. So... 15375/7335 -> a staggering 2.1x improvement in just under 2 years

perching_aix 1y ago

> You can't claim this when you also do a huge hardware jump Well, they did. Personally, I find it an interesting way of looking at it, it's a lens for the "real performance" one could get using this software year over year. (Not saying it isn't a misleading or fallacious claim though.)

bawolff 1y ago

Perhaps, but i think we are well past the moore's law era where a 3x speed up is to be expected just from hardware. Its still a pretty impressive feat in the modern era.