4 ms·

This was a great tool written in c, for handling large data files fast, unfortunately the new golang rewrite (version 6.0) is much slower (around 10x in a quick

This was a great tool written in c, for handling large data files fast, unfortunately the new golang rewrite (version 6.0) is much slower (around 10x in a quick test I did).

svnpenn 5y ago
Fine by me. The C version was a pain in the ass to compile. The Go version compiles in less than one minute, with a simple `go build`, on Windows, Linux or any Go platform. I will take that over the faster speed every time.
- tinix 5y ago
  That's a fascinating perspective... This mentality is why modern software seems slower... Because it is. So many technological advances, computers are exponentially faster in so many ways, yet we regress in performance. It's a shame, really...
  - svnpenn 5y ago
    Well, compared to Go, Rust, PHP, or many other modern tools, C is garbage in terms of build experience. Speed is important, but its not the only factor.
  - marginalia_nu 5y ago
    It's basically Parkinson's law. If you give someone 10x the performance, they'll find a way of doing the same things they did before using 10x more resources. It's why you have user interfaces today that are often as slow as, or slower than the equivalent Windows 3.11 interfaces. In 1993 things took time because the processor was an actual potato and the data fetched off a floppy disk, today things are slow because the code being run is 1-10k times slower and messages need to be continuously sent to a datacenter half way around the world to inform an analytics server about your every input.
    - john_kerl 5y ago
      The 10x number was before improvements on https://github.com/johnkerl/miller/pull/786 https://github.com/johnkerl/miller/pull/786 et al. The earlier negative perf results were my fault, not Go's -- I was focusing initially on the port and feature development, leaving benchmarking and optimization until the end. That said, Go is a bit slower than C line for line; however, Miller 5 (in C) was single-threaded and Miller 6 (in Go) actively uses multicore. This is why complex processing chains now run much quicker in Go than in C -- due to multicore and pipelining which are much easier to do in Go.
- jasonpeacock 5y ago
  You'd rather optimize for compile time, which you do (maybe) 1x, over runtime, which you easily do >100x? You're focusing on the developer, not the user, but this is a user tool :(
  - john_kerl 5y ago
    Agreed, very much a user tool -- once it's released most people will get it via brew/yum/apt/conda/chocolatey and compile time will not be relevant for most folks. Miller has always been a user tool and will remain so. I think the confusion arose because I did the port per se and feature-adds first (which has taken most of the time), and left analysis/optimization (even low-hanging fruit like output-buffering) until the very end -- https://github.com/johnkerl/miller/pull/786 https://github.com/johnkerl/miller/pull/786 for example merged just a couple days ago.
- john_kerl 5y ago
  Agreed; I've never really liked C compiles, and the Go build process is dreamy. Another win (besides more features, and recent performance improvements) -- the Windows version is now a snap. Now `mlr.exe` takes it rightful place alongside the Linux and Mac executables (https://github.com/johnkerl/miller/releases/tag/v6.0.0-beta https://github.com/johnkerl/miller/releases/tag/v6.0.0-beta). Go+Windows is heavenly & I'll never need to tweak MSYS2/Appveyor/DLLs/etc again. :) See also https://miller.readthedocs.io/en/latest/miller-on-windows/ https://miller.readthedocs.io/en/latest/miller-on-windows/
bee_rider 5y ago
The version in the Ubuntu repos is 5.10. I wonder if they'll "upgrade" to 6.0. Downthread people are complaining about the compilation time for the C version, which seems like a much less significant issue when the cost of compilation is amortized over many users. But if Go is a much more productive language, I guess a feature delta will accumulate...
- john_kerl 5y ago
  Distros are still at 5.10 pending a few final issues to be resolved -- then 6.0 can be generally released. Hopefully in a month or so. The most important release blocker (now resolved) is https://github.com/johnkerl/miller/pull/786 https://github.com/johnkerl/miller/pull/786 et al., thanks to which Miller 6 performance is now on par with Miller 5 for simple processing, and far better than Miller 5 for complex processing chains.
  - bee_rider 5y ago
    Awesome! Hopefully the distros will keep up. I saw in some other comments you mentioned some performance lift coming from thread level parallelism in Go. I wonder -- can you get oversubscription issues if the user is doing their own parallelism (I assume some folks will implement 'parallelism' by throwing a bunch of independent processes at a bunch of independent records). I know in openMP (for example) this would be something where the user is expected to keep track of it, but maybe the Go runtime handles this stuff gracefully?
    - john_kerl 5y ago
      Indeed -- in Go technically it's "concurrency" not "parallelism" (something the Go folks are careful to articulate) but with multiple processors available, in effect you get multicore processing so calling it "parallelism" isn't far off :). More specifically, Miller uses separate "goroutines" (more or less threads) -- 2 for input (one for raw byte stream ingest and one for forming records), 1 for each verb (sort then head would be two verbs), and 1 for output (records back to strings), all pipelined up. Some of the recent perf improvements came from splitting the record-reader into two concurrent goroutines like that. Some of the older perf improvements are from pipelining the verbs. But yes, if you have say 16 CPUs and you launch 10, or 20, or 30 Miller executables -- in the C impl each would soak a single CPU and anything beyond 16, the OS would have to multitask things. With Miller 6, basically the same kind of thing except each executable will be trying to use more than one CPU if it can. If it can't, the Go runtime and the OS will multitask. And both do tend to handle this stuff gracefully and without tuning on the part of the user. Also I'd point out that even with big files & deeper multi-verb processing chains, htop rarely shows over 250% CPU, maybe 350% for deeper chains -- the input and output processing typically take most of the time, and the verbs in between not as much. When we run out of memory, systems crash; when we run out of CPU, things just take a bit longer. I.e. the oversubscription is a real but non-fatal issue, for C or for Go ...
Zababa 5y ago
I hope that this is temporary to make working on the project easier, Go can get pretty close to C in terms of speed and it's usually easier to jump into a Go project. But 10 times slower seems way too much.
- john_kerl 5y ago
  Indeed -- that was before final optimization work on https://github.com/johnkerl/miller/pull/786 https://github.com/johnkerl/miller/pull/786 et al., thanks to which Miller 6 performance is now on par with Miller 5 for simple processing, and far better than Miller 5 for complex processing chains. See that PR for tabulated processing-time results.
  - Zababa 5y ago
    That's impressive, congratulations!
john_kerl 5y ago
Thanks -- my apologies for the sequencing of the dev work. I focused first on the port per se; second on feature-adds; leaving benchmarking, performance analysis, and optimizations (even some real low-hanging fruit) until last. The 10x number was indeed the case pre-optimization ... on https://github.com/johnkerl/miller/pull/786 https://github.com/johnkerl/miller/pull/786 you can see tabulated results where Miller 6 is now on par with Miller 5, and in some cases is far faster.

This was a great tool written in c, for handling large data files fast, unfortunately the new golang rewrite (version 6.0) is much slower (around 10x in a quick test I did).

svnpenn 5y ago

Fine by me. The C version was a pain in the ass to compile. The Go version compiles in less than one minute, with a simple `go build`, on Windows, Linux or any Go platform. I will take that over the faster speed every time.

tinix 5y ago
That's a fascinating perspective... This mentality is why modern software seems slower... Because it is. So many technological advances, computers are exponentially faster in so many ways, yet we regress in performance. It's a shame, really...
- svnpenn 5y ago
  Well, compared to Go, Rust, PHP, or many other modern tools, C is garbage in terms of build experience. Speed is important, but its not the only factor.
- marginalia_nu 5y ago
  It's basically Parkinson's law. If you give someone 10x the performance, they'll find a way of doing the same things they did before using 10x more resources. It's why you have user interfaces today that are often as slow as, or slower than the equivalent Windows 3.11 interfaces. In 1993 things took time because the processor was an actual potato and the data fetched off a floppy disk, today things are slow because the code being run is 1-10k times slower and messages need to be continuously sent to a datacenter half way around the world to inform an analytics server about your every input.
  - john_kerl 5y ago
    The 10x number was before improvements on https://github.com/johnkerl/miller/pull/786 https://github.com/johnkerl/miller/pull/786 et al. The earlier negative perf results were my fault, not Go's -- I was focusing initially on the port and feature development, leaving benchmarking and optimization until the end. That said, Go is a bit slower than C line for line; however, Miller 5 (in C) was single-threaded and Miller 6 (in Go) actively uses multicore. This is why complex processing chains now run much quicker in Go than in C -- due to multicore and pipelining which are much easier to do in Go.
jasonpeacock 5y ago
You'd rather optimize for compile time, which you do (maybe) 1x, over runtime, which you easily do >100x? You're focusing on the developer, not the user, but this is a user tool :(
- john_kerl 5y ago
  Agreed, very much a user tool -- once it's released most people will get it via brew/yum/apt/conda/chocolatey and compile time will not be relevant for most folks. Miller has always been a user tool and will remain so. I think the confusion arose because I did the port per se and feature-adds first (which has taken most of the time), and left analysis/optimization (even low-hanging fruit like output-buffering) until the very end -- https://github.com/johnkerl/miller/pull/786 https://github.com/johnkerl/miller/pull/786 for example merged just a couple days ago.
john_kerl 5y ago
Agreed; I've never really liked C compiles, and the Go build process is dreamy. Another win (besides more features, and recent performance improvements) -- the Windows version is now a snap. Now `mlr.exe` takes it rightful place alongside the Linux and Mac executables (https://github.com/johnkerl/miller/releases/tag/v6.0.0-beta https://github.com/johnkerl/miller/releases/tag/v6.0.0-beta). Go+Windows is heavenly & I'll never need to tweak MSYS2/Appveyor/DLLs/etc again. :) See also https://miller.readthedocs.io/en/latest/miller-on-windows/ https://miller.readthedocs.io/en/latest/miller-on-windows/

bee_rider 5y ago

The version in the Ubuntu repos is 5.10. I wonder if they'll "upgrade" to 6.0. Downthread people are complaining about the compilation time for the C version, which seems like a much less significant issue when the cost of compilation is amortized over many users. But if Go is a much more productive language, I guess a feature delta will accumulate...

john_kerl 5y ago
Distros are still at 5.10 pending a few final issues to be resolved -- then 6.0 can be generally released. Hopefully in a month or so. The most important release blocker (now resolved) is https://github.com/johnkerl/miller/pull/786 https://github.com/johnkerl/miller/pull/786 et al., thanks to which Miller 6 performance is now on par with Miller 5 for simple processing, and far better than Miller 5 for complex processing chains.
- bee_rider 5y ago
  Awesome! Hopefully the distros will keep up. I saw in some other comments you mentioned some performance lift coming from thread level parallelism in Go. I wonder -- can you get oversubscription issues if the user is doing their own parallelism (I assume some folks will implement 'parallelism' by throwing a bunch of independent processes at a bunch of independent records). I know in openMP (for example) this would be something where the user is expected to keep track of it, but maybe the Go runtime handles this stuff gracefully?
  - john_kerl 5y ago
    Indeed -- in Go technically it's "concurrency" not "parallelism" (something the Go folks are careful to articulate) but with multiple processors available, in effect you get multicore processing so calling it "parallelism" isn't far off :). More specifically, Miller uses separate "goroutines" (more or less threads) -- 2 for input (one for raw byte stream ingest and one for forming records), 1 for each verb (sort then head would be two verbs), and 1 for output (records back to strings), all pipelined up. Some of the recent perf improvements came from splitting the record-reader into two concurrent goroutines like that. Some of the older perf improvements are from pipelining the verbs. But yes, if you have say 16 CPUs and you launch 10, or 20, or 30 Miller executables -- in the C impl each would soak a single CPU and anything beyond 16, the OS would have to multitask things. With Miller 6, basically the same kind of thing except each executable will be trying to use more than one CPU if it can. If it can't, the Go runtime and the OS will multitask. And both do tend to handle this stuff gracefully and without tuning on the part of the user. Also I'd point out that even with big files & deeper multi-verb processing chains, htop rarely shows over 250% CPU, maybe 350% for deeper chains -- the input and output processing typically take most of the time, and the verbs in between not as much. When we run out of memory, systems crash; when we run out of CPU, things just take a bit longer. I.e. the oversubscription is a real but non-fatal issue, for C or for Go ...

Zababa 5y ago

I hope that this is temporary to make working on the project easier, Go can get pretty close to C in terms of speed and it's usually easier to jump into a Go project. But 10 times slower seems way too much.

john_kerl 5y ago
Indeed -- that was before final optimization work on https://github.com/johnkerl/miller/pull/786 https://github.com/johnkerl/miller/pull/786 et al., thanks to which Miller 6 performance is now on par with Miller 5 for simple processing, and far better than Miller 5 for complex processing chains. See that PR for tabulated processing-time results.
- Zababa 5y ago
  That's impressive, congratulations!

john_kerl 5y ago

Thanks -- my apologies for the sequencing of the dev work. I focused first on the port per se; second on feature-adds; leaving benchmarking, performance analysis, and optimizations (even some real low-hanging fruit) until last. The 10x number was indeed the case pre-optimization ... on https://github.com/johnkerl/miller/pull/786 https://github.com/johnkerl/miller/pull/786 you can see tabulated results where Miller 6 is now on par with Miller 5, and in some cases is far faster.