6 ms·
That's strange, you should be able to stream the file right into a tiny perl executable at the same speed as the bottlenecking hardware. The kernel will take ca
by lessbergstein 3y ago
That's strange, you should be able to stream the file right into a tiny perl executable at the same speed as the bottlenecking hardware. The kernel will take care of all the logistics. You're probably trying to do too much explicitly. Just use a pipe. Perl should be done before Jit completes.
- deleted 3y ago[deleted]
- gerikson 3y agoI profiled my attempt, actually reading each line is the bottleneck.
- lessbergstein 3y agoPerl is always going to be much faster than Java at tasks like this. Use stdin and chomp() instead of reading each line explicitly. This is really a small, trivial task for a perl script. Even with a billion lines this is nothing for a modern cpu and perl.
- gerikson 3y ago/r/perl would beg to differ: https://www.reddit.com/r/perl/comments/18ygpsi/1_billion_row_challenge_in_perl/kgbzpjz/ https://www.reddit.com/r/perl/comments/18ygpsi/1_billion_row...
- lessbergstein 3y agoReddit?
- gerikson 3y agoUsing cat to redirect the file to /dev/null takes 18s on my machine (a low-end NUC). Just running a noop on the file in Perl (ie. feeding it into a `while (<>)` loop but not acting on the contents) takes ~2 minutes. 1B lines is a lot, and Java ain't a slouch.
- lessbergstein 3y agoWhy are you using cat at all? Use a pipe. This isn't hard stuff. Don't use <>, feed the file into a scalar or array. it should only take a few seconds to process a billion lines. https://www.perl.com/pub/2003/11/21/slurp.html/#:~:text=Another%20major%20win%20for%20slurping%20over%20line%20by,Plenty%20of%20work%20for%20each%20line%20read%20in https://www.perl.com/pub/2003/11/21/slurp.html/#:~:text=Anot....
- sobellian 3y agoIf it isn't hard, then perhaps you could demonstrate with a complete perl program that you think should beat java.