4 ms·

Reading the code to help me understand how things are I got the wc_lines from coreutils: https://github.com/coreutils/coreutils/blob/master/src/wc.c#L248 https

Reading the code to help me understand how things are I got the wc_lines from coreutils: https://github.com/coreutils/coreutils/blob/master/src/wc.c#L248 https://github.com/coreutils/coreutils/blob/master/src/wc.c#... And I thought "damn, I understand nothing about the state-machine stuff, how did they made this faster ?" Truth is: they did not Of course, this is just the "count line" part. Other parts are indeed faster. Coreutils 9.4: 0.67 [jack:/tmp] /usr/bin/time -v wc -l debian-12.2.0-amd64-netinst.iso 2524855 debian-12.2.0-amd64-netinst.iso Command being timed: "wc -l debian-12.2.0-amd64-netinst.iso" User time (seconds): 0.00 System time (seconds): 0.07 Percent of CPU this job got: 98% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.07 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 2176 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 99 Voluntary context switches: 1 Involuntary context switches: 0 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 And wc2 (from gcc -O3 -o wc2 wc2.c): 0.47 [jack:/tmp] /usr/bin/time -v ./wc2 -l debian-12.2.0-amd64-netinst.iso 2524855 debian-12.2.0-amd64-netinst.iso Command being timed: "./wc2 -l debian-12.2.0-amd64- netinst.iso" User time (seconds): 0.97 System time (seconds): 0.05 Percent of CPU this job got: 99% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.03 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 2176 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 146 Voluntary context switches: 1 Involuntary context switches: 1 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 Gotta keep reading

orf 2y ago
Any notable difference if you pipe the file in, rather than read from disk? IO caches are a thing as well, don’t use the first measurement.
- JackSlateur 2y ago
  I've accounted for IO cache, the test is running from an in-memory dataset After checking a bit more, wc2 is indeed faster than wc in a specific use case: counting characters. That is, transforming a sequence of one or multiple bytes into meaningful character (wc --chars somefile). In wc, the related code is here: https://github.com/coreutils/coreutils/blob/master/src/wc.c#L444 https://github.com/coreutils/coreutils/blob/master/src/wc.c#... So in the end, wc uses the function while wc2 does not: https://en.cppreference.com/w/c/string/multibyte/mbrtoc32 https://en.cppreference.com/w/c/string/multibyte/mbrtoc32 Now, what I cannot tell : is wc2's code really comparable versus mbrtoc32 ? From a couple of tests, it works against ascii and utf8 dataset, but is that all ? Or are there edgecases cleverly handled by mbrtoc32 (with a performance cost) ?
roetlich 2y ago
From looking at the code, the clever bit is how this is doing word count. (and word & line count at the same time). So yeah, it makes -l isn't better.

orf 2y ago

Any notable difference if you pipe the file in, rather than read from disk? IO caches are a thing as well, don’t use the first measurement.

JackSlateur 2y ago
I've accounted for IO cache, the test is running from an in-memory dataset After checking a bit more, wc2 is indeed faster than wc in a specific use case: counting characters. That is, transforming a sequence of one or multiple bytes into meaningful character (wc --chars somefile). In wc, the related code is here: https://github.com/coreutils/coreutils/blob/master/src/wc.c#L444 https://github.com/coreutils/coreutils/blob/master/src/wc.c#... So in the end, wc uses the function while wc2 does not: https://en.cppreference.com/w/c/string/multibyte/mbrtoc32 https://en.cppreference.com/w/c/string/multibyte/mbrtoc32 Now, what I cannot tell : is wc2's code really comparable versus mbrtoc32 ? From a couple of tests, it works against ascii and utf8 dataset, but is that all ? Or are there edgecases cleverly handled by mbrtoc32 (with a performance cost) ?

roetlich 2y ago

From looking at the code, the clever bit is how this is doing word count. (and word & line count at the same time). So yeah, it makes -l isn't better.