5 ms·

I'm starting to sound like a broken record on this, but if you're going to compare to your system wc without trying to figure out if it was compiled with -O3 [E

I'm starting to sound like a broken record on this, but if you're going to compare to your system wc without trying to figure out if it was compiled with -O3 [EDIT: and what source code it was compiled from], you haven't shown anything in the sequential case. What this article does show is that Futhark really does allow one to express this in a much simpler way than Haskell.

Athas 7y ago
That's a good point, but the -O3 doesn't actually do a whole lot here. I recompiled the Futhark-generated C code with just -O and performance was unchanged. If you look at the generated C code, there isn't really a lot to do either: https://gist.github.com/athas/7c8ffc2620a9406e4bbb0df89f2fc9f6 https://gist.github.com/athas/7c8ffc2620a9406e4bbb0df89f2fc9... I hope I can assume that RHEL compiles their wc with at least -O.
- tom_mellior 7y ago
  > That's a good point, but the -O3 doesn't actually do a whole lot here. True, maybe it's not about -O3 but about some factor in the unknown source code of the system wc. I did compile one version of wc with -O3 and it beat my system wc (Ubuntu) by 2x: https://news.ycombinator.com/item?id=21271951 https://news.ycombinator.com/item?id=21271951
  - Athas 7y ago
    Honestly, I would expect the main reason my wc is faster is that mmap()ing the file and then reading it in a huge chunk is about as fast as the kernel's IO can go. GNU wc cannot do this in general because it's supposed to work on pipes as well, and I doubt anyone cared enough about the tiny performance difference to exploit the case where the input file is mmap()able. (I had actually hoped Futhark would be slower sequentially, just so this wouldn't be the focus of the discussion!)
    - deleted 7y ago
      [deleted]
    - tom_mellior 7y ago
      > GNU wc cannot do this in general because it's supposed to work on pipes as well I used the "reference" source code linked from the original Haskell post, a BSD version hosted by Apple: https://opensource.apple.com/source/text_cmds/text_cmds-68/wc/wc.c.auto.html https://opensource.apple.com/source/text_cmds/text_cmds-68/w... It uses raw read() from a file descriptor and works with pipes as well. I think the only special handling for stdin vs. an actual file it has is calling fstat() if only the number of characters is requested, which shouldn't apply here. So yes, this version does need to do more complicated I/O than a simple mmap(). And (broken record, but I'll stop after this) it's 2x as fast as my system's GNU wc (when compiled with -O3 vs. however the system wc was compiled). > I had actually hoped Futhark would be slower sequentially It might still turn out to be, if you see if you can get a faster C version of wc.
      - YSFEJ4SWJUVU6 7y ago
        >It might still turn out to be, if you see if you can get a faster C version of wc. You definitely can, at least if you allow manually vectorized code. On my system, with a 1.661GB file (256 times big.txt from the original Haskell post) GNU wc takes about 6.5s (real time), a stripped down version of Apple's implementation about 4.1s, and a single-threaded vectorized wc (written in C) only 0.27s. (These times are of course only with a hot cache. For reference, catting the same file to /dev/null takes about 0.18s.) edit: corrected the time for the BSD-derived implementation
        ummaycoc 7y ago
        Would you mind sharing the 0.27s version?
        YSFEJ4SWJUVU6 7y ago
        See https://git.io/JeEjB https://git.io/JeEjB
  - glouwbug 7y ago
    Most of -O3's performance is lost across translations units. Compile and link with `-flto --march=native -O3` and you're good to go.
OskarS 7y ago
Fair enough, but I'm honestly really enjoying this little round of programming language bragging. It's fun to see these smallish (well, Haskell is pretty big) languages all duke it out over how to solve this problem. Like: yeah, nobody's ACTUALLY going to replace their system wc with these implementations, but the articles themselves are nevertheless very fun. A very clever little case study!

Athas 7y ago

That's a good point, but the -O3 doesn't actually do a whole lot here. I recompiled the Futhark-generated C code with just -O and performance was unchanged. If you look at the generated C code, there isn't really a lot to do either: https://gist.github.com/athas/7c8ffc2620a9406e4bbb0df89f2fc9f6 https://gist.github.com/athas/7c8ffc2620a9406e4bbb0df89f2fc9... I hope I can assume that RHEL compiles their wc with at least -O.

tom_mellior 7y ago
> That's a good point, but the -O3 doesn't actually do a whole lot here. True, maybe it's not about -O3 but about some factor in the unknown source code of the system wc. I did compile one version of wc with -O3 and it beat my system wc (Ubuntu) by 2x: https://news.ycombinator.com/item?id=21271951 https://news.ycombinator.com/item?id=21271951
- Athas 7y ago
  Honestly, I would expect the main reason my wc is faster is that mmap()ing the file and then reading it in a huge chunk is about as fast as the kernel's IO can go. GNU wc cannot do this in general because it's supposed to work on pipes as well, and I doubt anyone cared enough about the tiny performance difference to exploit the case where the input file is mmap()able. (I had actually hoped Futhark would be slower sequentially, just so this wouldn't be the focus of the discussion!)
  - deleted 7y ago
    [deleted]
  - tom_mellior 7y ago
    > GNU wc cannot do this in general because it's supposed to work on pipes as well I used the "reference" source code linked from the original Haskell post, a BSD version hosted by Apple: https://opensource.apple.com/source/text_cmds/text_cmds-68/wc/wc.c.auto.html https://opensource.apple.com/source/text_cmds/text_cmds-68/w... It uses raw read() from a file descriptor and works with pipes as well. I think the only special handling for stdin vs. an actual file it has is calling fstat() if only the number of characters is requested, which shouldn't apply here. So yes, this version does need to do more complicated I/O than a simple mmap(). And (broken record, but I'll stop after this) it's 2x as fast as my system's GNU wc (when compiled with -O3 vs. however the system wc was compiled). > I had actually hoped Futhark would be slower sequentially It might still turn out to be, if you see if you can get a faster C version of wc.
    - YSFEJ4SWJUVU6 7y ago
      >It might still turn out to be, if you see if you can get a faster C version of wc. You definitely can, at least if you allow manually vectorized code. On my system, with a 1.661GB file (256 times big.txt from the original Haskell post) GNU wc takes about 6.5s (real time), a stripped down version of Apple's implementation about 4.1s, and a single-threaded vectorized wc (written in C) only 0.27s. (These times are of course only with a hot cache. For reference, catting the same file to /dev/null takes about 0.18s.) edit: corrected the time for the BSD-derived implementation
      - ummaycoc 7y ago
        Would you mind sharing the 0.27s version?
        YSFEJ4SWJUVU6 7y ago
        See https://git.io/JeEjB https://git.io/JeEjB
- glouwbug 7y ago
  Most of -O3's performance is lost across translations units. Compile and link with `-flto --march=native -O3` and you're good to go.

OskarS 7y ago

Fair enough, but I'm honestly really enjoying this little round of programming language bragging. It's fun to see these smallish (well, Haskell is pretty big) languages all duke it out over how to solve this problem. Like: yeah, nobody's ACTUALLY going to replace their system wc with these implementations, but the articles themselves are nevertheless very fun. A very clever little case study!