4 ms·

> 100k lines of multithreaded C++ Just for a JPEG library. Damn!

by hencoappel 2y ago

thrdbndndn 2y ago
I'm also surprised by this number. Granted it's not just old JPEG standard, still 100k sounds like a lot for a decoder.
- archerx 2y ago
  There must be a lot of edge cases that were discovered over the years.
- pixelesque 2y ago
  It's the encoder as well, plus converter code for things like JPEG reading as well, and JPEG XL supports animation...
  - thrdbndndn 2y ago
    I guess the original comment in TFA is a little bit misleading, then. > of the reference decoder (currently behind a pref in Firefox Nightly), which weighs in at more than 100,000 lines of multithreaded C++
- lonjil 2y ago
  That number includes all the different extra tools in the repo, and all tests. And as mentioned, the encoder.
bdemirkir 2y ago
I don't know where that line count comes from. It looks like libjxl has 38314 lines (https://app.codecov.io/gh/libjxl/libjxl/tree/main/lib%2Fjxl https://app.codecov.io/gh/libjxl/libjxl/tree/main/lib%2Fjxl) at the moment.
- JyrkiAlakuijala 2y ago
  and that 38 kloc includes the encoder, psychovisual model, etc. etc. that are not needed for the decoder current decoder is around 20 kloc
  - lifthrasiir 2y ago
    I think Highway is a critical component for libjxl and has to be counted towards the line count. (CMS can be ignored here, and any Rust implementation should ideally use qcms [1] which is already in use in Firefox anyway.) [1] https://github.com/FirefoxGraphics/qcms https://github.com/FirefoxGraphics/qcms
    - janwas 2y ago
      Indeed, though Highway is approaching a system library at this point. It is also a thin wrapper over the compiler's intrinsics headers: e.g. ~16KLOC for arm_neon.h, times the number of targets. That's a lot of code, but it is not comparable with actual codec logic lines.
    - JyrkiAlakuijala 2y ago
      Personally, I'm on the edge on this one. I think both viewpoints are valid. One way to think of Highway is that it is portable multi-platform SIMD intrinsics for C++. While we developed it originally as a part of JPEG XL, it has long ago graduated into a general-purpose library that has various uses, including the recent Gemma.cpp ML launch. Other modern highway uses include: Audio, Browsers, Computational biology, Computer graphics, Cryptography, Grok JPEG 2000, JPEGenc, Jpegli, OpenHTJ2K, Image processing, Image viewers, Information retrieval, Machine learning, Numpy, and Robotics... (copied from: https://github.com/google/highway https://github.com/google/highway) I derived the name from the CityHash, FarmHash, and then HighwayHash series, and considered that Highway would link this library to its roots in the HighwayHash (of course much is also based on Jan's previous work with SIMD). Notably, I resisted using the -li naming here :-D

> 100k lines of multithreaded C++ Just for a JPEG library. Damn!

thrdbndndn 2y ago

I'm also surprised by this number. Granted it's not just old JPEG standard, still 100k sounds like a lot for a decoder.

archerx 2y ago
There must be a lot of edge cases that were discovered over the years.
pixelesque 2y ago
It's the encoder as well, plus converter code for things like JPEG reading as well, and JPEG XL supports animation...
- thrdbndndn 2y ago
  I guess the original comment in TFA is a little bit misleading, then. > of the reference decoder (currently behind a pref in Firefox Nightly), which weighs in at more than 100,000 lines of multithreaded C++
lonjil 2y ago
That number includes all the different extra tools in the repo, and all tests. And as mentioned, the encoder.

bdemirkir 2y ago

I don't know where that line count comes from. It looks like libjxl has 38314 lines (https://app.codecov.io/gh/libjxl/libjxl/tree/main/lib%2Fjxl https://app.codecov.io/gh/libjxl/libjxl/tree/main/lib%2Fjxl) at the moment.

JyrkiAlakuijala 2y ago
and that 38 kloc includes the encoder, psychovisual model, etc. etc. that are not needed for the decoder current decoder is around 20 kloc
- lifthrasiir 2y ago
  I think Highway is a critical component for libjxl and has to be counted towards the line count. (CMS can be ignored here, and any Rust implementation should ideally use qcms [1] which is already in use in Firefox anyway.) [1] https://github.com/FirefoxGraphics/qcms https://github.com/FirefoxGraphics/qcms
  - janwas 2y ago
    Indeed, though Highway is approaching a system library at this point. It is also a thin wrapper over the compiler's intrinsics headers: e.g. ~16KLOC for arm_neon.h, times the number of targets. That's a lot of code, but it is not comparable with actual codec logic lines.
  - JyrkiAlakuijala 2y ago
    Personally, I'm on the edge on this one. I think both viewpoints are valid. One way to think of Highway is that it is portable multi-platform SIMD intrinsics for C++. While we developed it originally as a part of JPEG XL, it has long ago graduated into a general-purpose library that has various uses, including the recent Gemma.cpp ML launch. Other modern highway uses include: Audio, Browsers, Computational biology, Computer graphics, Cryptography, Grok JPEG 2000, JPEGenc, Jpegli, OpenHTJ2K, Image processing, Image viewers, Information retrieval, Machine learning, Numpy, and Robotics... (copied from: https://github.com/google/highway https://github.com/google/highway) I derived the name from the CityHash, FarmHash, and then HighwayHash series, and considered that Highway would link this library to its roots in the HighwayHash (of course much is also based on Jan's previous work with SIMD). Notably, I resisted using the -li naming here :-D