5 ms·

It is really unfortunate that AVX-512 has been such a mess when it came to support from Intel on their newer hardware platforms. I think Dr. Lemire has gone qui

It is really unfortunate that AVX-512 has been such a mess when it came to support from Intel on their newer hardware platforms. I think Dr. Lemire has gone quite a ways in demonstrating that there's a lot of potential for software to see performance gains with this type of platform, but I think we'll need further improvements in developer experience before it's more commonplace. The other way to look at it is that quite frankly most business services just don't hit a performance barrier where you're sitting and waiting on a string operation. You're usually waiting on some sort of I/O.

ArrayBoundCheck 4y ago
Parsing JSON takes more time than IO
- wahern 4y ago
  All the object allocation, pointer chasing, and general indirection is what makes JSON parsing slow, particularly building up tree structures. Bump allocators and the like don't help much, at least for the parsing--most malloc implementations already do fast, O(1) allocation for small objects, and have for decades. (Arenas are useful for making deallocation O(1), though.) If you care about raw JSON throughput, use Ragel or something similar to build a state machine that directly parses JSON into a flat, native data structure. Now you have zero allocations without even needing to shim malloc/free. AVX-512 would still be at least as useful, but it's a much more difficult problem to leverage SIMD in a parser generator than in a simple string escape routine or behind a more abstract interface like a regex library. Quite a few language environments these days provide in-language JSON deserializers, but they're still significantly slower than they could be even when they deserialize to flat data structures. The macro languages and internal compiler intrinsics used to accomplish this are the worst possible environments for development. Lisp-like languages aren't really an exception as they tend to trade easier in-language transforms for a steeper climb when it comes to generating optimized native code for the transform.
  - ArrayBoundCheck 4y ago
    I didn't care about speed. I was merely saying the waiting on I/O is false and didn't want to go as far as saying it hasn't been true for 15+ years
  - nerdponx 4y ago
    Is there any serialization format that is more "friendly" to memory allocators while still being human-readable and -writable?
    - deleted 4y ago
      [deleted]
    - hexane360 4y ago
      It's not about the format per se, but more about the fact that you're parsing an unknown/fully general structure in that format. APIs like Rust's serde can help to avoid excessive allocation when you have JSON in a known schema.
    - adgjlsfhk1 4y ago
      text isn't readable without software. why should we expect binary data formats to be?
    - oever 4y ago
      If a file is so large that the processor spends a lot of time parsing, then the file is too large to be conveniently edited by a person. For large files it is best to use a binary format that can be read quickly without parsing or allocation. https://rkyv.org/ https://rkyv.org/ is an example. Being 'friendly' is not why JSON is popular. JSON is popular because the decoder is included the web browser.
      - nerdponx 4y ago
        Thanks, and good point about dataset size. I appreciate the thorough "shootout" benchmarks provided by the authors as well!
  - 4y ago
- wruza 4y ago
  This claims 3gb/s, saturating nvme full queued read. It’s not clear if they construct data or just parse it, but you have to construct data with any format. Other libs also do a great job. Also, io latency shoud be accounted for, because average jsons usually fit into one sdd block, afaiu. https://lemire.me/blog/2020/03/31/we-released-simdjson-0-3-the-fastest-json-parser-in-the-world-is-even-better/ https://lemire.me/blog/2020/03/31/we-released-simdjson-0-3-t... (Edit: just realized this is the same site as subj, heh)
  - ArrayBoundCheck 4y ago
    Yep. I think the fastest nvme's were a tad above 3gb and network IO is certainly faster. It always irks me when something stop being true 15-20years ago is cited today
Negitivefrags 4y ago
"Are you actually IO bound? I seriously doubt it." https://twitter.com/rygorous/status/1423778960739950595 https://twitter.com/rygorous/status/1423778960739950595
- kadoban 4y ago
  That seems specific to games. Business logic is quite a bit more likely to just be waiting on IO, usually fetching from the DB or sending/receiving over the network. This is because there's a lot of business logic where those things are _everything it does_.
  - hresvelgr 4y ago
    I think the point of the twitter rant was addressing a sentiment of "Oh, it's IO bound, guess I can't do anything about it." When there is usually something that can be done about it.
  - janwas 4y ago
    Business logic with tons of branches is indeed not amenable to SIMD, nor is that where the compute is. In a world with 400 Gb/s NICs, being IO bound takes some doing. How much SW manages that without SIMD, even on multiple cores?
    - sakras 4y ago
      Until adoption of io_uring picks up (looking at you RHEL), I’m not sure it’s even really possible to hit the bandwidth of modern nvme drives. It takes a lot of parallelism to do, and eventually context switch overhead kills you.
      - janwas 4y ago
        FWIW Windows has had something basically equivalent (IO completion ports) for quite some time. Indeed helpful, I remember that early SSDs already required high queue depths.
    - 4y ago
anonymoushn 4y ago
If you do your string operations faster, you can get more jobs from the "got I/O" state to the "waiting on I/O" state per unit time!
jeroenhd 4y ago
The Ryzen 7000 cores will support AVX512 so soon enough consumers will finally be able to properly benefit from its broad availability. I/O wait is difficult to model. On an NVME drive over PCIe 5 I doubt you'll find yourself waiting on I/O all that often, especially if you're reading a lot of data. Decompression tools, for example, are often written in such a way to keep the CPU pipeline fed. They're still doing lots of output but while some other core(s) deal(s) with that, the performance optimized core can still crunch numbers. You're probably not storing hundreds of megabytes of JSON because there are much better formats for such data sets, but text algorithms like these make an impact regardless. Loading CSV files, for example, can be a huge CPU bottleneck. I've had to deal with CSV datasets of over a gigabyte and all libraries were kind of terrible at it. The data source also had a JSON version that contained fewer fields for some reason, I guess to save space? Either way, an optimized escape code path would've probably saved seconds on each run of the data analysis, and more if the string split was optimized better as well. The data was loaded into memory already beforehand so the only I/O would've been exchanging data between the RAM and the CPU cache.
- nullifidian 4y ago
  >The Ryzen 7000 cores will support AVX512 Anandtech: "Critically, however, AMD is diverging from Intel in one important aspect: whereas Intel built a true, 512-bit wide SIMD machine for executing AVX-512 instructions, AMD did not. Instead, AMD will be executing these instructions over two cycles. This means AMD’s implementation still benefits from all of the additional instructions, register file space, and other technical improvements that came as part of AVX-512, but they won’t gain the innate doubling in SIMD throughput."
  - secondcoming 4y ago
    But isn't Intel AVX-512 such a power hog that it will clock down the CPU? So the throughput loss might not be that bad if AMD can maintain the clock speed.
    - deleted 4y ago
      [deleted]

ArrayBoundCheck 4y ago

Parsing JSON takes more time than IO

wahern 4y ago
All the object allocation, pointer chasing, and general indirection is what makes JSON parsing slow, particularly building up tree structures. Bump allocators and the like don't help much, at least for the parsing--most malloc implementations already do fast, O(1) allocation for small objects, and have for decades. (Arenas are useful for making deallocation O(1), though.) If you care about raw JSON throughput, use Ragel or something similar to build a state machine that directly parses JSON into a flat, native data structure. Now you have zero allocations without even needing to shim malloc/free. AVX-512 would still be at least as useful, but it's a much more difficult problem to leverage SIMD in a parser generator than in a simple string escape routine or behind a more abstract interface like a regex library. Quite a few language environments these days provide in-language JSON deserializers, but they're still significantly slower than they could be even when they deserialize to flat data structures. The macro languages and internal compiler intrinsics used to accomplish this are the worst possible environments for development. Lisp-like languages aren't really an exception as they tend to trade easier in-language transforms for a steeper climb when it comes to generating optimized native code for the transform.
- ArrayBoundCheck 4y ago
  I didn't care about speed. I was merely saying the waiting on I/O is false and didn't want to go as far as saying it hasn't been true for 15+ years
- nerdponx 4y ago
  Is there any serialization format that is more "friendly" to memory allocators while still being human-readable and -writable?
  - deleted 4y ago
    [deleted]
  - hexane360 4y ago
    It's not about the format per se, but more about the fact that you're parsing an unknown/fully general structure in that format. APIs like Rust's serde can help to avoid excessive allocation when you have JSON in a known schema.
  - adgjlsfhk1 4y ago
    text isn't readable without software. why should we expect binary data formats to be?
  - oever 4y ago
    If a file is so large that the processor spends a lot of time parsing, then the file is too large to be conveniently edited by a person. For large files it is best to use a binary format that can be read quickly without parsing or allocation. https://rkyv.org/ https://rkyv.org/ is an example. Being 'friendly' is not why JSON is popular. JSON is popular because the decoder is included the web browser.
    - nerdponx 4y ago
      Thanks, and good point about dataset size. I appreciate the thorough "shootout" benchmarks provided by the authors as well!
- 4y ago
wruza 4y ago
This claims 3gb/s, saturating nvme full queued read. It’s not clear if they construct data or just parse it, but you have to construct data with any format. Other libs also do a great job. Also, io latency shoud be accounted for, because average jsons usually fit into one sdd block, afaiu. https://lemire.me/blog/2020/03/31/we-released-simdjson-0-3-the-fastest-json-parser-in-the-world-is-even-better/ https://lemire.me/blog/2020/03/31/we-released-simdjson-0-3-t... (Edit: just realized this is the same site as subj, heh)
- ArrayBoundCheck 4y ago
  Yep. I think the fastest nvme's were a tad above 3gb and network IO is certainly faster. It always irks me when something stop being true 15-20years ago is cited today

Negitivefrags 4y ago

"Are you actually IO bound? I seriously doubt it." https://twitter.com/rygorous/status/1423778960739950595 https://twitter.com/rygorous/status/1423778960739950595

kadoban 4y ago
That seems specific to games. Business logic is quite a bit more likely to just be waiting on IO, usually fetching from the DB or sending/receiving over the network. This is because there's a lot of business logic where those things are _everything it does_.
- hresvelgr 4y ago
  I think the point of the twitter rant was addressing a sentiment of "Oh, it's IO bound, guess I can't do anything about it." When there is usually something that can be done about it.
- janwas 4y ago
  Business logic with tons of branches is indeed not amenable to SIMD, nor is that where the compute is. In a world with 400 Gb/s NICs, being IO bound takes some doing. How much SW manages that without SIMD, even on multiple cores?
  - sakras 4y ago
    Until adoption of io_uring picks up (looking at you RHEL), I’m not sure it’s even really possible to hit the bandwidth of modern nvme drives. It takes a lot of parallelism to do, and eventually context switch overhead kills you.
    - janwas 4y ago
      FWIW Windows has had something basically equivalent (IO completion ports) for quite some time. Indeed helpful, I remember that early SSDs already required high queue depths.
  - 4y ago

anonymoushn 4y ago

If you do your string operations faster, you can get more jobs from the "got I/O" state to the "waiting on I/O" state per unit time!

jeroenhd 4y ago

The Ryzen 7000 cores will support AVX512 so soon enough consumers will finally be able to properly benefit from its broad availability. I/O wait is difficult to model. On an NVME drive over PCIe 5 I doubt you'll find yourself waiting on I/O all that often, especially if you're reading a lot of data. Decompression tools, for example, are often written in such a way to keep the CPU pipeline fed. They're still doing lots of output but while some other core(s) deal(s) with that, the performance optimized core can still crunch numbers. You're probably not storing hundreds of megabytes of JSON because there are much better formats for such data sets, but text algorithms like these make an impact regardless. Loading CSV files, for example, can be a huge CPU bottleneck. I've had to deal with CSV datasets of over a gigabyte and all libraries were kind of terrible at it. The data source also had a JSON version that contained fewer fields for some reason, I guess to save space? Either way, an optimized escape code path would've probably saved seconds on each run of the data analysis, and more if the string split was optimized better as well. The data was loaded into memory already beforehand so the only I/O would've been exchanging data between the RAM and the CPU cache.

nullifidian 4y ago
>The Ryzen 7000 cores will support AVX512 Anandtech: "Critically, however, AMD is diverging from Intel in one important aspect: whereas Intel built a true, 512-bit wide SIMD machine for executing AVX-512 instructions, AMD did not. Instead, AMD will be executing these instructions over two cycles. This means AMD’s implementation still benefits from all of the additional instructions, register file space, and other technical improvements that came as part of AVX-512, but they won’t gain the innate doubling in SIMD throughput."
- secondcoming 4y ago
  But isn't Intel AVX-512 such a power hog that it will clock down the CPU? So the throughput loss might not be that bad if AMD can maintain the clock speed.
  - deleted 4y ago
    [deleted]