15 ms·

This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.000

This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102 https://github.com/xi-editor/xi-mac/issues/102

azinman2 8y ago
What are you using? Have you tried NSJSONSerialization? It’s quite fast (am very curious how it shows in these benchmarks), but I don’t think it does the fancy Codable stuff.
- eridius 8y ago
  Swift has JSONEncoder and JSONDecoder types to do Codable, though internally they have to encode to/decode from the Foundation objects that JSONSerialization produces.
- jeremy_wiebe 8y ago
  You might want to check out the benchmark I wrote to compare exactly that. https://github.com/jeremywiebe/json-performance https://github.com/jeremywiebe/json-performance
eridius 8y ago
I wrote my own Swift JSON parser quite a while ago, https://github.com/postmates/PMJSON https://github.com/postmates/PMJSON. In my limited benchmarking it parses slower than Foundation's JSONSerialization (by a factor of 2–2.5 IIRC) but encodes faster, and my impression was most of the time was spent constructing Dictionaries, but I didn't do too much performance work on it. It might be interesting to have someone else take a crack at improving the performance. That said, it also includes an event-based parser (called JSONDecoder), so if you want to handle events in order to decode into your own data structure and skip the intermediate JSON data structure, you might be able to get faster than JSONSerialization that way.
saagarjha 8y ago
I ran one of the Codable benchmarks in instruments, and here's what the top functions were: 19.98 s swift_getGenericMetadata 19.15 s newJSONString 16.17 s objc_msgSend 15.33 s _swift_release_(swift::HeapObject*) 14.45 s tiny_malloc_should_clear 12.81 s _swift_retain_(swift::HeapObject*) 11.28 s searchInConformanceCache(swift::TargetMetadata<swift::InProcess> const*, swift::TargetProtocolDescriptor<swift::InProcess> const*) 10.46 s swift_dynamicCastImpl(swift::OpaqueValue*, swift::OpaqueValue*, swift::TargetMetadata<swift::InProcess> const*, swift::TargetMetadata<swift::InProcess> const*, swift::DynamicCastFlags) So it looks like a lot of the time is going into memory management or the Swift runtime performing type checking.
- raphlinus 8y ago
  Yeah, I've done some analysis, it's creating a ton of objects to conform to the Codable protocol, and a lot of those objects are for codingPath, which is updated for basically every node in the tree. It's not a mystery, we just don't know the best way to fix it.
  - saagarjha 8y ago
    Is there a reason you need to use Codable? Sorry if this sounds uninformed, I haven't taken that much time to look at what you're doing exactly (I just ran https://github.com/jeremywiebe/json-performance https://github.com/jeremywiebe/json-performance).
    - raphlinus 8y ago
      That's one of the things we're considering. But it is by far the most idiomatic way to do things in Swift. One of the alternatives we're considering is implementing the line cache (including the update protocol) in Rust, which would be a huge performance jump.
    - jeremy_wiebe 8y ago
      No, I don’t think the project needs to use Codable. The point of that benchmark was to evaluate Codable’s performance under Swift 5. It was posed that performance was much improved. The benchmark points out that it has a little bit but not significantly. Codable is desirable because it encodes/decides directly to strifes vs manually picking fields out of dicts.
marton78 8y ago
Why does Xi use JSON in the first place? It would be easier and faster to use a binary format, e.g. Protobufs, Flatbuffers or if the semantics of JSON is needed: CBOR.
- Skinney 8y ago
  Because JSON encoding/decoding was not found to be a typical performance bottleneck, and because JSON is supported in virtually every programming language (Xi allows you to write frontends in pretty much any language you want).
- aratno 8y ago
  From “Design Decisions”[1]: > JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty of the libraries available for the other ones. 1: https://github.com/xi-editor/xi-editor/blob/master/README.md#design-decisions https://github.com/xi-editor/xi-editor/blob/master/README.md...
  - saurik 8y ago
    ... claims the project whose contributor is here claiming that they are "struggling" with JSON performance. Yeah... "in the noise". LOL.
    - uasm 8y ago
      But... how else are the people that have never seen a byte array or had to flip endianness will be able to write plugins for my text editor?
    - e98cuenc 8y ago
      It seems they're getting parsing times 1,000x slower than any other parser, 10,000x slower than simdjson. The complaint is understandable, but ironic :)
mpweiher 8y ago
Yeah, Swift-most-everything is pretty slow, but particularly parsing/generating. Pre-Swift Foundation serialisation code was already...majestic, and in the Swift conversion they've typically managed to slow things down even further. Which didn't seem possible, but they managed. I have given a bunch of talks[1] on this topic, there's also a chapter in my iOS/macOS performance book[2], which I really recommend if you want to understand this particular topic. I did really fast XML[3][4], CSV[5] and binary plist parsers[6] for Cocoa and also a fast JSON serialiser[7]. All of these are usually around an order of magnitude faster than their Apple equivalents. Sadly, I haven't gotten around to doing a JSON parser. One reason for this is that parsing the JSON at character level is actually the smaller problem, performance-wise, same as for XML. Performance tends to be largely determined by what you create as a result. If you crate generic Foundation/Swift dictionaries/arrays/etc. you have already lost. The overhead of these generic data structure completely overwhelms the cost of scanning a few bytes. So you need something more akin to a steaming interface, and if you create objects you must create them directly, without generic temporary objects. This is where XML is easier, because it has an opening tag that you can use to determine what object to create. With JSON, you get "{" so basically you have to know what structure level corresponds to what objects. Maybe I should write that parser... [1] https://www.google.com/search?hl=en&q=marcel%20weiher%20performance%20talk https://www.google.com/search?hl=en&q=marcel%20weiher%20perf... [2] https://www.amazon.com/gp/product/0321842847/ https://www.amazon.com/gp/product/0321842847/ [3] https://github.com/mpw/Objective-XML https://github.com/mpw/Objective-XML [4] https://blog.metaobject.com/2010/05/xml-performance-revisited.html https://blog.metaobject.com/2010/05/xml-performance-revisite... [5] https://github.com/mpw/MPWFoundation/blob/master/Collections.subproj/MPWDelimitedTable.h https://github.com/mpw/MPWFoundation/blob/master/Collections... [6] https://github.com/mpw/MPWFoundation/blob/master/Collections.subproj/MPWBinaryPlist.m https://github.com/mpw/MPWFoundation/blob/master/Collections... [7] https://github.com/mpw/MPWFoundation/blob/master/Streams.subproj/MPWJSONWriter.m https://github.com/mpw/MPWFoundation/blob/master/Streams.sub...
- gritzko 8y ago
  That resonates well with my conclusions that led to the Replicated Object Notation project. [1]. If the parser creates an AST tree or some number of dictionaries or some other bullshit... "now you have two problems", that's it. I settled on a tabular-log format, which is streamed and immediately consumed most of the time, no intermediate object structures. Then, that "text vs binary" distinction became mostly moot. The binary is slightly more efficient, but grossly less readable, so no big gain, unless at grand scale. [1] http://replicated.cc http://replicated.cc
elizabeth_olu 8y ago
https://www.elizabethblog.com.ng/2019/02/la-police-released-identities-of-three.html https://www.elizabethblog.com.ng/2019/02/la-police-released-...
vlovich123 8y ago
Hey Raph, have you seen https://github.com/bmkor/gason https://github.com/bmkor/gason? Seems like a low-cost bridge to a high-performance C++ implementation.
- raphlinus 8y ago
  Hadn't seen that particular wrapper, but if we're going to take on an FFI solution, we're more likely to use Rust for this, and implement more logic than just JSON parsing.

azinman2 8y ago

What are you using? Have you tried NSJSONSerialization? It’s quite fast (am very curious how it shows in these benchmarks), but I don’t think it does the fancy Codable stuff.

eridius 8y ago
Swift has JSONEncoder and JSONDecoder types to do Codable, though internally they have to encode to/decode from the Foundation objects that JSONSerialization produces.
jeremy_wiebe 8y ago
You might want to check out the benchmark I wrote to compare exactly that. https://github.com/jeremywiebe/json-performance https://github.com/jeremywiebe/json-performance

eridius 8y ago

I wrote my own Swift JSON parser quite a while ago, https://github.com/postmates/PMJSON https://github.com/postmates/PMJSON. In my limited benchmarking it parses slower than Foundation's JSONSerialization (by a factor of 2–2.5 IIRC) but encodes faster, and my impression was most of the time was spent constructing Dictionaries, but I didn't do too much performance work on it. It might be interesting to have someone else take a crack at improving the performance. That said, it also includes an event-based parser (called JSONDecoder), so if you want to handle events in order to decode into your own data structure and skip the intermediate JSON data structure, you might be able to get faster than JSONSerialization that way.

saagarjha 8y ago

I ran one of the Codable benchmarks in instruments, and here's what the top functions were: 19.98 s swift_getGenericMetadata 19.15 s newJSONString 16.17 s objc_msgSend 15.33 s _swift_release_(swift::HeapObject*) 14.45 s tiny_malloc_should_clear 12.81 s _swift_retain_(swift::HeapObject*) 11.28 s searchInConformanceCache(swift::TargetMetadata<swift::InProcess> const*, swift::TargetProtocolDescriptor<swift::InProcess> const*) 10.46 s swift_dynamicCastImpl(swift::OpaqueValue*, swift::OpaqueValue*, swift::TargetMetadata<swift::InProcess> const*, swift::TargetMetadata<swift::InProcess> const*, swift::DynamicCastFlags) So it looks like a lot of the time is going into memory management or the Swift runtime performing type checking.

raphlinus 8y ago
Yeah, I've done some analysis, it's creating a ton of objects to conform to the Codable protocol, and a lot of those objects are for codingPath, which is updated for basically every node in the tree. It's not a mystery, we just don't know the best way to fix it.
- saagarjha 8y ago
  Is there a reason you need to use Codable? Sorry if this sounds uninformed, I haven't taken that much time to look at what you're doing exactly (I just ran https://github.com/jeremywiebe/json-performance https://github.com/jeremywiebe/json-performance).
  - raphlinus 8y ago
    That's one of the things we're considering. But it is by far the most idiomatic way to do things in Swift. One of the alternatives we're considering is implementing the line cache (including the update protocol) in Rust, which would be a huge performance jump.
  - jeremy_wiebe 8y ago
    No, I don’t think the project needs to use Codable. The point of that benchmark was to evaluate Codable’s performance under Swift 5. It was posed that performance was much improved. The benchmark points out that it has a little bit but not significantly. Codable is desirable because it encodes/decides directly to strifes vs manually picking fields out of dicts.

marton78 8y ago

Why does Xi use JSON in the first place? It would be easier and faster to use a binary format, e.g. Protobufs, Flatbuffers or if the semantics of JSON is needed: CBOR.

Skinney 8y ago
Because JSON encoding/decoding was not found to be a typical performance bottleneck, and because JSON is supported in virtually every programming language (Xi allows you to write frontends in pretty much any language you want).
aratno 8y ago
From “Design Decisions”[1]: > JSON. The protocol for front-end / back-end communication, as well as between the back-end and plug-ins, is based on simple JSON messages. I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty of the libraries available for the other ones. 1: https://github.com/xi-editor/xi-editor/blob/master/README.md#design-decisions https://github.com/xi-editor/xi-editor/blob/master/README.md...
- saurik 8y ago
  ... claims the project whose contributor is here claiming that they are "struggling" with JSON performance. Yeah... "in the noise". LOL.
  - uasm 8y ago
    But... how else are the people that have never seen a byte array or had to flip endianness will be able to write plugins for my text editor?
  - e98cuenc 8y ago
    It seems they're getting parsing times 1,000x slower than any other parser, 10,000x slower than simdjson. The complaint is understandable, but ironic :)

mpweiher 8y ago

Yeah, Swift-most-everything is pretty slow, but particularly parsing/generating. Pre-Swift Foundation serialisation code was already...majestic, and in the Swift conversion they've typically managed to slow things down even further. Which didn't seem possible, but they managed. I have given a bunch of talks[1] on this topic, there's also a chapter in my iOS/macOS performance book[2], which I really recommend if you want to understand this particular topic. I did really fast XML[3][4], CSV[5] and binary plist parsers[6] for Cocoa and also a fast JSON serialiser[7]. All of these are usually around an order of magnitude faster than their Apple equivalents. Sadly, I haven't gotten around to doing a JSON parser. One reason for this is that parsing the JSON at character level is actually the smaller problem, performance-wise, same as for XML. Performance tends to be largely determined by what you create as a result. If you crate generic Foundation/Swift dictionaries/arrays/etc. you have already lost. The overhead of these generic data structure completely overwhelms the cost of scanning a few bytes. So you need something more akin to a steaming interface, and if you create objects you must create them directly, without generic temporary objects. This is where XML is easier, because it has an opening tag that you can use to determine what object to create. With JSON, you get "{" so basically you have to know what structure level corresponds to what objects. Maybe I should write that parser... [1] https://www.google.com/search?hl=en&q=marcel%20weiher%20performance%20talk https://www.google.com/search?hl=en&q=marcel%20weiher%20perf... [2] https://www.amazon.com/gp/product/0321842847/ https://www.amazon.com/gp/product/0321842847/ [3] https://github.com/mpw/Objective-XML https://github.com/mpw/Objective-XML [4] https://blog.metaobject.com/2010/05/xml-performance-revisited.html https://blog.metaobject.com/2010/05/xml-performance-revisite... [5] https://github.com/mpw/MPWFoundation/blob/master/Collections.subproj/MPWDelimitedTable.h https://github.com/mpw/MPWFoundation/blob/master/Collections... [6] https://github.com/mpw/MPWFoundation/blob/master/Collections.subproj/MPWBinaryPlist.m https://github.com/mpw/MPWFoundation/blob/master/Collections... [7] https://github.com/mpw/MPWFoundation/blob/master/Streams.subproj/MPWJSONWriter.m https://github.com/mpw/MPWFoundation/blob/master/Streams.sub...

gritzko 8y ago
That resonates well with my conclusions that led to the Replicated Object Notation project. [1]. If the parser creates an AST tree or some number of dictionaries or some other bullshit... "now you have two problems", that's it. I settled on a tabular-log format, which is streamed and immediately consumed most of the time, no intermediate object structures. Then, that "text vs binary" distinction became mostly moot. The binary is slightly more efficient, but grossly less readable, so no big gain, unless at grand scale. [1] http://replicated.cc http://replicated.cc

elizabeth_olu 8y ago

https://www.elizabethblog.com.ng/2019/02/la-police-released-identities-of-three.html https://www.elizabethblog.com.ng/2019/02/la-police-released-...

vlovich123 8y ago

Hey Raph, have you seen https://github.com/bmkor/gason https://github.com/bmkor/gason? Seems like a low-cost bridge to a high-performance C++ implementation.

raphlinus 8y ago
Hadn't seen that particular wrapper, but if we're going to take on an FFI solution, we're more likely to use Rust for this, and implement more logic than just JSON parsing.