5 ms·

There are fully correct JSON parsers you aren't going to beat, even if you implement a subset. [1] [1]: https://branchfree.org/2019/02/25/paper-parsing-gigabyt

There are fully correct JSON parsers you aren't going to beat, even if you implement a subset. [1] [1]: https://branchfree.org/2019/02/25/paper-parsing-gigabytes-of-json-per-second/ https://branchfree.org/2019/02/25/paper-parsing-gigabytes-of...

Ma8ee 6y ago
No, I won't beat them. But if it a limited subset that I can implement with twenty lines straightforward code, that will often be cheaper. I've been on projects where they imported xml-parsers many times bigger than the rest of the whole codebase just to send a well formatted order number.
- dogma1138 6y ago
  And what happens when your parsing needs to be expanded?
  - fernandotakai 6y ago
    taking in consideration how business work, in a few years you are going to have a full parser in your hands.
    - dogma1138 6y ago
      With all the technical debt associated with it, which is the problem basing your project on a dependency that would allow you to easily scale and add features is a huge benefit. This is like saying you should roll your own crypto because you only need to do a very limited sub set of crypto operations so why use something like NaCl or Tink.
      - samus 6y ago
        Encryption is a terrible edge case. If you are forced to half-ass encryption, you should seriously question the project requirements. Bad encryption can be worse than none at all. Things won't end well if data security is treated as a detail.
  - Ma8ee 6y ago
    Then you reevaluate the situation. YAGNI is true here too.
    - dogma1138 6y ago
      YANGI is nice but when the PM asks you why it would take two months to accept a new JSON format from a client and you’ll answer well because we didn’t want to use an industry standard fully functional and vetted JSON parser so we essentially wrote our own edge case parser we both know how that conversation will end. And YANGI doesn’t have anything against dependencies.
      - Ma8ee 6y ago
        I said, then you reevaluate the situation. When there are new requirements you do a quick estimate if you should add four new lines to the existing 20 or if it is worth to switch to an external library. 4 new lines to the 20, just add them to the core. But if this is regularly occurring that you have to add things, or requirements affecting this particular little parser that was supposed to be simple and static isn’t, then you should probably change your decision and use the library. But you do that only then. Because chances are that with your approach you are going to drag along a large generic library that you only use a tiny fraction of. And that also has costs. In particular if your immediate impulse always is to add another library instead of writing things yourself.
    - mro_name 6y ago
      That's IMHO key – solve problems once you know them, not earlier. Old idea, also core to XP.
- abiogenesis 6y ago
  What do you mean by cheaper? By using a third party library you are writing twenty lines less code, so it's cheaper in that aspect. There are probably libraries that are faster than your twenty lines of un-optimized code, so it's cheaper as far as computing resources are considered too. The only time it could matter is when you ship the code to the client through the wire (such as a Javascript bundle).
  - Retric 6y ago
    You can’t use a library with zero lines of code. On top of this library’s always have development overhead outside of the code you write. Ex: What version number should you use? Did the latest version break something? Did the old version break something on the latest compiler? Etc etc.
  - Ma8ee 6y ago
    It’s cheaper in the sense that it is faster to write and maintain those 20 lines of code. Because someone has to evaluate the library, understand it well enough to actually call it and then make sure it stays up to date. And often there are a few lines of code to translate your data into a form that the library requires etc.
    - BigJono 6y ago
      Plus for every developer to come, one call to an external library usually also means 30 pages of documentation to trawl through if they ever want to change anything, 29.9 of which is completely irrelevant to whatever your narrow use case is. That's the real cost. The size of the code means absolutely nothing.
- benibela 6y ago
  xml and json are different beasts A json parser can probably be implemented in an afternoon. But a conformant xml parser can take months. There are some weird things in xml, for example, this is correct xml: <?xml version="1.0"?><!DOCTYPE abc[<?abc >]]<abc><abc/>]>>?>]><?x?><a/><?x <(x)>>?> And for external entities, you need an http client in the xml parser, although it is probably better to not support that part of the standard.
  - Ma8ee 6y ago
    And we didn’t need a conformant xml parser, which I know is huge and complex.
tnorthcutt 6y ago
Indeed: https://github.com/qntm/fastjson https://github.com/qntm/fastjson
flukus 6y ago
And it's going to take more than an afternoon to evaluate these parsers. You have to look at the options, evaluate the API, evaluate if they're stable and supported, evaluate if they integrate well with you're project, evaluate any dependencies they might have, etc. Then you need a plan to manage these dependencies long term. If you're needs can be solved adequately by strtok then that's a far simpler and more maintainable solution that can be knocked out in an afternoon.

Ma8ee 6y ago

No, I won't beat them. But if it a limited subset that I can implement with twenty lines straightforward code, that will often be cheaper. I've been on projects where they imported xml-parsers many times bigger than the rest of the whole codebase just to send a well formatted order number.

dogma1138 6y ago
And what happens when your parsing needs to be expanded?
- fernandotakai 6y ago
  taking in consideration how business work, in a few years you are going to have a full parser in your hands.
  - dogma1138 6y ago
    With all the technical debt associated with it, which is the problem basing your project on a dependency that would allow you to easily scale and add features is a huge benefit. This is like saying you should roll your own crypto because you only need to do a very limited sub set of crypto operations so why use something like NaCl or Tink.
    - samus 6y ago
      Encryption is a terrible edge case. If you are forced to half-ass encryption, you should seriously question the project requirements. Bad encryption can be worse than none at all. Things won't end well if data security is treated as a detail.
- Ma8ee 6y ago
  Then you reevaluate the situation. YAGNI is true here too.
  - dogma1138 6y ago
    YANGI is nice but when the PM asks you why it would take two months to accept a new JSON format from a client and you’ll answer well because we didn’t want to use an industry standard fully functional and vetted JSON parser so we essentially wrote our own edge case parser we both know how that conversation will end. And YANGI doesn’t have anything against dependencies.
    - Ma8ee 6y ago
      I said, then you reevaluate the situation. When there are new requirements you do a quick estimate if you should add four new lines to the existing 20 or if it is worth to switch to an external library. 4 new lines to the 20, just add them to the core. But if this is regularly occurring that you have to add things, or requirements affecting this particular little parser that was supposed to be simple and static isn’t, then you should probably change your decision and use the library. But you do that only then. Because chances are that with your approach you are going to drag along a large generic library that you only use a tiny fraction of. And that also has costs. In particular if your immediate impulse always is to add another library instead of writing things yourself.
  - mro_name 6y ago
    That's IMHO key – solve problems once you know them, not earlier. Old idea, also core to XP.
abiogenesis 6y ago
What do you mean by cheaper? By using a third party library you are writing twenty lines less code, so it's cheaper in that aspect. There are probably libraries that are faster than your twenty lines of un-optimized code, so it's cheaper as far as computing resources are considered too. The only time it could matter is when you ship the code to the client through the wire (such as a Javascript bundle).
- Retric 6y ago
  You can’t use a library with zero lines of code. On top of this library’s always have development overhead outside of the code you write. Ex: What version number should you use? Did the latest version break something? Did the old version break something on the latest compiler? Etc etc.
- Ma8ee 6y ago
  It’s cheaper in the sense that it is faster to write and maintain those 20 lines of code. Because someone has to evaluate the library, understand it well enough to actually call it and then make sure it stays up to date. And often there are a few lines of code to translate your data into a form that the library requires etc.
  - BigJono 6y ago
    Plus for every developer to come, one call to an external library usually also means 30 pages of documentation to trawl through if they ever want to change anything, 29.9 of which is completely irrelevant to whatever your narrow use case is. That's the real cost. The size of the code means absolutely nothing.
benibela 6y ago
xml and json are different beasts A json parser can probably be implemented in an afternoon. But a conformant xml parser can take months. There are some weird things in xml, for example, this is correct xml: <?xml version="1.0"?><!DOCTYPE abc[<?abc >]]<abc><abc/>]>>?>]><?x?><a/><?x <(x)>>?> And for external entities, you need an http client in the xml parser, although it is probably better to not support that part of the standard.
- Ma8ee 6y ago
  And we didn’t need a conformant xml parser, which I know is huge and complex.

tnorthcutt 6y ago

Indeed: https://github.com/qntm/fastjson https://github.com/qntm/fastjson

flukus 6y ago

And it's going to take more than an afternoon to evaluate these parsers. You have to look at the options, evaluate the API, evaluate if they're stable and supported, evaluate if they integrate well with you're project, evaluate any dependencies they might have, etc. Then you need a plan to manage these dependencies long term. If you're needs can be solved adequately by strtok then that's a far simpler and more maintainable solution that can be knocked out in an afternoon.