4 ms·

JSON extra uses orjson instead of ujson (2019)

69 points by coldcache 2y ago

coldcache 2y ago
Ran into this PR today. Thought Samuel Colvin's response to the migration request was prescient, especially considering what we later saw with the XZ Utils backdoor. I've no clue if there actually were/are any problems with orjson, but I admire this kind of dedication to security, especially years ago.
- jay-barronville 2y ago
  When you consider how large many of these open-source ecosystems are and the sheer number of contributors who go completely unnoticed, I think it’s pretty much guaranteed that there’s some software we’re all using or relying on that’s been compromised and the vulnerability has gone completely unnoticed. As an example, think about how many packages get downloaded from npm every single hour without even a single thought by the person downloading it, or in most cases, without even their direct knowledge—what are the odds that they’ve downloaded malware onto their systems? I remember watching a TypeScript-related livestream on YouTube and the guy installed the wrong npm package (due to a typo), which had a postinstall script. He pretty quickly realized it was the wrong package, but only after it downloaded and ran a script on his system. It turned out the package was harmless, but it’s just so easy to harm a large number of users nowadays if you can just find the right target, which is why package maintainers are in such a dangerous position, in my opinion. Sadly, I think the XZ Utils backdoor is just the one that got noticed.
- arp242 2y ago
  It wasn't "prescient" at all because in the five years since this discussion nothing happened with orjson. Casting aspersions on someone based on a five year old discussion with no evidence whatsoever by referring to a completely unrelated incident is ... not brilliant. This is the HN variant of "Twitter outrage" over some innocent five year old Tweet.
comex 2y ago
(2019)
- coldcache 2y ago
  Just edited the title to reflect that
cqqxo4zV46cp 2y ago
I definitely appreciate this degree of rigour.
omh1280 2y ago
I’ve also been wary of orjson considering ijl is anonymous and the only one authoring commits. Any ideas on if security folks are checking repository artifacts and verifying builds for projects like this?
3r7j6qzi9jvnve 2y ago
I'm really surprised ijl got angry that his mail was quoted, it looks innocent enough to me. For reference it's been edited out here: https://github.com/pydantic/pydantic/issues/589 https://github.com/pydantic/pydantic/issues/589 But github shows edits, so the edit is meaningless for privacy. Here's the original mail (yes, I'm blatantly ignoring his request to not publish this, I'm just this evil.) I've looked into replacing ujson in pydantic with orjson (https://github.com/ijl/orjson). In this implementation, the same JSON library is used for everything, and JSON outputs bytes without whitespace (as it's faster and JSON is a serialization format). If orjson is installed, it won't affect pydantic's benchmark for validation, but can be expected to improve whole-program performance. It's a large change with breaking changes to JSON methods, however, so rather than opening a pull request now, could you take a look and see if that's consistent and acceptable to the project? https://github.com/ijl/pydantic/commit/7c08f41edd340614d7c58888f025665dbc71d0e3 That passes tests, but that's all. I'll clean it up or modify if the idea's acceptable. Thanks.
- arp242 2y ago
  I've had people get angry at me for "quoting emails", even when it's small quotes from completely innocent non-private stuff like this. I guess it's a matter of principle for some shrug
meowface 2y ago
Has anyone done an analysis of it? I've used orjson in all my Python projects for years.
- gregw2 2y ago
  There is a orjson CVE recently, CVE-2024-27454 based on one person’s analysis: https://monicz.dev/CVE-2024-27454 https://monicz.dev/CVE-2024-27454 That is not necessarily the result of a systematic review of course.
- ranger_danger 2y ago
  I've never heard of either of these and just always used the built-in json library. Is there a great reason to add more technical debt to my projects?
  - meowface 2y ago
    It's a drop-in replacement that's orders of magnitude faster. Barring security/bug concerns, in my opinion there's no reason not to use it.
noitpmeder 2y ago
Hard to argue with his logic, especially re the fact that pydantic is used very widely by large organizations. Some degree of dependency visibility (i.e. non binary releases, >1 contributor, publicly attributable maintainer) is a good thing.
jay-barronville 2y ago
I’d be willing to bet that Samuel was perceived as a jerk by some for even implying that this contributor was a bit suspicious, yet it was the most honorable position to take as a maintainer so many folks are relying on and trusting, both directly and indirectly. Job well done.
ranger_danger 2y ago
Can someone ELI5 why this is news? I'm just not following...

69 points by coldcache 2y ago

Ran into this PR today. Thought Samuel Colvin's response to the migration request was prescient, especially considering what we later saw with the XZ Utils backdoor. I've no clue if there actually were/are any problems with orjson, but I admire this kind of dedication to security, especially years ago.

jay-barronville 2y ago
When you consider how large many of these open-source ecosystems are and the sheer number of contributors who go completely unnoticed, I think it’s pretty much guaranteed that there’s some software we’re all using or relying on that’s been compromised and the vulnerability has gone completely unnoticed. As an example, think about how many packages get downloaded from npm every single hour without even a single thought by the person downloading it, or in most cases, without even their direct knowledge—what are the odds that they’ve downloaded malware onto their systems? I remember watching a TypeScript-related livestream on YouTube and the guy installed the wrong npm package (due to a typo), which had a postinstall script. He pretty quickly realized it was the wrong package, but only after it downloaded and ran a script on his system. It turned out the package was harmless, but it’s just so easy to harm a large number of users nowadays if you can just find the right target, which is why package maintainers are in such a dangerous position, in my opinion. Sadly, I think the XZ Utils backdoor is just the one that got noticed.
arp242 2y ago
It wasn't "prescient" at all because in the five years since this discussion nothing happened with orjson. Casting aspersions on someone based on a five year old discussion with no evidence whatsoever by referring to a completely unrelated incident is ... not brilliant. This is the HN variant of "Twitter outrage" over some innocent five year old Tweet.

comex 2y ago

(2019)

coldcache 2y ago
Just edited the title to reflect that

cqqxo4zV46cp 2y ago

I definitely appreciate this degree of rigour.

omh1280 2y ago

I’ve also been wary of orjson considering ijl is anonymous and the only one authoring commits. Any ideas on if security folks are checking repository artifacts and verifying builds for projects like this?

3r7j6qzi9jvnve 2y ago

I'm really surprised ijl got angry that his mail was quoted, it looks innocent enough to me. For reference it's been edited out here: https://github.com/pydantic/pydantic/issues/589 https://github.com/pydantic/pydantic/issues/589 But github shows edits, so the edit is meaningless for privacy. Here's the original mail (yes, I'm blatantly ignoring his request to not publish this, I'm just this evil.) I've looked into replacing ujson in pydantic with orjson (https://github.com/ijl/orjson). In this implementation, the same JSON library is used for everything, and JSON outputs bytes without whitespace (as it's faster and JSON is a serialization format). If orjson is installed, it won't affect pydantic's benchmark for validation, but can be expected to improve whole-program performance. It's a large change with breaking changes to JSON methods, however, so rather than opening a pull request now, could you take a look and see if that's consistent and acceptable to the project? https://github.com/ijl/pydantic/commit/7c08f41edd340614d7c58888f025665dbc71d0e3 That passes tests, but that's all. I'll clean it up or modify if the idea's acceptable. Thanks.

arp242 2y ago
I've had people get angry at me for "quoting emails", even when it's small quotes from completely innocent non-private stuff like this. I guess it's a matter of principle for some shrug

meowface 2y ago

Has anyone done an analysis of it? I've used orjson in all my Python projects for years.

gregw2 2y ago
There is a orjson CVE recently, CVE-2024-27454 based on one person’s analysis: https://monicz.dev/CVE-2024-27454 https://monicz.dev/CVE-2024-27454 That is not necessarily the result of a systematic review of course.
ranger_danger 2y ago
I've never heard of either of these and just always used the built-in json library. Is there a great reason to add more technical debt to my projects?
- meowface 2y ago
  It's a drop-in replacement that's orders of magnitude faster. Barring security/bug concerns, in my opinion there's no reason not to use it.

noitpmeder 2y ago

Hard to argue with his logic, especially re the fact that pydantic is used very widely by large organizations. Some degree of dependency visibility (i.e. non binary releases, >1 contributor, publicly attributable maintainer) is a good thing.

jay-barronville 2y ago

I’d be willing to bet that Samuel was perceived as a jerk by some for even implying that this contributor was a bit suspicious, yet it was the most honorable position to take as a maintainer so many folks are relying on and trusting, both directly and indirectly. Job well done.

ranger_danger 2y ago

Can someone ELI5 why this is news? I'm just not following...