6 ms·

Defusedxml – defusing XML bombs and other exploits

92 points by gudzpoz 2y ago

slau 2y ago
DefusedXML is an amazing piece of code. This being said, many of the mitigations it enables are now also available by default in many “standard” libraries. For example, bandit will often tell you to not use lxml in Python, but instead use defusedxml. However, modern versions don’t suffer the same issues at all, and this is a case where automatically following the advice of the linter/SCA is not a great idea.
- metafunctor 2y ago
  Do you mean that it is, in fact, a mistake to use defusedxml instead of lxml in Python?
  - slau 2y ago
    From the author themselves, 6 years ago: > defusedxml.lxml is no longer needed and supported. Nowadays libxml2 has builtin limitation for entity expansion. https://github.com/tiran/defusedxml/issues/25#issuecomment-422436105 https://github.com/tiran/defusedxml/issues/25#issuecomment-4...
    - masklinn 2y ago
      Note that this is not enabled by default, although there is an upper bound on tree size which does limit the reach of the issue. See https://lxml.de/FAQ.html#is-lxml-vulnerable-to-xml-bombs https://lxml.de/FAQ.html#is-lxml-vulnerable-to-xml-bombs for more about the tuning knobs.
    - JonChesterfield 2y ago
      libxml2 segfaults on me whenever I give it vaguely complicated xsl templates so I'm doubtful about how effective that handling will be.
    - metafunctor 2y ago
      OK, so the defusedxml.lxml submodule is deprecated and one should use the other APIs from defusedxml instead. That does not mean that defusedxml in it's entirety would be useless.
  - masklinn 2y ago
    If you’re trying to use it for lxml then yes, it was only ever experimental and has been deprecated (it also failed to define some interfaces correctly causing issues). If you’re using it over the stdlib then no.
mjfisher 2y ago
Fascinating reading: > The majority of developers are unacquainted with features such as processing instructions and entity expansions that XML inherited from SGML. At best they know about <!DOCTYPE> from experience with HTML but they are not aware that a document type definition (DTD) can generate an HTTP request or load a file from the file system. I was one of them!
- tannhaeuser 2y ago
  Developers are even less aware that SGML has (and always had) quantities in the SGML declaration, allowing among other things to restrict the nesting/expansion level of entities (and hence to counter EE attacks without resorting to heuristics). Regarding DOCTYPE and DTDs, browsers at best made use of those to switch into or out of "quirks mode", on seeing special hardcoded public identifiers but ignored any declarations. WHATWG's cargo cult "<!DOCTYPE html>" is just telling an SGML parser that the "internal and external subset is empty", meaning there are no markup declarations necessary to parse HTML which is of course bogus when HTML makes abundant use of empty elements (aka void/self-closing elements in HTML parlance), tag omission, attribute shortforms, and other features that need per-element declarations for parsing. Btw that's what defines the XML subset of SGML: that XML can always be parsed without a DTD, unlike HTML or other vocabularies making use of above stated features. Keep in mind SGML is a markup language for text authoring, and it would be pretty lame for a markup language to not have text macros (entities). In fact, the lack of such a basic feature is frequently complained about in browsers. The problems came when people misused XML for service payloads or other generic data exchange. Note SOAP did forbid DTDs, and stacks checked for presence of DTDs in payloads. That said, XML and XML Schema with extensive types for money/decimals, dates, hashes, etc. is heavily used in eg ISO 20022 payments and other financial messages, and to this date, there hasn't evolved a single competitor with the same coverage and scope (with the potential exception of ASN.1 which is even older and certainly more baroque).
  - bawolff 2y ago
    > Regarding DOCTYPE and DTDs, browsers at best made use of those to switch into or out of "quirks mode", on seeing special hardcoded public identifiers but ignored any declarations. Not when processing XML mime types. In modern browsers that mostly means SVG files, but i think XHTML is still possible. (Modern) HTML is neither SGML nor XML, so it doesn't follow the rules of either.
Lance_ET_Compte 2y ago
Does `lxml` match `etree` in the table?
- masklinn 2y ago
  No. etree is xml.etree.ElementTree.
move-on-by 2y ago
I’ve always appreciated their drop-in replacement support. It’s so nice to just change an import and move on. I’ve used it on multiple legacy projects with great success- never a single compatibility issue. Great project!
redbell 2y ago
> XML Bomb This reminds me of Zip Bomb [1], aka, Zip of Death (ZOD) [2] 1. https://en.m.wikipedia.org/wiki/Zip_bomb https://en.m.wikipedia.org/wiki/Zip_bomb 2. https://github.com/iamtraction/ZOD https://github.com/iamtraction/ZOD
exabrial 2y ago
Back in one day… I saw my first xml/html exploit by another script kiddie on AOL. I was like wtf and began trying to crash my own account. A simple quote mismatch with a bunch of ampersands eventually did it (something 1000 nested ampersand escape sequences). Anyways was pretty proud of what I built, so I found a guy smarting off in an AOL chat and decided to bomb him. And yeah, he was a moderator for AOL… so at 12 years old I got our family amount blocked and my dad had a uncomfortable call with the AOL reps and how his son was hacking. Ah those were the days
- deleted 2y ago
  [deleted]
hannob 2y ago
This is largely historic. I had lengthy discussions about this with expat's maintainer. expat, the xml library underlying python's etree and other xml interfaces, has either mitigated these standard xml vulnerabilities or disables the dangerous features by default. The python docs are still a bit confusing there, but if you look at this table: https://docs.python.org/3/library/xml.html#xml-vulnerabilities https://docs.python.org/3/library/xml.html#xml-vulnerabiliti... While this table has a lot of "Vulnerable" in it, they all come with footnotes saying that up-to-date versions of expat are not vulnerable. So... if you want to have more secure xml parsing in python, make sure you use an up-to-date expat library or one where security fixes have been backported. You don't need anything else.

92 points by gudzpoz 2y ago

slau 2y ago

DefusedXML is an amazing piece of code. This being said, many of the mitigations it enables are now also available by default in many “standard” libraries. For example, bandit will often tell you to not use lxml in Python, but instead use defusedxml. However, modern versions don’t suffer the same issues at all, and this is a case where automatically following the advice of the linter/SCA is not a great idea.

metafunctor 2y ago
Do you mean that it is, in fact, a mistake to use defusedxml instead of lxml in Python?
- slau 2y ago
  From the author themselves, 6 years ago: > defusedxml.lxml is no longer needed and supported. Nowadays libxml2 has builtin limitation for entity expansion. https://github.com/tiran/defusedxml/issues/25#issuecomment-422436105 https://github.com/tiran/defusedxml/issues/25#issuecomment-4...
  - masklinn 2y ago
    Note that this is not enabled by default, although there is an upper bound on tree size which does limit the reach of the issue. See https://lxml.de/FAQ.html#is-lxml-vulnerable-to-xml-bombs https://lxml.de/FAQ.html#is-lxml-vulnerable-to-xml-bombs for more about the tuning knobs.
  - JonChesterfield 2y ago
    libxml2 segfaults on me whenever I give it vaguely complicated xsl templates so I'm doubtful about how effective that handling will be.
  - metafunctor 2y ago
    OK, so the defusedxml.lxml submodule is deprecated and one should use the other APIs from defusedxml instead. That does not mean that defusedxml in it's entirety would be useless.
- masklinn 2y ago
  If you’re trying to use it for lxml then yes, it was only ever experimental and has been deprecated (it also failed to define some interfaces correctly causing issues). If you’re using it over the stdlib then no.

mjfisher 2y ago

Fascinating reading: > The majority of developers are unacquainted with features such as processing instructions and entity expansions that XML inherited from SGML. At best they know about <!DOCTYPE> from experience with HTML but they are not aware that a document type definition (DTD) can generate an HTTP request or load a file from the file system. I was one of them!

tannhaeuser 2y ago
Developers are even less aware that SGML has (and always had) quantities in the SGML declaration, allowing among other things to restrict the nesting/expansion level of entities (and hence to counter EE attacks without resorting to heuristics). Regarding DOCTYPE and DTDs, browsers at best made use of those to switch into or out of "quirks mode", on seeing special hardcoded public identifiers but ignored any declarations. WHATWG's cargo cult "<!DOCTYPE html>" is just telling an SGML parser that the "internal and external subset is empty", meaning there are no markup declarations necessary to parse HTML which is of course bogus when HTML makes abundant use of empty elements (aka void/self-closing elements in HTML parlance), tag omission, attribute shortforms, and other features that need per-element declarations for parsing. Btw that's what defines the XML subset of SGML: that XML can always be parsed without a DTD, unlike HTML or other vocabularies making use of above stated features. Keep in mind SGML is a markup language for text authoring, and it would be pretty lame for a markup language to not have text macros (entities). In fact, the lack of such a basic feature is frequently complained about in browsers. The problems came when people misused XML for service payloads or other generic data exchange. Note SOAP did forbid DTDs, and stacks checked for presence of DTDs in payloads. That said, XML and XML Schema with extensive types for money/decimals, dates, hashes, etc. is heavily used in eg ISO 20022 payments and other financial messages, and to this date, there hasn't evolved a single competitor with the same coverage and scope (with the potential exception of ASN.1 which is even older and certainly more baroque).
- bawolff 2y ago
  > Regarding DOCTYPE and DTDs, browsers at best made use of those to switch into or out of "quirks mode", on seeing special hardcoded public identifiers but ignored any declarations. Not when processing XML mime types. In modern browsers that mostly means SVG files, but i think XHTML is still possible. (Modern) HTML is neither SGML nor XML, so it doesn't follow the rules of either.

Lance_ET_Compte 2y ago

Does `lxml` match `etree` in the table?

masklinn 2y ago
No. etree is xml.etree.ElementTree.

move-on-by 2y ago

I’ve always appreciated their drop-in replacement support. It’s so nice to just change an import and move on. I’ve used it on multiple legacy projects with great success- never a single compatibility issue. Great project!

redbell 2y ago

> XML Bomb This reminds me of Zip Bomb [1], aka, Zip of Death (ZOD) [2] 1. https://en.m.wikipedia.org/wiki/Zip_bomb https://en.m.wikipedia.org/wiki/Zip_bomb 2. https://github.com/iamtraction/ZOD https://github.com/iamtraction/ZOD

exabrial 2y ago

Back in one day… I saw my first xml/html exploit by another script kiddie on AOL. I was like wtf and began trying to crash my own account. A simple quote mismatch with a bunch of ampersands eventually did it (something 1000 nested ampersand escape sequences). Anyways was pretty proud of what I built, so I found a guy smarting off in an AOL chat and decided to bomb him. And yeah, he was a moderator for AOL… so at 12 years old I got our family amount blocked and my dad had a uncomfortable call with the AOL reps and how his son was hacking. Ah those were the days

deleted 2y ago
[deleted]

hannob 2y ago

This is largely historic. I had lengthy discussions about this with expat's maintainer. expat, the xml library underlying python's etree and other xml interfaces, has either mitigated these standard xml vulnerabilities or disables the dangerous features by default. The python docs are still a bit confusing there, but if you look at this table: https://docs.python.org/3/library/xml.html#xml-vulnerabilities https://docs.python.org/3/library/xml.html#xml-vulnerabiliti... While this table has a lot of "Vulnerable" in it, they all come with footnotes saying that up-to-date versions of expat are not vulnerable. So... if you want to have more secure xml parsing in python, make sure you use an up-to-date expat library or one where security fixes have been backported. You don't need anything else.