6 ms·

Yes, it’s unusual to get an email with more than a couple dozen attachments, but that’s not for the parser to decide. Hard-coding a limit like is fundamentally

by 1f60c 6y ago

Yes, it’s unusual to get an email with more than a couple dozen attachments, but that’s not for the parser to decide. Hard-coding a limit like is fundamentally the wrong design. A library should be able to parse anything you throw at it, and if that’s an email with 10,000 attachments, it should parse it correctly. Sooner or later, it’ll run out of memory, of course, but that’s my problem. A better design would’ve been something passing the limit as a configuration object, something like: { attachmentLimit?: number | boolean }

megous 6y ago
Better design would be not allocating any memory based on the message content and do parsing with callbacks or iterator approach.
- stingraycharles 6y ago
  Depends on what your goals are as an API designer. If you want a fast and scalable approach, your suggestion would indeed be most appropriate. On the other hand, most users are likely best served with a parser that just returns a single object, which they then can interact with as they desire.
  - indentit 6y ago
    Could that not be easily done as an add on that utilizes the callback approach? The parser library could offer both APIs and the caller could decide which to use and what to do when running out of memory etc.
    - deleted 6y ago
      [deleted]
jeffbee 6y ago
Eh, there's a limit where it is not reasonable to continue parsing. I worked on the mailer for a huge webmail service, and it will at a certain point stop considering your inputs and respond with SMTP 5xx. Its just about knowing the limits of your software. If you knowingly write a recursive descent MIME parser then you have to also reason about the limits of how many stack frames you are permitted, or whatever limitations your language and runtime have. It's not OK to write such a parser and then just obliviously accept all inputs.
- xiconfjs 6y ago
  We had a similar problem at my last company, where out boss sent a mail with his trip photos to the access limited everyone@ list - so our mailserver had a good 2 hours downtime while filling 2k email boxes with 20MB of attachments...what a mess
jolmg 6y ago
I wonder if the lazy nature of Haskell's Attoparsec parsing library would have prevented the issue by default (without needing to set-up a limit in the parser). I think it depends on how the resulting lazy structure is used, but it might be true that the person writing the parser may not need to worry in the least about it, since evaluation and resulting memory allocation would be up to the person that uses the result from the parser because of the language's lazy-by-default nature.
jorangreef 6y ago
There are things that the parser must necessarily decide because nobody higher up the stack will understand them or think to limit them and override the default, even if a configuration object were to be exposed. For example, one variant of this kind of MIME bomb would be defeated by a limit on the number of RFC 2231 parameter continuations acceptable per header. Even if you were to allow the user to override the default limit as you propose, your library would still need a sane default, because you would probably find that not many people would see RFC 2231 parameter continuations as a threat vector or even think to specify a limit, let alone know what a sane limit would be. In the end, to be secure, you would have to expose so many limits that just using the library would be a mess. As soon as you say "that's not for the parser to decide" you run the risk of things falling through the gap and no policy decisions being made at all. You also have to accept that sometimes the parser designer alone knows best the complexity analysis of the resources the library will need.

by 1f60c 6y ago

megous 6y ago

Better design would be not allocating any memory based on the message content and do parsing with callbacks or iterator approach.

stingraycharles 6y ago
Depends on what your goals are as an API designer. If you want a fast and scalable approach, your suggestion would indeed be most appropriate. On the other hand, most users are likely best served with a parser that just returns a single object, which they then can interact with as they desire.
- indentit 6y ago
  Could that not be easily done as an add on that utilizes the callback approach? The parser library could offer both APIs and the caller could decide which to use and what to do when running out of memory etc.
  - deleted 6y ago
    [deleted]

jeffbee 6y ago

Eh, there's a limit where it is not reasonable to continue parsing. I worked on the mailer for a huge webmail service, and it will at a certain point stop considering your inputs and respond with SMTP 5xx. Its just about knowing the limits of your software. If you knowingly write a recursive descent MIME parser then you have to also reason about the limits of how many stack frames you are permitted, or whatever limitations your language and runtime have. It's not OK to write such a parser and then just obliviously accept all inputs.

xiconfjs 6y ago
We had a similar problem at my last company, where out boss sent a mail with his trip photos to the access limited everyone@ list - so our mailserver had a good 2 hours downtime while filling 2k email boxes with 20MB of attachments...what a mess

jolmg 6y ago

I wonder if the lazy nature of Haskell's Attoparsec parsing library would have prevented the issue by default (without needing to set-up a limit in the parser). I think it depends on how the resulting lazy structure is used, but it might be true that the person writing the parser may not need to worry in the least about it, since evaluation and resulting memory allocation would be up to the person that uses the result from the parser because of the language's lazy-by-default nature.

jorangreef 6y ago

There are things that the parser must necessarily decide because nobody higher up the stack will understand them or think to limit them and override the default, even if a configuration object were to be exposed. For example, one variant of this kind of MIME bomb would be defeated by a limit on the number of RFC 2231 parameter continuations acceptable per header. Even if you were to allow the user to override the default limit as you propose, your library would still need a sane default, because you would probably find that not many people would see RFC 2231 parameter continuations as a threat vector or even think to specify a limit, let alone know what a sane limit would be. In the end, to be secure, you would have to expose so many limits that just using the library would be a mess. As soon as you say "that's not for the parser to decide" you run the risk of things falling through the gap and no policy decisions being made at all. You also have to accept that sometimes the parser designer alone knows best the complexity analysis of the resources the library will need.