3 ms·
> LineReader splits input into lines, handles \n and \r\n, and trims the stray trailing \r that malformed input likes to leave behind Is there a common source
by Retr0id 2mo ago
> LineReader splits input into lines, handles \n and \r\n, and trims the stray trailing \r that malformed input likes to leave behind
Is there a common source of extra \r in malformed inputs, beyond those existing as part of \r\n? Or is this just a dig at Windows-style line endings? If there's something weird going on I think I'd rather fail loudly.
> Bounding the inner scanner to a single line makes “run past the end of a malformed line” unrepresentable rather than merely unlikely.
I don't really see what makes it "unrepresentable", and this reads more like "if you used the right scanning logic, you can't have used the wrong scanning logic".
- inigyou 2mo agoSure, start with \r\n, split on \n, now you have a stray \r at the end of every input.
- Retr0id 2mo agoBut the preceding clause says it handles \r\n. If you're already handling \r\n, what remaining sources of \r are there, that you'd actually want to silently ignore?
- HackerThemAll 2mo ago\r\n?|\n handles all EOL sequences without backtracking. Or write a non-regex equivalent of that.
- inigyou 2mo agoI'm sure every time you split something on newlines you remember to use a regex.
- HackerThemAll 2mo agoI tend not to, because it's an overkill, but this regex nicely sums what needs to be done.
- thesz 2mo agoEnd of line on Classic Mac is \r.
- dspillett 2mo ago> Is there a common source of extra \r in malformed inputs, beyond those existing as part of \r\n? Old Macs and some other systems use \r as their EOL, I still sometimes see that with string values in CSV files y code has to deal with (though I don't think I've seen it as an EOL marker in the format itself for a _long_ time). Sometimes incorrect cleaning steps can leave them behind, such as replacing \r\n with \n but that replacement not being global: it tests fine on strings with zero or one \r\n but subsequent ones will retain their \r. Also code splitting on \n assuming it will always see just that as EOLs will leave trailing \r characters in place. Also, code cleaning EOLs from strings that are supposed to be one-line-only may replace \n (or \n or \r\n, ignoring the possibility of just \r) with a space or a comma and a space, that could be where the \r characters in certain string values I see in files from clients are coming from. I suspect that off-by-one errors caused by character counting bugs in UTF8/UTF16 handling may cause splitting on EOLs to be a bit off in some cases, though here you will probably be seeing other data corruption at the same time and an errant \r is one of your smaller problems.