3 ms·
> So how do you read a line from a text file? In that case you need to know upfront how the text file is encoded anyway, since text files don't carry that info
by flohofwoe 3y ago
> So how do you read a line from a text file?
In that case you need to know upfront how the text file is encoded anyway, since text files don't carry that information around.
If it is a byte-stream encoding from the "ASCII heritage" like UTF-8, 7-bit ASCII, or codepaged 8-bit "ASCII" - whatever that is actually called...): load bytes until you encounter a 0x0A or 0x0D (and skip those when continuing), what has been loaded until then is a line in the text file's encoding. If the original encoding was codepaged 8-bit ASCII you probably want to convert that to UTF-8 next, for that you also need to know the proper codepage though (not needed for 7-bit ASCII since that already is valid UTF-8 - in UTF-8, every byte with the topmost bit cleared is guaranteed to be a standalone 7-bit ASCII character and every byte with the topmost bit set is part of a multi-byte sequence for codepoints above 127, that's why one can simply iterate byte by byte over an UTF-8 encoded byte stream when looking for 7-bit ASCII characters (such as newline and carriage-return).
The gist is that the file IO functions themselves should never be aware of text encodings, they should only work on bytes. The "text awareness" should happen in higher level code above the IO layer.
> Are we talking about the same C?
What I meant here - but expressed poorly - was that C also got that wrong (or rather the C stdlib, C itself isn't involved). There should be no "text mode IO" in the C stdlib IO functions either, only raw byte IO. And functions like gets(), fgets() etc... shouldn't be in the C stdlib in the first place.
- amluto 3y agoPython 3 actually works approximately the way you’re describing: https://docs.python.org/3/library/io.html#io.TextIOWrapper https://docs.python.org/3/library/io.html#io.TextIOWrapper open is just a factory function, conceptually inherited (I think) from C.