3 ms·
Yes, line splitting is built-in. But this is more about split versus tokenize. I have split and tokenize functions which have the same interface. Split places
by kazinator 3y ago
Yes, line splitting is built-in. But this is more about split versus tokenize.
I have split and tokenize functions which have the same interface.
Split places the focus on identifying separators: the "negative space" between what we want to keep. When we use that for lines, it has problems with edge cases, like not giving us an empty list of pieces when the string being split is empty.
- eesmith 3y agoI find it difficult to be interested in this philosophical difference between the two when only difference is in how they treat the empty string. That is, splitting with regex look-behind/look-ahead, like: (?<=\n)(?=.) with a suitable definition of dot, also matches "negative space", and produces the same results as Python's str.splitlines(True), except for the empty string. >>> import re >>> p = re.compile(r"(?<=\n)(?=.)", re.DOTALL) >>> def check(s): ... lines = p.split(s) ... assert lines == s.splitlines(True), lines ... return lines ... >>> check("no newline at end of file") ['no newline at end of file'] >>> check("\nno-newline") ['\n', 'no-newline'] >>> check("line\nno-newline") ['line\n', 'no-newline'] >>> check("line\nline\n") ['line\n', 'line\n'] >>> check("\n") ['\n'] >>> check("\n\n") ['\n', '\n'] >>> check("\n\n\n") ['\n', '\n', '\n'] >>> check("") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in check AssertionError: ['']