4 ms·
Python's splitlines does more than just newlines
- 7bit 1y agoThis article provides no additional value to the splitlines() docs.
- diath 1y agoThe value of this article, to me, is that I'd never read the splitlines documentation, so this is a little detail that I just learned thanks to it being linked here.
- rsyring 1y agoSometimes value is measured by awareness. I benefited from becoming aware of the behavior because of the article. Yes, it's in the docs, but the docs are not something I would have gone looking to read today.
- woodruffw 1y agoThe "article" is my TIL mini-blog. What were you expecting besides a "today I learned"?
- kstrauser 1y agoI already knew this information, more or less, but I like reading TIL posts like this. It's fun seeing the someone learn new things, and sometimes I pick up something myself, or at least look at it in a new way.
- cap11235 1y agoYeah, don't listen to parent. I like these sorts of articles a lot; its only useless if you assume that everyone interested has also memorized the Python docs fully (which I imagine is zero people). Fun technical tangents are quite fun indeed.
- zahlman 1y agoWhat is "yossarian", BTW? I'd gotten confused thinking it was someone else's blog, because I naturally parse that as a surname.
- woodruffw 1y agoJohn Yossarian is the protagonist of Joseph Heller’s Catch-22[1], which was my favorite book in high school. Like a lot of people, my handle is a slightly embarrassing memorialization of my younger self :-) [1]: https://en.wikipedia.org/wiki/Catch-22 https://en.wikipedia.org/wiki/Catch-22
- felipelemos 1y agoFor all of us that don't read all documentation for every single method, tool, function or similar, it is, by awarenes, very useful.
- happytoexplain 1y agoI've been working with Python for a year or so now, and never knew this. I'm grateful to the author.
- dleeftink 1y agoFor more controlled splitting, I really like Unicode named characters classes[0] for more precise splitting and matching tasks. [0]: https://en.wikipedia.org/wiki/Unicode_character_property#General_Category https://en.wikipedia.org/wiki/Unicode_character_property#Gen...
- Rendello 1y agoGiven that encoded characters must have one and only one General_Category value, it might be too imprecise or arbitrary in some cases. If you ever need more power, it's worth browsing the other character properties Unicode exposes. For example, `Lu` (Uppercase_Letter) only covers some uppercase letters, whereas the `Uppercase` property covers all of them. --- For anyone that wants to learn more about specific Unicode stuff, the three big data sources are The Core Spec, the Unicode Technical Annexes (UAXs), and the Unicode Character Database itself (the database is a bunch of text files. There's an XML version now as well). For further reading on this specifically, it might be worth looking at: [Unicode Core Spec - Chapter 4: Character Properties] https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-4/ https://www.unicode.org/versions/Unicode17.0.0/core-spec/cha... ├ [General Category] https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-4/#G124142 https://www.unicode.org/versions/Unicode17.0.0/core-spec/cha... └ [Properties for Text Boundaries] https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-4/#G124804 https://www.unicode.org/versions/Unicode17.0.0/core-spec/cha... [UAX #44 - Unicode Character Database (Technical Report)] https://www.unicode.org/reports/tr44/ https://www.unicode.org/reports/tr44/ ├ [General Category Values] https://www.unicode.org/reports/tr44/#General_Category_Values https://www.unicode.org/reports/tr44/#General_Category_Value... └ [Property Definitions] https://www.unicode.org/reports/tr44/#Property_Definitions https://www.unicode.org/reports/tr44/#Property_Definitions And, if you're brave and want to see the data itself (skim through UAX #44 first): [Unicode Character Database] https://www.unicode.org/Public/17.0.0/ucd/ https://www.unicode.org/Public/17.0.0/ucd/
- sundarurfriend 1y agoUnfortunately, even Properties aren't fully reliable. The Tamil pulli for eg., does not have the Alphabetic property despite being a central part of the Tamil alphabet, pretty much due to historical accident: other Indic languages have a similar looking Virama character that aren't alphabetic, and pulli (despite being a separate, unrelated character) was lumped in with those. When I tried to raise this in the mailing list and get it rectified, the response I got was pretty much that many properties need language-specific processing anyway, and shouldn't be relied upon fully, and so this wasn't worth fixing. [1] https://util.unicode.org/UnicodeJsps/character.jsp?a=0BCD&B1=Show https://util.unicode.org/UnicodeJsps/character.jsp?a=0BCD&B1...
- wvbdmp 1y agoWhat, no <br\s*\/?>?
- meken 1y agoTIL: Python has a splitlines function
- Frotag 1y agoThere's so many super useful things in the Python docs that you never see in the wild. For example, I recently learned that the sqlite3 module has a set_authorizer function that lets you limit the types of statements that can be run / tables that can be accessed. https://www.sqlite.org/c3ref/set_authorizer.html https://www.sqlite.org/c3ref/set_authorizer.html https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.set_authorizer https://docs.python.org/3/library/sqlite3.html#sqlite3.Conne...
- mixmastamyk 1y agoSplitlines is generally not needed. for line in file: is more idiomatic.
- tiltowait 1y agoSplitlines additionally strips the newline character, functionality which is often (maybe even usually?) desired.
- masklinn 1y agoThis has been controlled via a boolean parameter since at least 2.0, which as far as I can tell is when this method was added to `str`.
- drdrey 1y agonot every line is read from a file
- mixmastamyk 1y agoThat's where the generally fits in.
- crazygringo 1y agoNo, because that still assumes files are the general usage. In my experience, they're not. It's strings.
- mixmastamyk 1y agoAnd where do you get these input strings? Big enough that .split() is not sufficient? Files, and yes sockets support the interface as well with a method call.
- crazygringo 1y ago> And where do you get these input strings? From database fields, API calls, JSON values, HTML tag content, function inputs generally, you know -- the normal places. In my experience, most people aren't dealing directly with files (or streams) most of the time.
- cuckoos-jicamas 1y agostr.split() function does the same: >>> s = "line1\nline2\rline3\r\nline4\vline5\x1dhello" >>> s.split() ['line1', 'line2', 'line3', 'line4', 'line5', 'hello'] >>> s.splitlines() ['line1', 'line2', 'line3', 'line4', 'line5', 'hello'] But split() has sep argument to define delimiter according which to split the string.. In which case it provides what you expected to happen: >>> s.split('\n') ['line1', 'line2\rline3\r', 'line4\x0bline5\x1dhello'] In general you want this: >>> linesep_splitter = re.compile(r'\n|\r\n?') >>> linesep_splitter.split(s) ['line1', 'line2', 'line3', 'line4\x0bline5\x1dhello']
- gertlex 1y agosplitlines() is sometimes nice for adhoc parsing (of well behaved stuff...) because it throws out whitespace-only lines from the resulting list of strings. #1 use-case of that for me is probably just avoiding the cases where there's a trailing newline character in the output of a command I ran by subprocess.
- roelschroeven 1y agoIn that example str.split() has the same result as str.splitlines(), but it's not in general the same, even without custom delimiter. str.split() splits on runs of consecutive whitespace, any type of whitespace, including tabs and spaces which splitlines() doesn't do. >>> 'one two'.split() ['one', 'two'] >>> 'one two'.splitlines() ['one two'] split() without custom delimiter also splits on runs of whitespace, which splitline() also doesn't do (except for \r\n because that combination counts as one line ending): >>> 'one\n\ntwo'.split() ['one', 'two'] >>> 'one\n\ntwo'.splitlines() ['one', '', 'two']
- zzzeek 1y agoin the same theme, NTLAIL strip(), rstrip(), lstrip() can strip other kinds of characters besides whitespace.
- masklinn 1y agoOne thing to note tho is that they take character sets, as long as they encounter characters in the specified set they will keep stripping. Lots of people think if you give it a string it will remove that string. That feature was added in 3.9 with the addition of `removeprefix` and `removesuffix`. Sadly, 1. unlike Rust's version they provide no way of knowing whether they stripped things out 2. unlike startswith/endswith they do not take tuples of prefixes/suffixes
- zb3 1y agoUseful to know for security purposes, surprises like that might cause vulnerabilities..
- deleted 1y ago[deleted]
- RainyDayTmrw 1y agoIs there a parser ambiguity/confusion vector here?