6 ms·
Speaking of hidden Python tools, I'm a big fan of re.Scanner[0]. It's a regex-based tokenizer[1] in the `re` module, that for reasons is completely missing from
by BoppreH 3y ago
Speaking of hidden Python tools, I'm a big fan of re.Scanner[0]. It's a regex-based tokenizer[1] in the `re` module, that for reasons is completely missing from any official documentation.
You give it a pattern for each token type, and a function to be called on each match, and you get back a list of processed tokens.
Importantly, it processes the list in one pass and ensures the matches are contiguous, where a naive `re.findall` with capture groups will ignore unmatched characters. You also get a reference to the running scanner, so you can record the location of the match for reporting errors.
import re
scanner = re.Scanner([
(r"[0-9]+", lambda scanner, token:("INTEGER", int(token))),
(r"[a-z_]+", lambda scanner, token:("IDENTIFIER", token)),
(r"[,.]+", lambda scanner, token:("PUNCTUATION", token)),
(r"\s+", None), # None == skip token.
])
results, remainder = scanner.scan("45 pigeons, 23 cows, 11 spiders.")
assert not remainder
print(results)
[('INTEGER', 45),
('IDENTIFIER', 'pigeons'),
('PUNCTUATION', ','),
('INTEGER', 23),
('IDENTIFIER', 'cows'),
('PUNCTUATION', ','),
('INTEGER', 11),
('IDENTIFIER', 'spiders'),
('PUNCTUATION', '.')]
[0]: https://stackoverflow.com/a/693818/252218 https://stackoverflow.com/a/693818/252218
[1]: https://en.wikipedia.org/wiki/Lexical_analysis#Tokenization https://en.wikipedia.org/wiki/Lexical_analysis#Tokenization
- Timon3 3y agoIt seems like the discussion of whether to document it died in April of 2003: https://mail.python.org/pipermail/python-dev/2003-April/035076.html https://mail.python.org/pipermail/python-dev/2003-April/0350... Bummer, it's a cool feature, but I don't feel safe relying on undocumented features.
- bombolo 3y agoEspecially now that they are on a crusade against their own stdlib. They regret including most modules… it seems they regret making python altogether instead of sticking with C? :D
- kzrdude 3y agoThis is a typical programmer, you'll regret code as soon as you turn your back on it. :)
- NegativeLatency 3y agoOr before!
- Qem 3y ago> They regret including most modules… it seems they regret making python altogether instead of sticking with C? :D I also find it odd. Python would probably be a little known language without the huge batteries included by default. It's invaluable when you're working in a environment where you can't fully control what is installed, what is the case of most people at work. I believe this crusade endangers the language long-term.
- oefrha 3y agoYou're reading a ridiculous mischaracterization of reality (~20 deprecated modules [1] out of the ~300 PSL modules [2] is "most"?). Of course you find it odd. [1] https://peps.python.org/pep-0594/ https://peps.python.org/pep-0594/ [2] https://docs.python.org/3/library/ https://docs.python.org/3/library/
- toyg 3y agoIt's the trend that is a bit worrying. It's definitely been floated more than once that the stdlib should be fundamentally gutted. It has not become a reality yet, but there are some loud voices advocating to just chuck most modules as a principle into pypi.
- eesmith 3y ago"Trend"? Python has dropped quite a few modules over the decades. "gl", "sgi", "fl", "sunaudiodev", "audioop", "stdwin", "rotor", "poly", "whatsound", "gopherlib" .. the list goes on. Some of these were dropped with 1.6 (see "Obsolete Modules" at https://www.python.org/download/releases/1.6/ https://www.python.org/download/releases/1.6/ ). Some with 3.0 (see https://docs.python.org/3.10/whatsnew/2.6.html https://docs.python.org/3.10/whatsnew/2.6.html for then-upcoming removals.)
- matsemann 3y ago> completely missing from any official documentation To be fair, most things are missing from the official documentation. When I learned kotlin, I read through their official docs, and knew about most language features in a day. When I learned python, I constantly got surprised by things I hadn't seen come up in the docs. For instance decorators was (still is?) not mentioned at all in the official tutorial.
- BoppreH 3y agoDecorators seem to be documented now: https://docs.python.org/3/glossary.html#term-decorator https://docs.python.org/3/glossary.html#term-decorator https://docs.python.org/3/reference/compound_stmts.html#function https://docs.python.org/3/reference/compound_stmts.html#func...
- bragr 3y agoThese have always been documented. Hit the version selector: https://docs.python.org/2.7/glossary.html#term-decorator https://docs.python.org/2.7/glossary.html#term-decorator
- roelschroeven 3y agoThe tutorial is not supposed to cover all language features: "This tutorial does not attempt to be comprehensive and cover every single feature, or even every commonly used feature. Instead, it introduces many of Python’s most noteworthy features, and will give you a good idea of the language’s flavor and style." But then I don't know how you're supposed to learn the features that are not in the tutorial. You can have a look at the table of contents of the standard library documentation for modules that might interest you, but that doesn't cover language features. Those are documented in The Python Language Reference, but that document is not really suited for learning from. There are lots of websites and Youtube channels and so on, but you have to find them, and filter out the not-so-good ones which is not easy, especially for a beginner. I think there is room for some kind of official advanced tutorial to cover the gap.
- fastasucan 3y agoI agree. Wish Python could been better with the documentation. Its a bit absurd that things feel more clear and simple reading Rust documentation than Python documentation for me, given that Python actually is a lot more simple and clear (for me).
- emmelaich 3y agoFor a long time, MatchObject was missing too. There now -- https://docs.python.org/3/library/re.html#match-objects https://docs.python.org/3/library/re.html#match-objects
- anentropic 3y agoFWIW there is an example of building a tokenizer/scanner using documented features of the re module here: https://docs.python.org/3.11/library/re.html#writing-a-tokenizer https://docs.python.org/3.11/library/re.html#writing-a-token... re.Scanner looks more succinct though...!
- czx4f4bd 3y agoI find that even more bizarre tbh. Thst seems like the perfect place to document the Scanner class.
- carlossouza 3y agoThis is one of the best ChatGPT use cases: creating regex complex patterns
- nmstoker 3y agoIt's excellent at both producing them and explaining what it has generated. And they're now just regurgitated things from the web, I've had novel ones generated fine (obviously you need to test them carefully still)
- extasia 3y agoI agree. I made a little code formatter for aligning variable assignments on adjacent lines and used chatgpt for a lot of help with the regex
- tl_donson 3y agothis one use case justified copilot as an expense for me.
- anitil 3y agoAmazing! I had just written a regex tokenizer in python the other day, this would have been great!
- cricalix 3y agoCurious cat - had you considered using Antlr4 and a Python visitor/listener? (Were you aware of antlr?). Depending what you're trying to do with a regex tokenizer, it might be suitable.
- anitil 3y agoIt was more just for fun than anything else. I was more interested in the recursive descent parser so the tokenizer was just a step along the way. I've played with Antlr but never really vibed with it to be honest (and it was years ago)
- burntsushi 3y agoYou'll be able to do this soon with the Rust regex crate as well. Well, by using one of its dependencies. Once regex 1.9 is out, you'll be able to do this with regex-automata: use regex_automata::{ meta::Regex, util::iter::Searcher, Anchored, Input, }; #[derive(Clone, Copy, Debug)] enum Token { Integer, Identifier, Punctuation, } fn main() { let re = Regex::new_many(&[ r"[0-9]+", r"[a-z_]+", r"[,.]+", r"\s+", ]).unwrap(); let hay = "45 pigeons, 23 cows, 11 spiders"; let input = Input::new(hay).anchored(Anchored::Yes); let mut it = Searcher::new(input).into_matches_iter(|input| { Ok(re.search(input)) }).infallible(); for m in &mut it { let token = match m.pattern().as_usize() { 0 => Token::Integer, 1 => Token::Identifier, 2 => Token::Punctuation, 3 => continue, pid => unreachable!("unrecognized pattern ID: {:?}", pid), }; println!("{:?}: {:?}", token, &hay[m.range()]); } let remainder = &hay[it.input().get_span()]; if !remainder.is_empty() { println!("did not consume entire haystack"); } } And its output: $ cargo run Compiling regex-scanner v0.1.0 (/home/andrew/tmp/scratch/regex-scanner) Finished dev [unoptimized + debuginfo] target(s) in 0.31s Running `target/debug/regex-scanner` Integer: "45" Identifier: "pigeons" Punctuation: "," Integer: "23" Identifier: "cows" Punctuation: "," Integer: "11" Identifier: "spiders" A bit more verbose than the Python, but the library is exposing much lower level components. You have to do a little more stitching to get the `Scanner` behavior. But it does everything the Python does: a single scan (using finite automata and not backtracking like Python), skip certain token types and guarantees that the entirety of the haystack is consumed.
- n8henrie 3y agoInteresting! Not often using either crate, this example looks like something for which I might usually look to nom. Is there a reason I should consider using regex for this use case instead (if neither is a pre-existing dependency)?
- burntsushi 3y ago
- sam2426679 3y agoI believe this is used internally by json.loads, which is not very surprising in hindsight.
- fastasucan 3y agoWoah, thats a pretty cool feature! I allways feel a bit dirty trying to do anything like that manually (usually involving a string.split(",")[0][:2] etc, just asking to break).