8 ms·
Parsing URLs in Python
- pyuser583 3y agoThe Ada programming language is cursed with overlapping acronyms. GPS, ADA, SPARK, AWS. Seems it just got a little bit worse.
- Solvency 3y agoIt is mindboggling to me how often developers create project names without even trying to search for precedent names in their own domain/industry. Calling this Ada is just ridiculous.
- gjvc 3y agoyes but they think it's punny/funny/clever
- nine_k 3y agoIt's usually a poor reason. Calling the GNU Image Manipulation Program "GIMP" was kinda funny and punny, and maybe humble, but not wise. Inkscape or Krita have less poignant, but more reasonable name. (But yes, such an approach removes some of the teenage fun from doing a project.)
- pyuser583 3y agoAda needs to fight back. Java templating library! Python dependency resolver! Zag image manipulation library! SQL event logging persistence framework! I can’t believe Amazon wasn’t violating some anti-competition rule by using “AWS” for “Amazon Web Services” when it already meant “Ada Web Server.” Wow when I search for “Ada GPS” I get global position system support libraries before GNAT Programming Studio.
- yagiznizipli 3y agoAda developer here. Ada URL parser is named after my daughter Ada. We chose this name in particular as a reference to Ada Lovelace.
- Hendrikto 3y ago> We chose this name in particular as a reference to Ada Lovelace. Just like the ~20 other projects named Ada.
- masklinn 3y ago> Ada is a WHATWG-compliant and fast URL parser written in modern C++ Why would you do that, Daniel?
- TkTech 3y agoHah :) It's named after Yagiz Nizipli's (Ada dev) newborn daughter, Ada.
- notso411 3y ago[dead]
- yagiznizipli 3y agoAda developer here, Ada is the name of my daughter, and this project is my gift to her, to remember me.
- pyuser583 3y agoCongrats on having a daughter! I hope and day she’ll need to parse a url, and be able to see your love for her in her code!
- deleted 3y ago[deleted]
- EmilStenstrom 3y agoAt least the can_ada (canada!) makes it a little bit more unique.
- TZubiri 3y agoA url is composed by <protocol>://<protocol-specific-params> Boom, you have now parsed a URL. Wanna parse an http query params into host,port, resource? Speak appropriately, ask how to parse an http request, wanna parse a resource? Get into those semantics, be precise
- chirau 3y ago[flagged]
- simonw 3y agoI thought the linked article did a good job of explaining how deceptively hard it is to parse URLs correctly and handle all edge cases.
- williamdclt 3y ago> Parsing urls is not difficult at all I’d like to have 100 developers each write a url parser, and see how many bugs per implementation we can find. I’d guess an average in the double-digits
- otherme123 3y agoMaybe this is a case of "I could write curl in a weekend", proceeds to use libcurl one way or the other.
- zzo38computer 3y agoI did write a URL parser (including converting relative URLs into absolute) in C (I also wrote a simple HTTP client, and other protocols). However, it is only intended for use with a limited set of URI schemes (including "hashed" and "jar", both of which are unusual in the way they are handled). (See the function called "scogem_parse_url" in the https://raw.githubusercontent.com/zzo38/scorpion/trunk/scogem.c https://raw.githubusercontent.com/zzo38/scorpion/trunk/scoge... file.) Now, we can find bug in that one, and then if other people mention theirs too, and find bug in other one, then we can see how accurate your guess is. However, there are other considerations. For one thing, WHATWG is not the only specification of the working of URLs, so not everyone will comply anyways. And, some features might be necessary or not necessary in specific applications.
- deleted 3y ago[deleted]
- memco 3y agoThis is intriguing to me for the performance and correctness reasons, but also if it makes the result more dev friendly than the urllib.parse tiple-ish-object result thing.
- nerdponx 3y agoI posted this elsewhere in the thread, but there absolutely is prior art here. Check out Yarl (urllib.parse wrapper with the nicer interface) and Hyperlink (green field, immutable OO style, focus on correctness). Both on PyPI for many years now.
- memco 3y agoThanks! I have come across yarl, but not hyperlink.
- diarrhea 3y agoYeah, that tuple API is bizarre. It really doesn’t play well with type annotations either.
- Areading314 3y agoHard to imagine the tradeoff of using a third party binary library developed this year vs just using urllib.parse being worth it. Is this solving a real problem?
- pyuser583 3y agourlib.parse is a pain. We really need something more like pathlib.Path.
- masklinn 3y agoThat used to be werkzeug.urls, kinda (it certainly had a more convenient API than urllib.parse), but it was killed in Werkzeug 3.
- pyuser583 3y agoI remember and miss that. But I’m not going to install werkzeug just for the url parsing.
- d_kahneman7 3y agoIs it that inconvenient?
- Ch00k 3y agoThere is https://github.com/gruns/furl https://github.com/gruns/furl
- ulrischa 3y agoParsing an url is really a pain in the a*
- deleted 3y ago[deleted]
- SloopJon 3y agoAnother post in this thread was downvoted and flagged (really?) for claiming that URL parsing isn't difficult. The linked article claims that "Parsing URLs correctly is surprisingly hard." As a software tester, I'm very willing to believe that, but I don't know that the article really made the case. I did find a paper describing some vulnerabilities in popular URL parsing libraries, including urllib and urllib3. Blog post here: https://claroty.com/team82/research/exploiting-url-parsing-confusion https://claroty.com/team82/research/exploiting-url-parsing-c... Paper here: https://web-assets.claroty.com/exploiting-url-parsing-confusion.pdf https://web-assets.claroty.com/exploiting-url-parsing-confus... If you remember the Log4j vulnerability from a couple of years ago, that was an URL parsing bug.
- masklinn 3y ago> If you remember the Log4j vulnerability from a couple of years ago, that was an URL parsing bug. I don't think that's a fair description of the issue. The log4j vulnerability was that it specifically added JNDI support (https://issues.apache.org/jira/browse/LOG4J2-313 https://issues.apache.org/jira/browse/LOG4J2-313) to property substitution (https://logging.apache.org/log4j/2.x/manual/configuration.html#PropertySubstitution https://logging.apache.org/log4j/2.x/manual/configuration.ht...), which it would apply on logged messages. So it was a pretty literal feature of log4j. log4j would just pass the URL to JNDI for resolution, and substitute the result.
- SloopJon 3y agoI didn't look into this in detail at the time, but the report's summary of CVE-2021-45046 is that the parser that validated an URL behaved differently than a separate parser used to fetch the URL, so an URL like jndi:ldap://127.0.0.1#.evilhost.com:1389/a is validated as 127.0.0.1, which may be whitelisted, but fetched from evilhost.com, which probably isn't.
- bqmjjx0kac 3y agoWriting a new parser in C++ is a mistake IMO. At the very least, you need to write a fuzzer. At best, you should be using one of the many memory safe languages available to you. I retract my criticism if this project is just for fun. Edit: downvoters, do you disagree? Edit2: OK, I may have judged a bit prematurely. Ada itself has fuzzers and tests. They're just not exported to the can_ada project.
- yagiznizipli 3y agoAda developer here. Ada has more than 5000 tests, is included in oss-fuzz project and battle tested in Node.js and Cloudflare workers.
- bqmjjx0kac 3y agoI apologize for the misjudgment. I just followed the link to can_ada and saw really minimal tests, e.g. https://github.com/TkTech/can_ada/blob/main/tests/test_parsing.py https://github.com/TkTech/can_ada/blob/main/tests/test_parsi... I didn't understand that can_ada is not where the parser is developed.
- VagabundoP 3y agoOkay so some googling found me that the "xn--" means the rest of the hostname will be unicode, but why does é become -fsa in www.xn--googl-fsa.com. Google failed on the second part.
- whalesalad 3y agoIt’s called an IDN. This is an encoding format called puny code that transforms international domains into ascii
- js2 3y agoBecause that's the Punycode representation: https://en.wikipedia.org/wiki/Punycode https://en.wikipedia.org/wiki/Punycode https://www.punycoder.com/ https://www.punycoder.com/
- andy99 3y agoI wasn't aware of this, I'd seen those URLs before but only in the context of Chinese ones and thought it was Chinese-specific. It's interesting because I just went down an apparent rabbit hole inplementing Byte-level encoding for using language models with unicode. There each byte in a unicode character is mapped to a printable character that goes up to 255 < ord(x) < 511 (I don't remember the highest but the point is each byte is mapped to another printable unicode character. See https://github.com/openai/gpt-2/blob/9b63575ef42771a015060c964af2c3da4cf7c8ab/src/encoder.py#L19 https://github.com/openai/gpt-2/blob/9b63575ef42771a015060c9... And the actual list of characters: https://github.com/rbitr/llm.f90/blob/dev/phi2/phi2/pretokenize.f90#L34 https://github.com/rbitr/llm.f90/blob/dev/phi2/phi2/pretoken...
- deleted 3y ago[deleted]
- ekimekim 3y agoTo expand on the sibling comments: This encoding (called Punycode) works by combining the character to encode (é) and the position the character should be in (the 7th position out of a possible 7) into a single number. é is 233, there are 7 possible positions, and it is in position 6 (0-indexed) so that single number is 233 * 7 + 6 = 1637. This is then encoded via a fairly complex variable-length encoding scheme into the letters "fsa". See https://en.wikipedia.org/wiki/Punycode#Encoding_the_non-ASCII_characters https://en.wikipedia.org/wiki/Punycode#Encoding_the_non-ASCI...
- bormaj 3y agoWhy not just use `httpx`? If you're not bound to the stdlib, it's a great alternative to `requests` and url parse
- deleted 3y ago[deleted]
- kmike84 3y agoThe URL parsing in httpx is rfc3986, which is not the same as WHATWG URL living standard. rfc3986 may reject URLs which browsers accept, or it can handle them in a different way. WHATWG URL living standard tries to put on paper the real browser behavior, so it's a much better standard if you need to parse URLs extracted from real-world web pages.
- gjvc 3y agohttpx is great, and "needs to be in base"
- ojbyrne 3y agoLooking at the github site for can_ada, I discovered that the developers live in Montreal, Canada. Nice one.
- deleted 3y ago[deleted]
- TkTech 3y agoI do! :) I should probably also never be allowed to name things.
- foxhop 3y ago[dead]
- kmike84 3y agoA great initiative! We need a better URL parser in Scrapy, for similar reasons. Speed and WHATWG standard compliance (i.e. do the same as web browsers) are the main things. It's possible to get closer to WHATWG behavior by using urllib and some hacks. This is what https://github.com/scrapy/w3lib https://github.com/scrapy/w3lib does, which Scrapy currently uses. But it's still not quite compliant. Also, surprisingly, on some crawls URL parsing can take CPU amounts similar to HTML parsing. Ada / can_ada look very promising!
- TkTech 3y agocan_ada dev here. Scrapy is a fantastic project, we used it extensively at 360pi (now Numerator), making trillions of requests. Let me know if I can help :)
- JulianWasTaken 3y agoNice. I'll also throw in that I've recently wrote bindings to Mozilla's servo URL library. Those live at https://github.com/crate-py/url https://github.com/crate-py/url They're not complete yet (meaning only the parsing bits are exposed, not URL modification) but I too was frustrated with the state of URL parsing.
- timhh 3y agoIMO that URL crate is not especially high quality. I barely work with URLs and I quickly found an embarrassingly trivial bug: https://github.com/servo/rust-url/issues/864#issuecomment-1694749589 https://github.com/servo/rust-url/issues/864#issuecomment-16...
- phyzome 3y agoHonestly to hell with the WHATWG's weird pseudo-standard. Backslashes? Five forward slashes? No one is sending those URLs around, and if they are, they should fix it. The only thing that needs to deal with that kind of malformed URL is the browser's address bar. If browsers want to do fixups on broken URLs at that point, they should feel free to, but it shouldn't pollute an entire spec. (It's not even a real standard -- it's a "living standard", which means it just changes randomly and there's no way to actually say "yes, you're compliant with that".)
- notso411 3y ago[dead]
- domenicd 3y agoIt's always been extremely funny to me how arguments like this and from the curl author go. "Yes, I had to change curl away from strictly accepting two slashes, for web compatibility. But `while(slash) { advance_parser() }`? That's completely unreasonable! `while(slash && ++i <= 3)` is so much better, and works almost as well!" Ok, whatever... As for your claim about living standards, I'd encourage you to read https://whatwg.org/faq#living-standard https://whatwg.org/faq#living-standard
- phyzome 3y agoI don't think there's anything new and interesting in that FAQ. It describes what they mean by a Living Standard, but it doesn't mean they're not running a treadmill operation. They seem to be making reference to things like the RFC system, but those get updated too.
- nerdponx 3y agoNo mention of prior art? The Hyperlink library has stated correctness as its goal for a long time: https://pypi.org/project/hyperlink https://pypi.org/project/hyperlink Of course there is always room for new projects, but it still feels weird to act as if this is the first time anybody has ever tried this. It seems like a lot of people are under this same mistaken impression, at least according to the sample of HN users who commented in this thread.
- TkTech 3y agoHyperlink (which I didn't know existed, by the way) is not a parser for the WHATWG spec, it's for RFC3986. You seem to be getting things confused.
- nerdponx 3y agoThe article makes it sound like the only parser for URLS in the entire Python ecosystem is urllib.parse, regardless of which spec it supports. Hyperlink and Yarl are absolutely prior art here IMO, and at least deserve a mention in an article like this.
- abstractbslayer 3y agoPretty urls are the most unnecessary thing that was invented, they don't provide anything that non pretty urls can't provide and millions of parsers have to process them on every request. What a waste.
- edflsafoiewq 3y agoWhat is a "pretty url"?
- d0mine 3y agoThere are people who are not native English speakers. They might find Unicode useful.
- d0mine 3y agostdlib behavior may be more preferable: - resolving "../" may have security implications - unicode hostname seems more readable. To get punycode, one can call .encode("idna") if necessary. How often parsing urls is a performance bottleneck?