5 ms·
Great memories with this library, one of my all time favs. It is fast? no. But it had a fantastic mission: extracting data from malformed HTML. Might be less
by djtriptych 4y ago
Great memories with this library, one of my all time favs.
It is fast? no.
But it had a fantastic mission: extracting data from malformed HTML.
Might be less common now but back then (~10+ years ago) it was still rampant. Many if not most parsers would barf on any deviation from the standard, leaving you to hand-roll regex solutions and ugly corner cases.
BS covered a LOT of these cases without forcing you to write terrible code. It mostly just worked, with a reasonable API, and stellar, well-written, example-laden docs.
- ssimpson 4y agoabsolutely. i wrote this mobile allergy data thing and getting the data was mostly scraping from news websites that did terrible things with javascript to keep scrapers and ad blockers out. Beautiful Soup worked past all of that easily. Probably my favorite Python library.
- jerf 4y ago"Might be less common now" It is less necessary now. One of the most important parts of the HTML5 standards, IMHO, is that it specifies how to parse HTML that doesn't conform to the standards in a standard way. In principle, every bag of bytes now has a standard-compliant way to parse it that every HTML5 parser should agree on. I don't use this to know how many edge cases the standard and/or implementations have, but it's a lot better than it used to be, and means that every HTML5 parser has many of the capabilities that Beautiful Soup used to (nearly-)uniquely have for parsing messy HTML. I suspect Beautiful Soup was a non-trivial aspect of how the decision to implement such a spec was decided upon. It proved the idea to be a very valuable one at a time when most languages lacked such a library. Basically, BS won so hard that while it wasn't necessarily directly adopted as a standard, the essence of it certainly was.
- ThatPlayer 4y agoNot just HTML standards, but I've used their detwingle function because one of the sites I'm scraping has a mixture of Windows-1252 and Unicode. It was clearly stored correctly, but encoded differently depending on what page view you were looking at. For example titles in an outline were broken, but on the actual individual pages fine. Their rendering also treated multibyte characters incorrectly during truncation.
- cogman10 4y agoI work in finance and this thing is still indispensable for us. A LOT of financial info is only presented on a dynamically rendered HTML page that was last written in the bad old days.
- gjvc 4y agoAbsolutely. Until you have had to deal with this kind of problem first-hand, you have no idea how much of a relief it is that it exists. Sometimes, finding and using the right library can completely turn around a f'd project.
- xbar 4y agoTrue. And BS has been that library for me on at least 2 such projects.
- moneywoes 4y agoAny poignant examples?
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- toyg 4y ago> Might be less common now That's because it's often deeply buried under more fashionable abstractions.
- samwillis 4y agoHave literally been building something with BS today. It’s very much still a current library, I imagine in some areas people have moved on but I will continue to reach for it.
- agumonkey 4y agothe jquery of backend
- Der_Einzige 4y agoI remember that I needed to do something involving performance with beautiful soup. Switching the HTML parser backend (as they mention in the docs in BS4) gave me an order of magnitude speedup...
- dmayle 4y agoIt's actually pretty trivial to speed up, if you have multiple documents to parse, you can use multiprocessing. from multiprocessing import Pool def parse(html): result = [] soup = BeautifulSoup(html, 'html.parser') for p in soup.select('div > p'): result.append(p.text) return result with Pool(processes=16) as pool: for texts in pool.imap_unordered(parse, my_html_texts): for text in texts: print(text)
- djtriptych 4y agoit's a pretty common case to be parsing thousands of locally-stored pages. There are only so many cores and the task of scraping an entire site can still easily still be CPU limited.
- driscoll42 4y agoYou might want to try a different parser as well, I tried a basic performance test comparison in another comment (https://gist.github.com/MercuryRising/4061368 https://gist.github.com/MercuryRising/4061368) and the html.parser was very slow compared to the lxml-xml or xml parsers for bs4 ==== Total trials: 100000 ===== bs4 lxml total time: 110.9 bs4 html.parser total time: 87.6 bs4 lxml-xml total time: 0.5 bs4 xml total time: 0.5 bs4 html5lib total time: 103.6 pq total time: 8.7 lxml (cssselect) total time: 8.8 lxml (xpath) total time: 5.6 regex total time: 13.8 (doesn't find all p)
- aumerle 4y agoIf you want fast HTML parsing in python+lxml, use html5-parser https://github.com/kovidgoyal/html5-parser https://github.com/kovidgoyal/html5-parser
- anigbrowl 4y agoI'm not sure why you're using the past tense, the current version is only a year old, and I will never stop using it because panning messy datasets for hidden gold is my thing.
- altdataseller 4y agoWhat type of datasets?
- anigbrowl 4y agoThings with janky APIs or built on abandonware like imageboards, niche forums, dead comment communities, data dumps etc.
- djtriptych 4y agodoing this professionally? Just curious..
- galoisgirl 4y agoSame. I had to parse HTML written by eBay sellers. They were professional sellers, not webmasters.