3 ms·
MIT Webscraper :)
- deleted 2mo ago[deleted]
- codingdave 2mo agoActual Title: MIT Webscraper "...please use the original title, unless it is misleading or linkbait; don't editorialize." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- ForgottenMail 2mo agoMy bad, bad decision on my part, its fixed!
- ventana 2mo agoI was worried that getting this data would be difficult, but unironically, they literally have a single div for all of the text in the article and a single div for the header that contains the article name, author and date released. IT'S LIKE THEY WANTED ME TO SCRAPE THE SITE. Why such a conclusion, and why is this "unironically"? A normal blog page, MIT or not, which is not obsessed with monetization or showing ads after each paragraph, is literally just a text with some very minor formatting (adding a few HTML tags here and there). It's very much expected that you would have a single <div class="content"> and an actual content inside of it.
- ForgottenMail 2mo ago¯\(ツ)/¯ I was expecting it to be vauge and or have many divisions that would have made scraping harder, I was surprised that it did not. I also just have not had that much experience with large professional sites like that before