3 ms·
Man, I feel old. Does anyone remember learning web scraping from one section of Fravia's site? Ever try to move forward from that to write a fully fledged searc
by contingencies 11y ago
Man, I feel old. Does anyone remember learning web scraping from one section of Fravia's site? Ever try to move forward from that to write a fully fledged search engine? These memories are from 15 years ago... quite amusing how much hasn't changed. In hindsight it was probably easier back then due to the lack of JS-reliant pages, less awareness of automation and less scraper detection algorithms.
- melling 11y agoWhat is the state of web scraping? I've got a few thousand URLs that is like to build a search engine around: https://github.com/melling/SwiftResources/blob/master/swift_urls.tsv https://github.com/melling/SwiftResources/blob/master/swift_... I'm using Swift to preprocess the data and I host my server on AppEngine using Go: http://www.h4labs.com/dev/ios/swift.html http://www.h4labs.com/dev/ios/swift.html Since my engine is stitched together, I could use Python or Perl to scrape the sites and extract the words, ignoring JavaScript and css.