2 ms·
What is the state of web scraping? I've got a few thousand URLs that is like to build a search engine around: https://github.com/melling/SwiftResources/blob/m
by melling 11y ago
What is the state of web scraping? I've got a few thousand URLs that is like to build a search engine around:
https://github.com/melling/SwiftResources/blob/master/swift_urls.tsv https://github.com/melling/SwiftResources/blob/master/swift_...
I'm using Swift to preprocess the data and I host my server on AppEngine using Go:
http://www.h4labs.com/dev/ios/swift.html http://www.h4labs.com/dev/ios/swift.html
Since my engine is stitched together, I could use Python or Perl to scrape the sites and extract the words, ignoring JavaScript and css.