3 ms·

It would be much difficult to write many more complex scrapers just using regexes, which is why methods like the one posted above scale well with complexity. F

• mmaunder 16y ago
  It scales. I ran WorkZoo.com (Time Mag top 50 website of 2005 - sold it the same year) and we scraped over 500 job boards and aggregated the jobs into a search engine. A team of devs developed and maintained the regex for each board and I managed them and I wrote the dev tools they used to develop the regex for each site we scraped. It was incredibly effective and maintainable. Incidentally, the dev tools I wrote were in javascript and they created regex that we'd test in javascript and deploy in Perl. The regex engines are identical which is why it worked. Scraper abstraction is for people too lazy to learn regex. Get a good book on regex, and learn how to use Perl's s/// regex with the 'e' modifier. It'll change your life.
  • mishoo 16y ago
    ... and later you begin to discover different, much better ways to solve problems that you used to do with regexps, and your life will get back to normality and happiness. ;-) For a solid scrapper in Perl I'd use HTML::TreeBuilder / HTML::Element. Perhaps slower than regexps, but does real parsing and understands tag-soup HTML.
    • parasctr 16y ago
      These modules are very resource intensive and they become a bottleneck if you are scraping at high volume. I had to stop using these modules in one of my tools because of that reason. However, for smaller jobs they are awesome and much easier to use and understand.