2 ms·

What kind of framework was this? Who designed it? So it sounds like if I build a big enough fence, it can work out pretty well?

• bartonfink 15y ago
  It wasn't an off the shelf framework. "Scraping engine" may be a better phrase to describe what it actually did. External components would make a call, saying something like "I need the data for this particular patient from this particular hospital". The engine would then spawn off a thread containing the scraping code for that particular hospital, and it would run off, scrape the website and then return a well formatted version of that data or one of a series of error messages. If the thread got hung up, the engine killed it and returned an error. We built the engine when we realized that these scrapers were a competitive advantage for the company and we'd be writing a lot of them (when I left, we had around 250 of them with a backlog of near 1000 to be written). It became worth the investment to write the framework because there were features we wanted that would have been hell to integrate any other way - for instance, killing long-running scrapers that may have been hung up. The engine bought us lots of other niceties too - for instance, centralized error reporting. If one particular scraper returned lots of errors, odds are that the site changed and we knew to give the code some attention. When you describe it as a "big enough fence", that's only partially true. It's more about a series of independent tasks that don't depend on each other. When you have that and those tasks have only minimal impact to the system as a whole that you have a scenario which suggests drone programming.