3 ms·

I used to work in a company where the drone approach did work out well. We wrote screen scrapers for insurance websites that ran in a framework that would kill

I used to work in a company where the drone approach did work out well. We wrote screen scrapers for insurance websites that ran in a framework that would kill them if they went off the rails, handled serious errors, and was the major interface between our component and the front end products the company actually sold. The scrapers weren't terribly technically challenging, and at the time I left, I could crank one out in roughly 3 days which was a pretty good pace. When I left, I mentioned in the exit interview that, for my salary and benefits, the company could hire 4 interns and put them to work. Even if they were twice as slow as I was, which was unlikely given all the work that went into the framework, the company would have a net productivity win, not to mention the benefits that come with having interns around to recruit from. Last I heard, they took me up on that suggestion and picked up 10 interns who are all writing and maintaining scrapers and freeing up full-timers to work on less routine work. It can work, but you need a steady supply of relatively routine work that's set up in such a way that nobody can break the system with a simple bug or logic error.

diminium 15y ago
What kind of framework was this? Who designed it? So it sounds like if I build a big enough fence, it can work out pretty well?
- bartonfink 15y ago
  It wasn't an off the shelf framework. "Scraping engine" may be a better phrase to describe what it actually did. External components would make a call, saying something like "I need the data for this particular patient from this particular hospital". The engine would then spawn off a thread containing the scraping code for that particular hospital, and it would run off, scrape the website and then return a well formatted version of that data or one of a series of error messages. If the thread got hung up, the engine killed it and returned an error. We built the engine when we realized that these scrapers were a competitive advantage for the company and we'd be writing a lot of them (when I left, we had around 250 of them with a backlog of near 1000 to be written). It became worth the investment to write the framework because there were features we wanted that would have been hell to integrate any other way - for instance, killing long-running scrapers that may have been hung up. The engine bought us lots of other niceties too - for instance, centralized error reporting. If one particular scraper returned lots of errors, odds are that the site changed and we knew to give the code some attention. When you describe it as a "big enough fence", that's only partially true. It's more about a series of independent tasks that don't depend on each other. When you have that and those tasks have only minimal impact to the system as a whole that you have a scenario which suggests drone programming.

diminium 15y ago

What kind of framework was this? Who designed it? So it sounds like if I build a big enough fence, it can work out pretty well?

bartonfink 15y ago
It wasn't an off the shelf framework. "Scraping engine" may be a better phrase to describe what it actually did. External components would make a call, saying something like "I need the data for this particular patient from this particular hospital". The engine would then spawn off a thread containing the scraping code for that particular hospital, and it would run off, scrape the website and then return a well formatted version of that data or one of a series of error messages. If the thread got hung up, the engine killed it and returned an error. We built the engine when we realized that these scrapers were a competitive advantage for the company and we'd be writing a lot of them (when I left, we had around 250 of them with a backlog of near 1000 to be written). It became worth the investment to write the framework because there were features we wanted that would have been hell to integrate any other way - for instance, killing long-running scrapers that may have been hung up. The engine bought us lots of other niceties too - for instance, centralized error reporting. If one particular scraper returned lots of errors, odds are that the site changed and we knew to give the code some attention. When you describe it as a "big enough fence", that's only partially true. It's more about a series of independent tasks that don't depend on each other. When you have that and those tasks have only minimal impact to the system as a whole that you have a scenario which suggests drone programming.