3 ms·

The last time I wrote a generalized web scraper (as opposed to something specific that we had permission for) I put a lot of effort into distributing the load a

The last time I wrote a generalized web scraper (as opposed to something specific that we had permission for) I put a lot of effort into distributing the load across many websites so that no single site would feel any pain more than if you were just browsing as a normal human being. We were--at the time, scraping for lead information to add to our marketing database, and this isn't the thing I'm exactly the most proud of in my career. But we all make mistakes. I wouldn't do that again. At the same time, we rotated things so that we weren't killing the websites in the niche market that we were trying to scrape for leads. The algorithm was that we would seed Google, Yahoo, and Bing with certain keyword searches that were relevant. Then we'd take the search results from the APIs and stuff them into an array. Then we would sort them proportionally. If we (like we did) most often get the most hits from google, followed by yahoo, and then by Bing, we'd stuff the results into an array and intersperse them. So if we had 3x google results and 2x yahoo, and 1x bing, we wouldn't hit the google results first. We'd hit a google result 3x then a yahoo 2x, then a bing 1x and cycle. It was a decent way of doing things. We never broke anyone's stuff. Even if it should have been.

AznHisoka 9y ago
i dont understand why you wouldnt be proud of this.. what exactly did you do wrong?
- ianamartin 9y ago
  I feel uncomfortable about what we were doing in general. I didn't feel bad about the implementation. We were using the results of the web scraper to feed what was basically spam emails trying to recruit people to sign up for our product. In the grand scheme of things, I think that I did the most responsible thing I could have with the task I was assigned. But the task falls into a general category of things I don't approve of. It certainly wasn't illegal, and it probably wasn't unethical, but it was definitely gross, and my internal standards tell me to avoid things that make me feel gross. That said, I'm kind of pleased with the technical results. Up until that time in my career, I'd never encountered a sorting algorithm that handled things based on the proportion of similar items in an array. I'm sure that other people have done this and that there's no way it's novel at all. But it was a cool challenge to make a shady task perform in a way that didn't break other people's shit just so that we could maximize our own efficiency.

AznHisoka 9y ago

i dont understand why you wouldnt be proud of this.. what exactly did you do wrong?

ianamartin 9y ago
I feel uncomfortable about what we were doing in general. I didn't feel bad about the implementation. We were using the results of the web scraper to feed what was basically spam emails trying to recruit people to sign up for our product. In the grand scheme of things, I think that I did the most responsible thing I could have with the task I was assigned. But the task falls into a general category of things I don't approve of. It certainly wasn't illegal, and it probably wasn't unethical, but it was definitely gross, and my internal standards tell me to avoid things that make me feel gross. That said, I'm kind of pleased with the technical results. Up until that time in my career, I'd never encountered a sorting algorithm that handled things based on the proportion of similar items in an array. I'm sure that other people have done this and that there's no way it's novel at all. But it was a cool challenge to make a shady task perform in a way that didn't break other people's shit just so that we could maximize our own efficiency.