4 ms·

I did the exact same thing a while back. The company I worked with used 4 fully-loaded servers driving Java spiders while I used 2 y/o laptop and 100 lines of E

by damir 15y ago

I did the exact same thing a while back. The company I worked with used 4 fully-loaded servers driving Java spiders while I used 2 y/o laptop and 100 lines of Erlang. Showed the code to my manager, he left the room saying I should do some "real" sysadmin work and stop playing around. Just for a test, I started spidering big local news sites with I think 10000 threads and left it running for about 15 minutes. Two things happened: 1. I consumed all bandwidth, rendering Internet unusable for the whole company, and 2. got about 10% of data other 4 Java servers could gather in a day This stunt didn't get me fired, but my manager insisted I delete the program I wrote. Nobody was interested in results I got so nothing happened. But it was fun watching bandwidth usage RRD graphs go through the roof :)

ChuckMcM 15y ago
Interesting, the link to your blog in your profile goes to a parked domain (just FYI). I hear these sorts of tales occasionally and I recognize a manager who is out of their league when comes to keeping someone with your skill set engaged. If crawling the big local news sites was something useful for the company, moving your application to a hosted machine where it had access to more bandwidth would have been a start, of course a demonstration is just that, a demonstration. The next step which is a bit harder might be to crunch the data into some sort of data structure for later analysis. If this is the kind of problem you enjoy working on and are in the SF Bay area we should talk :-)
api 15y ago
Yeah. In general, nobody knows how to write network code. I have seen tons of network apps that perform horribly and use utterly absurd amounts of resources for trivial tasks.
- librarydweller 15y ago
  Used to work in a fortune 500 company that would send email notifications to customers telling them their monthly statements were available. Unfortunately, the monkeys who coded the email sending didn't understand the protocol, obviously had not read the RFC. The code would blindly send smtp commands without waiting for a response. One day the smtp server decided to change its behavior due to a dns misconfiguration and many emails were lost.

by damir 15y ago

ChuckMcM 15y ago

Interesting, the link to your blog in your profile goes to a parked domain (just FYI). I hear these sorts of tales occasionally and I recognize a manager who is out of their league when comes to keeping someone with your skill set engaged. If crawling the big local news sites was something useful for the company, moving your application to a hosted machine where it had access to more bandwidth would have been a start, of course a demonstration is just that, a demonstration. The next step which is a bit harder might be to crunch the data into some sort of data structure for later analysis. If this is the kind of problem you enjoy working on and are in the SF Bay area we should talk :-)

api 15y ago

Yeah. In general, nobody knows how to write network code. I have seen tons of network apps that perform horribly and use utterly absurd amounts of resources for trivial tasks.

librarydweller 15y ago
Used to work in a fortune 500 company that would send email notifications to customers telling them their monthly statements were available. Unfortunately, the monkeys who coded the email sending didn't understand the protocol, obviously had not read the RFC. The code would blindly send smtp commands without waiting for a response. One day the smtp server decided to change its behavior due to a dns misconfiguration and many emails were lost.