5 ms·

> There are so many that just loading, copying, and pasting all of them took the little script I wrote more than 20 hours. I want to see that script.

lstamour 13y ago
It ran on a netbook and was actually controlling a browser and copying and pasting. It's not optimized, but it got the job done, so I'll applaud anyway. Great article, I expect we'll see more like this soon. ;-)
- lstamour 13y ago
  Oh and if you're nice on Netflix's servers, perhaps at 1 request a second, it would still take 25 minutes to run through 90,000 requests without the overhead of the browser. Sounds like they averaged one request every 24 seconds.
- tonfa 13y ago
  It actually looks pretty wasteful, since it probably needed to load the full webpage instead of just the html.
  - ihsw 13y ago
    It was enough for the author to write a good story on it. Wastefulness be damned, it got the job done.
jerf 13y ago
This sort of thing is generally pretty easy. For instance, this would be a good start: for i in {0..10000}; do wget http://whatever.invalid/genres/$i sleep 1 done From there processing the results to pull out the genre would be a regex away; a single properly formed grep could probably do it. Determining the top number would have to be done by hand; having holes in the sequences makes it more complicated but you could still use a sort of basic binary search.
- pravka 13y ago
  That would be a good start on a site that doesn't require authentication. :)
  - lstamour 13y ago
    You can add a cookie jar, probably good enough for these purposes when combined with browser string override.
visarga 13y ago
I laughed at that line. Here is someone who never scraped a website or run a spider to collect data from the net. It's a magical feeling next day, when you see your data.
- scott_s 13y ago
  To people who do their work on computers, but have never programmed before, programming is like a superpower.

lstamour 13y ago

It ran on a netbook and was actually controlling a browser and copying and pasting. It's not optimized, but it got the job done, so I'll applaud anyway. Great article, I expect we'll see more like this soon. ;-)

lstamour 13y ago
Oh and if you're nice on Netflix's servers, perhaps at 1 request a second, it would still take 25 minutes to run through 90,000 requests without the overhead of the browser. Sounds like they averaged one request every 24 seconds.
tonfa 13y ago
It actually looks pretty wasteful, since it probably needed to load the full webpage instead of just the html.
- ihsw 13y ago
  It was enough for the author to write a good story on it. Wastefulness be damned, it got the job done.

jerf 13y ago

This sort of thing is generally pretty easy. For instance, this would be a good start: for i in {0..10000}; do wget http://whatever.invalid/genres/$i sleep 1 done From there processing the results to pull out the genre would be a regex away; a single properly formed grep could probably do it. Determining the top number would have to be done by hand; having holes in the sequences makes it more complicated but you could still use a sort of basic binary search.

pravka 13y ago
That would be a good start on a site that doesn't require authentication. :)
- lstamour 13y ago
  You can add a cookie jar, probably good enough for these purposes when combined with browser string override.

visarga 13y ago

I laughed at that line. Here is someone who never scraped a website or run a spider to collect data from the net. It's a magical feeling next day, when you see your data.

scott_s 13y ago
To people who do their work on computers, but have never programmed before, programming is like a superpower.