3 ms·
I cringe at the "I wrote a web scraping program to go through the Brickset database". Brickset and Rebrickable both have awesome APIs that can provide all this
by someotheridiot 12y ago
I cringe at the "I wrote a web scraping program to go through the Brickset database". Brickset and Rebrickable both have awesome APIs that can provide all this data in an easily digestible format.
- sebastianavina 12y agobut he wrote a web scraping program for mining the data, bonded it together using wget, launched several aws instances with hadoop, and deployed a mapreduce job for cleaning up the data. It took three weeks and $200 usd, but damn, modern technology is awesome, it would be impossible to do this just 10 years ago. Thanks google, thanks amazon.
- ibotty 12y agoseems overkill. still great to live now for doing that kind of analysis.