I just wrote a Ruby script that does pretty much the same thing: https://gist.github.com/865841 https://gist.github.com/865841. I used a simple bloom filter for

I just wrote a Ruby script that does pretty much the same thing: https://gist.github.com/865841 https://gist.github.com/865841. I used a simple bloom filter for de-duplication. It's a bit overkill. A simple hash on the twitpic_id will probably work just as well if not better.