3 ms·

Show HN: I built an auto aggregating bot, collecting trending funny pictures

11 points by rezashirazian 10y ago

zackKidd 10y ago
Is Reddit the only source? If so, do you plan to add any more?
- rezashirazian 10y ago
  I have written a crawler for buzzfeed and 9gag but I don't have them running because there is a lot of overlap between 9gag and reddit and they require more work. To be honest this was a weekend project I did awhile ago. If there is viable interest I may pursue expanding it. But for now I'm going to keep it as is.
ksashikumar 10y ago
How did you do this? What tech stack do you use?
- rezashirazian 10y ago
  Ruby on Rails for the backend, Angular on the front. It's hosted on Heroku, it uses cloudfront, S3 for storage, Postgres for DB along with httpClient and mechanize for crawling and Paperclip for image manipulation.
  - ksashikumar 10y ago
    Do you use any kinda algorithm for crawling/fetching content from reddit?
    - rezashirazian 10y ago
      Not particularly. The one thing that helps but most people don't know about reddit is the fact that adding a .json to the end of each url displays the content of that page in json format. for example: reddit.com/r/funny.json This make crawling/fetching content from reddit much more trivial than old school web crawling.
deleted 10y ago
[deleted]
0xfff 10y ago
Is this just a reddit scraper or am I missing something?
- rezashirazian 10y ago
  I aggregate from Buzzfeed and 9gag as well, although I don't have them actively running. I try to give the images a clutter-free, mobile friendly, reliable, fast and ad-free home. (I compress and host all the images myself) I try to provide an easy mechanism for navigating to previous snapshots of what was aggregated for anyone who is passively browsing and has viewed all the recent images.
wingerlang 10y ago
Why are you watermarking the images?
- rezashirazian 10y ago
  Because many people hotlink directly from the site. And If I'm going to host the images myself and eat the cost of bandwidth I might as well derive some value from it. I provide a link to where the original images was taken from, so if anyone is interested in the image without the watermark they can get it from there.

11 points by rezashirazian 10y ago

zackKidd 10y ago

Is Reddit the only source? If so, do you plan to add any more?

rezashirazian 10y ago
I have written a crawler for buzzfeed and 9gag but I don't have them running because there is a lot of overlap between 9gag and reddit and they require more work. To be honest this was a weekend project I did awhile ago. If there is viable interest I may pursue expanding it. But for now I'm going to keep it as is.

ksashikumar 10y ago

How did you do this? What tech stack do you use?

rezashirazian 10y ago
Ruby on Rails for the backend, Angular on the front. It's hosted on Heroku, it uses cloudfront, S3 for storage, Postgres for DB along with httpClient and mechanize for crawling and Paperclip for image manipulation.
- ksashikumar 10y ago
  Do you use any kinda algorithm for crawling/fetching content from reddit?
  - rezashirazian 10y ago
    Not particularly. The one thing that helps but most people don't know about reddit is the fact that adding a .json to the end of each url displays the content of that page in json format. for example: reddit.com/r/funny.json This make crawling/fetching content from reddit much more trivial than old school web crawling.

deleted 10y ago

[deleted]

0xfff 10y ago

Is this just a reddit scraper or am I missing something?

rezashirazian 10y ago
I aggregate from Buzzfeed and 9gag as well, although I don't have them actively running. I try to give the images a clutter-free, mobile friendly, reliable, fast and ad-free home. (I compress and host all the images myself) I try to provide an easy mechanism for navigating to previous snapshots of what was aggregated for anyone who is passively browsing and has viewed all the recent images.

wingerlang 10y ago

Why are you watermarking the images?

rezashirazian 10y ago
Because many people hotlink directly from the site. And If I'm going to host the images myself and eat the cost of bandwidth I might as well derive some value from it. I provide a link to where the original images was taken from, so if anyone is interested in the image without the watermark they can get it from there.