10 ms·
Show HN: A search engine based on social signals
We built a search engine that shows you the most engaging stories/topics being shared across Twitter, Facebook, Linkedin, and Google+. We crawled over 15 million articles the past 3 months, retrieved the total number of Facebook likes, tweets, Google+’s etc and built a search index around it.
Here's what our infrastructure looks like:
Rails/Redis: We use the Sidekiq gem as a message queue. We have hundreds of workers that do the crawling, data mining, and number crunching.
ElasticSearch: We built the search index using ElasticSearch, with the data imported from our Postgres database.
AngularJS: Front-end is built using AngularJS and Laravel 4 (PHP), which sends the requests to our Rails server in the backend.
- jwblackwell 13y agoWe built a search engine that shows you the most engaging stories/topics being shared across Twitter, Facebook, Linkedin, and Google+. We crawled over 15 million articles the past 3 months, retrieved the total number of Facebook likes, tweets, Google+’s etc and built a search index around it. Here's what our infrastructure looks like: Rails/Redis: We use the Sidekiq gem as a message queue. We have hundreds of workers that do the crawling, data mining, and number crunching. ElasticSearch: We built the search index using ElasticSearch, with the data imported from our Postgres database. AngularJS: Front-end is built using AngularJS and Laravel 4 (PHP), which sends the requests to our Rails server in the backend.
- kixxauth 13y agoNice. The speed is really impressive, at least for the queries I tried. There are still a lot of queries with no results, and you might want to think of something clever to put there in those cases to keep the user turned onto your brand. When there are results, they seem quite relevant and useful. The "Show me how it works" tour is great too. I think you just need to work on your value proposition on the landing page. Don't be afraid to sell something. Put together few paid packages as soon as possible. Don't worry if it's not perfect.
- jwblackwell 13y agoThanks! We really appreciate the feedback. We're constantly working on growing the index and improving the search so expect that to improve over the coming months.
- toumhi 13y agoYup. Agreed with everything. Tried "customer success" for example and the results were not valuable. Tried "startups" and got lots of results, but that might be too broad of a topic. Technically I'd say this is pretty good, I agree that OP needs to work on value proposition and to sell: that's something for content marketers, but try to dive more into the problem you're solving. Be specific. "Social insights" seems too broad to me. For example, the problem could be not knowing which content to write, wanting to keep tabs on what your competiton writes about, spending too much time researching all this... You're already writing about benefits("discover what people are sharing..."), I'd try to connect that with how your app is better than current existing methods used by content marketers to create their content strategy. Congrats with the launch !
- dancecodes 13y agolooks nice my test about gentoo - passed!
- dancecodes 13y agosuggestion: please add your engine to http://mycroftproject.com/search-engines.html?name=buzzsumo http://mycroftproject.com/search-engines.html?name=buzzsumo
- dancecodes 13y agoi think its will explosive trend to internet success to you
- AznHisoka 13y agoPossible feature to add: 1) Ability to search for influencers that shared an article related to a specific topic. For instance, I want a way to search for SEO influencers who shared something related to Breaking Bad b/c I'm making an article about how SEO is related to Breaking Bad, etc.
- jwblackwell 13y agoGreat idea! We're currently working on adding more advanced search operators/an advanced search option so you can do stuff like this.
- jackmaney 13y agoVery interesting! I got a timeout when trying to find sharers, but I suspect that's due to the fact that you've hit the front page of HN. ;)
- Kiro 13y agoSince OP is dead: jwblackwell 48 minutes ago | link [dead] We built a search engine that shows you the most engaging stories/topics being shared across Twitter, Facebook, Linkedin, and Google+. We crawled over 15 million articles the past 3 months, retrieved the total number of Facebook likes, tweets, Google+’s etc and built a search index around it. Here's what our infrastructure looks like: Rails/Redis: We use the Sidekiq gem as a message queue. We have hundreds of workers that do the crawling, data mining, and number crunching. ElasticSearch: We built the search index using ElasticSearch, with the data imported from our Postgres database. AngularJS: Front-end is built using AngularJS and Laravel 4 (PHP), which sends the requests to our Rails server in the backend.
- deleted 13y ago[deleted]
- jackmaney 13y agoI don't get it....why would it be dead? Did the OP delete that post, or was it just heavily downvoted? And if the latter, why?!
- jwblackwell 13y agoIt's way down at the bottom! Obviously no one upvoted :( https://news.ycombinator.com/item?id=6437715 https://news.ycombinator.com/item?id=6437715
- Kiro 13y agoNo, it's killed so no-one can upvote or reply. Not sure why though but maybe you were flagged as a spammer.
- Kiro 13y agoI really like this! A small question regarding the tech stack: how is the relationship AngularJS-Laravel-Rails working? What are you using Laravel for?
- jwblackwell 13y agoWe're two developers, to get to this point Henley built a crawler and search index around the data. This is then available via a private API. I (James) built the web app using Angular and Laravel. Laravel is currently just used for user auth and as a proxy to the api. Nearly all the front end logic is handled by AngularJS (it's a single page app).
- vdaniuk 13y agoYeah, quite interesting project. As someone who is interested in marketing and social media, I find value in the results of buzzsumo. I guess thats because posts on these topics tend to get a lot of shares and there are many signals to feed the algorithm.
- dougk7 13y agoThis is nice. Feature request: Add a safe-search setting.
- deleted 13y ago[deleted]
- jwblackwell 13y agoThe sorting can be adjusted - just click on a column to see the most popular stuff on Twitter, Linkedin: http://app.buzzsumo.com/#/top-content?result_type=twitter&num_days=7&general_article&infographic&video&guest_post&giveaway&q=Syria&page=1&type=articles&interview http://app.buzzsumo.com/#/top-content?result_type=twitter&nu... (twitter)
- abemassry 13y agoThis is awesome, we made a realtime news like this http://mashrd.com http://mashrd.com would love to work together.
- deleted 13y ago[deleted]
- hnriot 13y agoGoogle Chrome 30.0.1599.47 (Official Build 223717) beta OS Mac OS X All I get are angular javascript exceptions, no results, blank white page.
- jwblackwell 13y agoPlease do a hard refresh (ctrl-r) and let me know if this doesn't fix it (email in profile and on site). I had to push some bug fixes recently and you might have some old cache files causing problems.
- colmtuite 13y agoSeems like a pretty smart idea. Good work. I'm impressed by the speed. I like the tour too. I think the UI is very text-heavy right now, it might benefit from swapping some text for icons. I'd like to see more of a pitch by way of how this app can help me and why I should be using it. Other than that, good job. Hope it does well.
- parth21shah 13y agoAwesome tool and the couple of queries I made has pretty much accurate results
- bapi 13y agoReally good to know in few minutes what to know about a topic from social media perspective. I will not miss things of these social media I don't read but I have to know what they are talking about, and you search engine aggreator is good for this sum up.
- sixQuarks 13y agoGreat execution. My searches were actually very useful.
- patdennis 13y agoThis is extremely useful, I'm very impressed.
- jwblackwell 13y agoThanks! Don't forget to sign up to the beta if you haven't already. It's free and you can use the app unrestricted!
- skram 13y agoThis is really interesting but I worry about the share of the net you all have crawled/are actively gathering stats on. Perhaps you need a way to request the crawling of a site or keyword?