outpan

4 ms·

1.
by outpan 9y ago
Outpan here, We use a combination of polite massive-scale web crawling and user contribution. We had an in-house web crawler for years until recently when we released it as a stand-alone service.
2.
Show HN: A Multi-billion page web crawler in the cloud(mixnode.com)
14 points by outpan 10y ago|0 comments
3.
by outpan 10y ago
Semantic web is too good of an idea not too iterate on constantly :)
4.
by outpan 10y ago
haha. We should use this as the project mantra.
5.
by outpan 10y ago
Would you be open to having a little chat via email? hi@outpan.com
6.
by outpan 10y ago
This is great! thanks. I will look into adding the dbpedia.org data.
7.
by outpan 10y ago
"a journey of a thousand miles begins with a single step"
8.
by outpan 10y ago
As for your question about real world applications: Outpan was only a product database up until last week. It is used in over a hundred apps, some with more than a million users. I expect this to work on a larger key space as well. It is in
9.
by outpan 10y ago
I think graph overlap is what actually determines what data is "accurate". There are currently bots writing to the database by people who are not connected and I have yet to see the overlap (since the key space is too large for th
10.
by outpan 10y ago
I was driving so couldn't stay on the phone. The number of keys for different concepts is just very large (if not infinite for the sake of avoiding philosophical debates). It will take a some time to have enough data to cover even popu
11.
by outpan 10y ago
Up until last week, outpan was strictly used for gathering data on product barcodes. We are in the process of adding data in other categories.
12.
by outpan 10y ago
Twitter url seems like a strong key since it is referenced in many other contexts. As for curation, it is intended to provide examples of what key, attr and values are regardless of their content. This is an experimental feature and might b
13.
Show HN: A database of everything (over 55M keys)(outpan.com)
97 points by outpan 10y ago|42 comments
14.
by outpan 10y ago
Common Crawl is great! however, some use cases require larger crawls with a higher frequency.
15.
by outpan 10y ago
Thanks a lot! This sounds reasonable. Did you guys look into professional services for this?
16.
by outpan 10y ago
Would you be able to share what your stack was? and the resources it took? Thanks a lot.
17.
by outpan 10y ago
I'm not sure how he manages to crawl with this speed using such low amount of resources. We did a benchmark on Nutch and couldn't really pass the 10-14 M(B)ps on a $1200/month machine. Even though we hired a professional to o
18.
by outpan 10y ago
That post is what triggered my Ask post. The problem is the huge contrast with https://www.quora.com/How-much-would-it-cost-to-crawl-1-bill... Even taking into account the drop in prices on AWS. Also, if you take a quick lo
19.
Ask HN: Has anyone ever crawled over a billion pages? How much did it cost?
10 points by outpan 10y ago|14 comments
20.
by outpan 10y ago
Awesome job! For the life of me I can't figure out how you manage to crawl over a billion web pages (even in 2-3 months), index the data and run the server with €300 per month. Especially the crawler part...
21.
Show HN: The largest database of products with a free API(outpan.com)
3 points by outpan 10y ago|0 comments
22.
Random lyrics generated by a Markov Chain algorithm(azlyrics.com)
4 points by outpan 12y ago|1 comments
23.
by outpan 12y ago
Up to 2000 calls/minute. We will hopefully increase this soon.
24.
by outpan 12y ago
That's true; the mandatory sing-up is a temporary measure, I'm sure there are a lot of ways to fight spammers including the method you mentioned. Thank you for your suggestion.
25.
by outpan 12y ago
wow, thanks a lot for your thorough advice :) tbh I really hate putting road blocks in front of people who would like to contribute to the database, however we were the victim of a huge spam attack with completely random ip rotations. I wi
26.
by outpan 12y ago
user contributions, purchased databases, paid moderation and web crawlers.
27.
Show HN: A free ISBN/UPC/EAN database and API with 30+ million items(outpan.com)
28 points by outpan 12y ago|9 comments