5 ms·
Ask YC: Grabbing content
First of all I would like to thank some of you who offered some life saving advice for my previous question.
Now that we have pulled up our socks and are making some headway, we have run into our first content problem.
We are building a business listings directory for a niche market. We have identified a bunch of ad-hoc sources for sourcing content. We have manually gathered content for our mockup, but to take it to the next level we need a lot more content than what we have going.
What are the different techniques used to gather content? It would be great if some of you have had specific experience in gathering content for some kind of a business listings directory.
Looking forward to some valuable insight!
Thanks again!
- thorax 19y agoHmm, I really don't know how to help. I did just inquire into something not specifically related but might be randomly useful. There's a company called DeepData @ http://www.deepdata.com http://www.deepdata.com which might be of some use? They have APIs for business listing queries, and may be able to help on some categorizations.
- lux 19y agoIf you mean screen-scraping, isn't this roughly what you're looking for? http://www.crummy.com/software/BeautifulSoup/ http://www.crummy.com/software/BeautifulSoup/
- kleevr 19y agolixto
- FiReaNG3L 19y agoDapper.net is your friend
- davidw 19y agoJust be very very careful if you grab any application forms;-)
- spage 19y agoMy favorite technique is: wget URL > HTML tidy HTML > XHTML xslt [identity transform based content extraction] XHTML > XML XML > DB The whole process glued together with PERL or shell scripts. Depending on how you construct your content extraction, this technique can weather lots of the inevitable content style changes and easily adjusts when changes need to be made.