3 ms·
I would design a project with the following components. 1. A web crawler/spider that visits several sites every day. While there the crawler should captures
by mswen 13y ago
I would design a project with the following components.
1. A web crawler/spider that visits several sites every day. While there the crawler should captures data in a systematic way through parsing the DOM, using regular expressions and accessing APIs (not all on the same site).
2. Then create scripts that clean, organize and store the data in a database of your choosing.
3. Add value through algorithms and analysis. Apply some analytical processes to the data to create new insight. It might be statistical processing, it might be machine learning or even just a clever mash-up from the multiple sites.
4. Design for human consumption. Practice design principles and data visualization to create an interesting and insightful presentation layer for human readers.
5. Design for machine consumption. Now make your new value-added data available to others in an API that other systems can consume.
All of this can be accomplished on a VPS server costing you $12 a month. You can build it all out of open source software.
While what I described could be quite complex and the basis for a business, it could also be very limited in scope, such that the main complexity is learning the various technologies and showing that you have developed the skills to put together a full working system.