3 ms·
> I've built crawlers that retrieve billions of web pages every month. Wow, what were you doing with the data?
by mkl 9y ago
> I've built crawlers that retrieve billions of web pages every month.
Wow, what were you doing with the data?
- wrath 9y agoCompetitive intelligence. Crawling thousands of website, mashing up the data to analyze competitiveness between them, and selling it back. For example, cost of flights. Different websites provide different prices for the same flight. The technology crawls all the prices, combines the data, then resells it back to the websites. Everyone knows everyones prices, keeps competition high, lower prices for consumers.
- ketralnis 9y agoTravel companies pay their GDS for every search they do. It costs so much that it's the primary cost centre for some of them. You were costing them thousands of dollars a day.
- deleted 9y ago[deleted]
- RodericDay 9y agoGood.
- darksim905 9y agoGDS?