4 ms·
The sites/apps will be writing (as well as reading) ALL the time... One of them is a (proprietary) web crawler that will crawl and archive certain types of web
by webtechgal 10y ago
The sites/apps will be writing (as well as reading) ALL the time... One of them is a (proprietary) web crawler that will crawl and archive certain types of web resources. (Mainly RSS/XML feeds). Then, another one will crawl/archive URLs, take a screen shot of the URLs too and store them as gif/jpeg/png files.