3 ms·
Put planet.osm on torrent. Allow "scraping" only through torrent. Now scrapers are sharing the network load between themselves. Not to mention improved network
by butz 2y ago
Put planet.osm on torrent. Allow "scraping" only through torrent. Now scrapers are sharing the network load between themselves. Not to mention improved network speed, as probably they all sit on the same AWS instance.
- Firefishy 2y agoOur data is already publish via torrent, see https://planet.openstreetmap.org https://planet.openstreetmap.org Our data, including minutely updates is available via public S3 buckets (EU & US) supported by the AWS Open Data Sponsorship Program.
- butz 2y agoAs expected. Now for the hard part: how to redirect scrapers to use torrents, and keep them uploading data? Should we write some "data scraping tutorials" specifically targeting OSM and describe using torrents as the best method?