Every unbounded attempt (e.g. scraping) at collecting data in the wild that I've seen (and I've seen a lot) ends up borrowing against future tech (just push thr

Every unbounded attempt (e.g. scraping) at collecting data in the wild that I've seen (and I've seen a lot) ends up borrowing against future tech (just push through, we'll solve it at scale) and ends up with an increasingly unmaintainable pile of "integrations" when that advantage of scale doesn't arrive. After you've done and seen this a few times you want to give some avuncular advice, but people generally will have none of it. There are a few hints of it here. Your comment about brittleness. The comment about having them send USB drives. Retaining the unaltered original data. "3000 small efforts". Typically there won't be do many basic formats, but even the version of the software may treat the semantics of that format differently. (Could be a bug!) This is not a new problem. It's been around at the network layer long enough that there is an early RFC ranting about it in terms of "heffalumps". If this is not for profit, feel free to stalk me and reach out for that avuncular advice.