Story time... My company Datastreamer has been around for a decade. We provide crawl data (usually a massive amount of crawl data, north of 300GB per day) to

Story time... My company Datastreamer has been around for a decade. We provide crawl data (usually a massive amount of crawl data, north of 300GB per day) to our customers. ... so we have a LOT of real-world experience pushing data to customers in production over long time periods. Here's what we've learned. Networking libraries around HTTP are and have been fundamentally broken for a long time and they're broken in pathological ways that you don't realize until years later in production. DNS caching is a good one. A lot of systems do infinite DNS caching. Java, until at least Java 8, does infinite DNS caching. Some do infinite HTTP timeout. Timeouts are awesome. You should use a timeout. Without a timeout if the network breaks your code just locks up. Some libs provide no API to change TCP buffer sizes (which you have to do at the kernel level). So about 5 years ago we took a harsh stance. NO CUSTOM CLIENTS. We have a streaming firehose client that we implemented from the ground up to do everything properly. The API is literally that we just stream JSON files to disk. It's a docker container now so not too hard to deploy. Your job is to just to listen to the disk and wait for new files to be written. We do a move from a tmpdir to the final dir so the entire file is written and you don't have to worry about partial reads. About 80% of our customers love it. The other 20% of customers seem to initially hate it and we have to explain to their CTO or senior architect that, no, you DO NOT want to implement this from the ground up. What happens is that it works immediately, but then 18 months in it will break pathologically and everyone running it has moved on or it's in some datacenter that no one has access too. This causes us to break our SLAs and means we have upset customers. This decision by far was one of the best decisions I've ever made and has really helped our growth and stability over time. It's really really really nice to keep customers for 5-10 years. They're happy and you get steady checks and predictable growth.