4 ms·

“The Weather Service’s proposed remedy is to limit users to 60 connections per minute on a large number of its websites that provide weather observations, forec

“The Weather Service’s proposed remedy is to limit users to 60 connections per minute on a large number of its websites that provide weather observations, forecasts, warnings, computer model data, air quality information, aviation weather support and ocean conditions.” Well that sounds incredibly reasonable in general, but it completely depends upon what type of interface they’re providing. Do you have to query all of those things separately for every locality you serve? Honestly, this is why so many big companies still prefer things that HN would call “stupid” like an FTP drop of a file 5 times per day/hour/etc. Every FTP implementation I’ve ever seen has included either checking the timestamp, timestamping the filename and filtering on that, or moving to an “archive” folder after processing. Most of the API implementations I see ignore the fact that HEAD requests exist, cache headers can and should be used, etc.

tenebrisalietum 6y ago
Still, it's not 1971 anymore. Can we move to SFTP at least?
- chrismeller 6y ago
  Heh. Well, perhaps, but for public data it doesn’t really matter in general, does it? Just like that installer you just downloaded off the internet, did you check the md5/sha to make sure it wasn’t tampered with? Just because it was over HTTPS doesn’t really mean much.
- adamcstephens 6y ago
  Move to SFTP requires you to use authentication and increases load on the backend servers.
brootstrap 6y ago
You have any experience with numerical weather prediction files? The ftp files you download are typically 'the whole globe' with a bunch of variable stacked like layers of images in a big image file. One of the problems is that a typical numerical weather prediction output has so much fucking data you dont need. GFS for example. IT has maybe 200 or so timesteps in each forecast, runs 4x a day. There are a shit load of configs to run for the GFS. And each config + output has maybe 300/400 variables in each file? So you download 200 files, each with 400 images in it, each image is pretty dense. Each file is ~200 megs, X 200 files. You do the math. Get that 4x a day. And that is just one forecast model. We have lots more, and many more that have higher resolution and denser data. The biggest problem i think is they have a goddamn free-ass oldschool FTP server. They are allowing dimwit grad school scrip kiddies to code up whatever shit they want. Infinite loops to try and download gobs of data you dont understand? sure go ahead. They dont even know how to trakc users. they want to keep an open server, and use IP Address as unique ID. I suggested to use fucking tokens, or username/pass combo at least haha

tenebrisalietum 6y ago

Still, it's not 1971 anymore. Can we move to SFTP at least?

chrismeller 6y ago
Heh. Well, perhaps, but for public data it doesn’t really matter in general, does it? Just like that installer you just downloaded off the internet, did you check the md5/sha to make sure it wasn’t tampered with? Just because it was over HTTPS doesn’t really mean much.
adamcstephens 6y ago
Move to SFTP requires you to use authentication and increases load on the backend servers.

brootstrap 6y ago

You have any experience with numerical weather prediction files? The ftp files you download are typically 'the whole globe' with a bunch of variable stacked like layers of images in a big image file. One of the problems is that a typical numerical weather prediction output has so much fucking data you dont need. GFS for example. IT has maybe 200 or so timesteps in each forecast, runs 4x a day. There are a shit load of configs to run for the GFS. And each config + output has maybe 300/400 variables in each file? So you download 200 files, each with 400 images in it, each image is pretty dense. Each file is ~200 megs, X 200 files. You do the math. Get that 4x a day. And that is just one forecast model. We have lots more, and many more that have higher resolution and denser data. The biggest problem i think is they have a goddamn free-ass oldschool FTP server. They are allowing dimwit grad school scrip kiddies to code up whatever shit they want. Infinite loops to try and download gobs of data you dont understand? sure go ahead. They dont even know how to trakc users. they want to keep an open server, and use IP Address as unique ID. I suggested to use fucking tokens, or username/pass combo at least haha