6 ms·
Um, maybe to use it in an commercial setting? I've literally no idea what you're talking about as I've never come across that before. I've always just pulled do
by cwal37 11y ago
Um, maybe to use it in an commercial setting? I've literally no idea what you're talking about as I've never come across that before. I've always just pulled down the station stuff I need by hand (since you can get all the station's data at once), or by some lightly mechanized webscraping. Your comment sounded significantly more chill than what it seems like you might need.
It wouldn't be terribly difficult to mechanize this and get all the data at once, but I thought you wanted to occasionally see how much it rained, not build a service out of this.
Does this page[1] not contain what you need?
[1] http://cdiac.ornl.gov/ftp/ushcn_daily/ http://cdiac.ornl.gov/ftp/ushcn_daily/
- tcdent 11y ago*.txt.gz is not exactly a modern data format. I share the parent's sentiment; finding easy access to good weather data is hard. Do you want your side project to be an experience in learning how to parse obscure formats and coordinating obsolete dependencies, or HTTP GET some JSON and immediately do something with the data?
- cwal37 11y agoNot modern doesn't have to mean difficult to parse, it depends on your personal outlook. You can also download individually customized csv files if you want, which my original link led to. csv and txt files seem eminently reasonable to me when dealing with a huge historical cache of data that's being put out by a public institution for free.
- Eridrus 11y agoYou're shitting me, right? They're literally giving you all of the data in a plaintext format with documentation, and you're still complaining? Sure, having everything handed to you on a silver platter is nice, but why should they do your project for you?
- spiralpolitik 11y agoImagine what is going to happen when they discover the entire financial system is built on copying CSV files around by FTP.
- kbenson 11y agoDon't forget the emails back and forth and manual fixes and resubmissions from either end when there's a problem causing the CSV to not parse!
- mvf0 11y agoI could write a node server that converts all that fixed length ASCII data into JSON faster than you guys can complain.
- deleted 11y ago[deleted]
- ska 11y agoWhat entitled nonsense. *.txt.gz is not a problem for anyone of even minimum competence.
- mikestew 11y ago> *.txt.gz is not exactly a modern data format. You know what you get with a format that's not "modern"? A huge, mature set of tooling that works. For instance, without trying because just responding to this whinging is barely worth the wear on my keyboard, I'd be disappointed if OS X wouldn't cheerily open the file, and unarchive it into the original directory structure simply by typing "open weather_data.txt.gz" from a terminal window. In other words, one could script it. I'd show you how, but holding your hand leaves me with only one hand for typing. > HTTP GET some JSON and immediately do something with the data Well, that's the magic of JSON: any code I write just wondrously knows what to do it without me telling it how to parse it. How can poor ol' ASCII text possibly compete?
- tcdent 11y agoWow, you guys really missed the point. Why can't we ask for nice things, even if they are "free"? How are you going to get a kid interested in programming historical weather analysis when he's got to figure out how to configure an entire stack just to query your dataset. If you really want my opinion, this should be an AI problem, not yet another tedious MVC application. Furthermore, when I jump into projects, I am most definitely appreciative that there is a responsive system on the other end, no matter how much I'm getting paid. I'm not some noob and yes I have imported gigabytes of minimally-structured data into my own applications, but I don't want to have to do that every time. Especially just to tinker. If you're not thinking about how your software development experience can be made quicker, more direct, and less painful, I pity you. Shit on JSON all you want, but I'll take a populated native object and a blinking cursor over FTP'd text files any day.
- kaishiro 11y ago"Shit on JSON all you want, but I'll take a populated native object and a blinking cursor over FTP'd text files any day." I'm not sure this particular hill is worth dying for. Remember, text was the universal transport long before JSON was (and probably will be long after).
- nl 11y agowget -O - http://cdiac.ornl.gov/ftp/ushcn_daily/state01_AL.txt.gz | gunzip -c | in2csv -s schema.txt | csvjson Fixed that for you. And in less space than it took for you to complain.