4 ms·
I interned this summer in a genetics lab, and was shocked the extent to which this is the case. Both the input and output files are absurdly huge, so I'm not su
by philipkimmey 16y ago
I interned this summer in a genetics lab, and was shocked the extent to which this is the case. Both the input and output files are absurdly huge, so I'm not sure anything else would've been viable, but still, at least as a student it always seemed like there should be "some other way."
After a week or two of getting to know the system, for clarification I asked: "You mean this entire sophisticated system is just some Perl, with a database that keeps track of the flat files?"
- protomyth 16y agoA lot of companies still do all the processing on text files as a nightly run (particularly those companies getting once a day data files (e.g. credit card rewards processors)). Some put it in an actual database, but you can really churn through text files faster than a lot of relational databases can do their queries. The most interesting example I saw of this was a company that had an automated process to add a header with time/date/source and check it into code control (subversion I think). The nightly batch job checked what it needed out and did all the processing including generating some control spreadsheets.