4 ms·
A non-scientific thing I've been doing for the last few months at the day job, with Julia. 1) querying a time series database of systems metrics at scale for (
by hpcjoe 5y ago
A non-scientific thing I've been doing for the last few months at the day job, with Julia.
1) querying a time series database of systems metrics at scale for (large) fleets. This is being done via a JSON API. Directly in Julia.
2) Creating data frames from these queries, and performing fleet wide analytics. Quickly. Millions to hundreds of millions of rows in the data frames, typically 4-20 columns. Directly in Julia, no appeal to a 2nd language.
3) leveraging the power of the language to post process these data sets before analysis, to remove an "optimization" that reduced data quality.
4) operate quickly on gigabytes of queried data, threading and sharding my requests, as the server can't handle large requests, but it can handle parallel ones. Poor design, but I can work around it ... trivially ... with Julia
5) creating jupyter lab notebooks for simple consumption of these more complex data sets by wider audiences, complete with plots, and other things.
No science done here ... well ... data science maybe ... and this is specifically in support of business analytics, process optimization, etc.
Julia is an excellent language for this, 10 out of 10, would recommend.
- neolog 5y agoWhat's a fleet in this case?