3 ms·
Probably not much. I started working with bigger data some months ago and now I notice that I need some real techniques and not just my "ok try this and this".
by pfanner 14y ago
Probably not much. I started working with bigger data some months ago and now I notice that I need some real techniques and not just my "ok try this and this". I'm coding in C (where I "create" the data (numerical integration of stochastic differential equations)) and Python (plotting). I need methods/algorithms/techniques to analyse the data "on the fly" because I can't save it all (it's too much data).
- noelwelsh 14y agoThis might be of interest: http://noelwelsh.com/streaming-algorithms/2012/08/29/lean-data/ http://noelwelsh.com/streaming-algorithms/2012/08/29/lean-da... Don't have a great deal of time right now so drop me an email if you'd like more info (see profile) and I'll get on it tomorrow.
- tst 14y agoJust a small tip which may ease the search for methods. The general term for "on the fly" learning is online learning [1]. The rest depends on your problem but there are often online variants of offline methods, e.g. when you work with Gaussian process regressions [1]: http://en.wikipedia.org/wiki/Online_machine_learning http://en.wikipedia.org/wiki/Online_machine_learning