4 ms·
Hardest part of implementing a high-functioning production machine learning stack for me isn't the idea-articulating, prototyping, iterating, then polished refa
by textminer 14y ago
Hardest part of implementing a high-functioning production machine learning stack for me isn't the idea-articulating, prototyping, iterating, then polished refactoring. It's knowing when to go from a quick-producing language like Python/MATLAB/Julia to something painfully-written but smooth like C++ or into a scalable Elastic MapReduce or Mahout process (the former of which, sure, is language-agnostic).
You can only spend so much time optimizing on memory/CPU-times with smart data chunkings or low-dimensional representations or approximation operations. EC2 time and space is relatively cheap, but Python on a single machine with the multiprocessing module can only speed up by a multiple of < [# of Cores]...
- beambot 14y agoYou should checkout PiCloud [1] or MrJob [2]. Both of these seek to make MapReduce dead simple using Python. More importantly, you can do all your testing on a Desktop PC. Then when you need real horsepower, you just tell it to "spin up on EC2." Disclaimer: I have yet to use either, but I've heard good things. [1] http://www.picloud.com/ http://www.picloud.com/ [2] http://musicmachinery.com/2011/09/04/how-to-process-a-million-songs-in-20-minutes/ http://musicmachinery.com/2011/09/04/how-to-process-a-millio...