4 ms·
1) Will there be limits on the size of data sets/ what size data are you optimizing for? Some algos focus on hundreds of records, some on billions, and the idea
by morganherlocker 13y ago
1) Will there be limits on the size of data sets/ what size data are you optimizing for? Some algos focus on hundreds of records, some on billions, and the ideal system for either are quite different (small data works great by transferring data back and forth over http, data with hundreds of thousands of records and up... not so much).
2) Same question as above, but for processing times. Some algos are aimed at operating on the fly and might take ~1 second or less. Others (many that I deal with quite often), might run for days, or even weeks. What sort of processes are you optimizing for, and are long running procs on the radar?
It is an interesting idea, but there is a high bar when competing against my language's package manager, and the 10s of thousands of "algorithms" already out there. Best of luck!
- platypii 13y agoThe data question is something we are taking very seriously. I talked a bit more about this in another comment, but we know that in many cases data is heavy and is often the limiting factor. We are building a robust data platform ontop of Hadoop + Spark, so be able to handle large amounts of data, and so that we can deploy code to where the data is. That said, I think there are many algorithms that will work as standalone algorithms. Lots of machine learning type algorithms are mostly CPU-bound. There are companies like http://www.kooaba.com http://www.kooaba.com that do image recognition as a service over HTTP. Also things like Siri and Google Now seem to work well enough, despite network latency. Thanks for the feedback!