5 ms·
Catboost is implemented in C. Does anyone know how stuff like this is run at scale over multiple machines ? For example, if I want to run a distributed computat
by sandGorgon 9y ago
Catboost is implemented in C. Does anyone know how stuff like this is run at scale over multiple machines ? For example, if I want to run a distributed computation in spark - I use some primitives that are distributed in nature.
But how does someone use Catboost across a cluster of 10 machines ? All the help documents are heavily single machine. Is there any kind of infra-framework that will distributed the jobs across all the machines running Catboost, etc ?
- ma2rten 9y agoYou could just run 10 versions of the same classifier and put a load balancer in front of it.
- zintinio5 9y agoThe problem is distributed training, not inference.
- s0ulmate 9y agoHey! CatBoost team here. CatBoost is currently single host, the version of training distributed on cluster will be open-sourced later.
- zintinio5 9y agoThere might not be a need to do so at the moment. Otherwise, they might open that up at a later time.
- sandGorgon 9y agoI think someone of Yandex scale would be using a HUUGE cluster. I wonder if there is already something out there to do it.
- prdonahue 9y agoLast time I did something in C requiring spreading work across a cluster of machines it was with MPI(CH). Docs available at mpich.org. This was Monte Carlo simulation for hyper-dimensional (~8 IIRC) asset allocation and thus the simulations were easily divisible—recombining was just simple arithmetic—and the network I/O was minimal. Interesting tidbit (to me anyway): This was 2006, before you could use something like AWS for the purpose and we were trying to keep costs to a minimum. (IBM had their "public one" grid but it was unusable.) The Core 2 Duo processor had just come out so I hired an intern from Cal—now a PhD and brilliant engineer at Netflix—to figure out the optimal overclocking rig and we built 32 chassis consisting of just motherboard, RAM, NIC, custom cooling/heatsink, and power supply. The problem was then how to deploy these in a colo. At the time there were some low end providers at 200 Paul willing to get creative with a cabinet so I found a machinist (metal worker?) able to cut some custom aluminum shelving on top of which we could stack the ATX cases. Rigged the boxes up to network boot off one of the nodes, compiled our application with Intel's C++ compiler to take advantage of the SIMD/SSE3 instruction set, and away we went running billions of simulations on a startup budget.
- tlarkworthy 9y agoDid it work out?
- sandGorgon 9y agothis is incredible. could you talk about OS and deployment ?
- srean 9y agoIf you were using a distributed workload (not clear form your comment if you were) curious if you tried out multiple NICs per machine. By 2006 there already were mutliple queues per machine, but prior to that, multiple NICs were sometimes helpful.
- kgwgk 9y agoCatboost is implemented in C++. Like XGBoost, LightGBM, or TensorFlow, by the way.
- defgeneric 9y agoThe github page shows a large % of C LOC but that's mostly all in 3rd-party libraries like openssl. CatBoost itself seems to be C++.