4 ms·
Yes I didn't mean to imply that, only that it can be challenging dealing with a resource constrained environment. For example our deep learning clusters do not
by malux85 9y ago
Yes I didn't mean to imply that, only that it can be challenging dealing with a resource constrained environment.
For example our deep learning clusters do not use hadoop, but use multi-process python daemons (written by me) and communicate with protocol buffers. Writing these high performance daemons myself has taught me a lot about cluster computing and some of the tradeoffs that have to be made in order to keep the GPU pipelines full.
For anyone else reading: Pyro is quite fun if you're in a limited environment and want to learn https://pythonhosted.org/Pyro4/ https://pythonhosted.org/Pyro4/ Also, setting up a spark https://spark.apache.org/ https://spark.apache.org/ environment can be done on smaller machines too
- stephenr 9y agoRight, but I think you still missed my point. The vast majority of software developers will interact with, if it’s running clustered it’s because the project needs higher availability for that service (typically the database) and potentially higher performance in production.
- malux85 9y agoYeah, I was commenting on learning distributed programming, rather than configuring distributed service (regardless if goal is performance or redundancy). Configuring distributed services is boring
- stephenr 9y agoOh right well I guess all the ops in the world should just pack up and go home, it’s “boring” so no one ever need learn about it?