4 ms·

Is there any benefit to having your own clustered system vs just using AWS or other cloud services?

by bkcreate 9y ago

jlgaddis 9y ago
For learning, perhaps. The power costs would likely exceed your AWS bill very quickly, though.
stephenr 9y ago
For learning? Absolutely, particularly compared with something like aws.
malux85 9y ago
short answer: not really. longer answer: depending on how old the systems are, they're likely not useful for learning modern distributed computing. For example, a friend of mine had four 4 core, 8GB machines and tried to configure a hortonworks cluster, but the system spent so much time just running the cluster state, that you could only do trivial computing on top of the cluster before the nodes started timing out. He then spent ages disabling unnecessary services, re configuring components to be more memory efficient - and spent so long doing all of this maintenance he never actually learnt how to compute on top of the cluster, which was his original goal. So it's best just to pay a few hundred a month to amazon and run your own small cluster that you can spin up and down as you need
- stephenr 9y ago
  FYI Hadoop is not a synonym for “cluster”. A clustered system could be a load balancer and a few servers running percona cluster, or a couple of ldap masters with read-only relicas and proxy-writes. Doing it on bare metal also gives exposure to the parts amazon specifically don’t
  - malux85 9y ago
    Yes I didn't mean to imply that, only that it can be challenging dealing with a resource constrained environment. For example our deep learning clusters do not use hadoop, but use multi-process python daemons (written by me) and communicate with protocol buffers. Writing these high performance daemons myself has taught me a lot about cluster computing and some of the tradeoffs that have to be made in order to keep the GPU pipelines full. For anyone else reading: Pyro is quite fun if you're in a limited environment and want to learn https://pythonhosted.org/Pyro4/ https://pythonhosted.org/Pyro4/ Also, setting up a spark https://spark.apache.org/ https://spark.apache.org/ environment can be done on smaller machines too
    - stephenr 9y ago
      Right, but I think you still missed my point. The vast majority of software developers will interact with, if it’s running clustered it’s because the project needs higher availability for that service (typically the database) and potentially higher performance in production.
      - malux85 9y ago
        Yeah, I was commenting on learning distributed programming, rather than configuring distributed service (regardless if goal is performance or redundancy). Configuring distributed services is boring
        stephenr 9y ago
        Oh right well I guess all the ops in the world should just pack up and go home, it’s “boring” so no one ever need learn about it?

Is there any benefit to having your own clustered system vs just using AWS or other cloud services?

jlgaddis 9y ago

For learning, perhaps. The power costs would likely exceed your AWS bill very quickly, though.

stephenr 9y ago

For learning? Absolutely, particularly compared with something like aws.

malux85 9y ago

short answer: not really. longer answer: depending on how old the systems are, they're likely not useful for learning modern distributed computing. For example, a friend of mine had four 4 core, 8GB machines and tried to configure a hortonworks cluster, but the system spent so much time just running the cluster state, that you could only do trivial computing on top of the cluster before the nodes started timing out. He then spent ages disabling unnecessary services, re configuring components to be more memory efficient - and spent so long doing all of this maintenance he never actually learnt how to compute on top of the cluster, which was his original goal. So it's best just to pay a few hundred a month to amazon and run your own small cluster that you can spin up and down as you need

stephenr 9y ago
FYI Hadoop is not a synonym for “cluster”. A clustered system could be a load balancer and a few servers running percona cluster, or a couple of ldap masters with read-only relicas and proxy-writes. Doing it on bare metal also gives exposure to the parts amazon specifically don’t
- malux85 9y ago
  Yes I didn't mean to imply that, only that it can be challenging dealing with a resource constrained environment. For example our deep learning clusters do not use hadoop, but use multi-process python daemons (written by me) and communicate with protocol buffers. Writing these high performance daemons myself has taught me a lot about cluster computing and some of the tradeoffs that have to be made in order to keep the GPU pipelines full. For anyone else reading: Pyro is quite fun if you're in a limited environment and want to learn https://pythonhosted.org/Pyro4/ https://pythonhosted.org/Pyro4/ Also, setting up a spark https://spark.apache.org/ https://spark.apache.org/ environment can be done on smaller machines too
  - stephenr 9y ago
    Right, but I think you still missed my point. The vast majority of software developers will interact with, if it’s running clustered it’s because the project needs higher availability for that service (typically the database) and potentially higher performance in production.
    - malux85 9y ago
      Yeah, I was commenting on learning distributed programming, rather than configuring distributed service (regardless if goal is performance or redundancy). Configuring distributed services is boring
      - stephenr 9y ago
        Oh right well I guess all the ops in the world should just pack up and go home, it’s “boring” so no one ever need learn about it?