4 ms·
With all due respect, this poor design on part of Hadoop and/or Hadoop configuration. Hadoop workers should be bounded to certain amount hardware resources, and
by pacala 7y ago
With all due respect, this poor design on part of Hadoop and/or Hadoop configuration. Hadoop workers should be bounded to certain amount hardware resources, and killed if they surpass them. As opposed to killing random processes / kernel panic / etc. On Linux, this is realised via cgroups, which originated in Borg and today is widely used by Kubernetes.
- GauntletWizard 7y agoI am not-incredibly-surprised that none of these many talks about swap have mentioned kubernetes. Kubernetes does not solve this problem conclusively, but it does solve it to a huge extent. Your Application Cgroups have lower RAM priority than system tasks. You allocate enough RAM for system-level tasks in kubernetes ( --system-reserved and --kube-reserved). You run your application tasks in a fixed reservation, and any that are over their reservation are first to be killed. If application tasks are repeatedly getting killed, your node is faulty and gets restarted. All of this is done with little fuss. At some higher level, faulty processes are easy to identify because they are the most evicted (And if your system-level daemons are what's at fault, it gets really obvious because of how unobvious the application is) https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ https://kubernetes.io/docs/tasks/administer-cluster/reserve-...
- nerpderp82 7y agoI was only "a dev" and had an extremely difficult time getting cluster configuration changes pushed through. It was only because I bribed a sysop that we got a swap file added. We had our own Mr. No to deal with. Lots of things could/should and would be done differently.