3 ms·

This was one of the most surprising parts about Nomad for me. I really want to like it, and for the most part I think it's a great experience, but if you're lo

This was one of the most surprising parts about Nomad for me. I really want to like it, and for the most part I think it's a great experience, but if you're looking for a lightweight single-host PaaS-type thing, the minimum recommendation and default configuration for 3 servers is a bit wild. I have Nomad running in single host mode on a DO box for personal stuff, and I get the distinct feeling I'm not really meant to be doing it. 3 instances minimum is a big ask IMO

3np 6y ago
Go read the best practices for Vault and the "best practices minimum" are even crazier. IME it's all fine to run all these on low-resource single instance in practice, it's just the target audience for the "minimum configuration" isn't really your homelab tinkerer or even SMB/small start-up. Consider that Hashicorp's revenue streams are consulting, enterprise licenses and, since recently, hosted PaaS. If a penny-pinching user comes with replication issues because of resource contention they can just point to those docs. RedHat does the same thing for minimum recommendations for a lot of linux software that in practice will work perfectly fine on much smaller machines. Your single host mode DO box is most likely perfectly fine. You just won't have the availability benefits during downtime of that instance but, well, duh. I guess the one practical catch is that if you feel strongly about security and noise neighbor issues, you really should at least separate out Consul Server/Vault Server/Nomad Server into separate machines. Just whatever you do, make sure to run an odd number of servers (1 is better than 2, 3 is better than 4), or you may run into split-brain grief. Clients are fine to scale as you see fit as they don't partake in raft consensus. So you can have a single instance "client/server", but if you add just a single new instance to schedule jobs on, make it's a pure client. I have a dozen Nomad clients and about 20 jobs at home. My active Nomad leader is yet to use over 300 MiB RAM, 10% CPU or 0.2 load on RPi-equivalents, even during leader rotation and restarts. It's only when you start to have very high throughput of evaluations and allocations and start to put pressure on raft that you may need to increase your raft_multiplier to make it more relaxed with timeouts. Tl;dr take the minimum configuration advice with a heap of salt.
- 3np 6y ago
  For home-use, I think you could run your servers without issues (apart from that SD cards are not great at all for root or raft storage) on $20 RK3328 SBCs like a NanoPi NEO3. I may scale down to something like this at some point, given what I have now (Rpi4B-equivalents) is pretty overkill. https://www.friendlyarm.com/index.php?route=product/product&path=69&product_id=279&sort=p.price&order=ASC&limit=75 https://www.friendlyarm.com/index.php?route=product/product&...
heipei 6y ago
3 servers is always going to be the minimum recommended production configuration for any highly available system (container schedulers, databases, search indexes, etc). You can run Nomad perfectly fine just by itself on a single server, you don't even need Consul in some cases.

3np 6y ago

Go read the best practices for Vault and the "best practices minimum" are even crazier. IME it's all fine to run all these on low-resource single instance in practice, it's just the target audience for the "minimum configuration" isn't really your homelab tinkerer or even SMB/small start-up. Consider that Hashicorp's revenue streams are consulting, enterprise licenses and, since recently, hosted PaaS. If a penny-pinching user comes with replication issues because of resource contention they can just point to those docs. RedHat does the same thing for minimum recommendations for a lot of linux software that in practice will work perfectly fine on much smaller machines. Your single host mode DO box is most likely perfectly fine. You just won't have the availability benefits during downtime of that instance but, well, duh. I guess the one practical catch is that if you feel strongly about security and noise neighbor issues, you really should at least separate out Consul Server/Vault Server/Nomad Server into separate machines. Just whatever you do, make sure to run an odd number of servers (1 is better than 2, 3 is better than 4), or you may run into split-brain grief. Clients are fine to scale as you see fit as they don't partake in raft consensus. So you can have a single instance "client/server", but if you add just a single new instance to schedule jobs on, make it's a pure client. I have a dozen Nomad clients and about 20 jobs at home. My active Nomad leader is yet to use over 300 MiB RAM, 10% CPU or 0.2 load on RPi-equivalents, even during leader rotation and restarts. It's only when you start to have very high throughput of evaluations and allocations and start to put pressure on raft that you may need to increase your raft_multiplier to make it more relaxed with timeouts. Tl;dr take the minimum configuration advice with a heap of salt.

3np 6y ago
For home-use, I think you could run your servers without issues (apart from that SD cards are not great at all for root or raft storage) on $20 RK3328 SBCs like a NanoPi NEO3. I may scale down to something like this at some point, given what I have now (Rpi4B-equivalents) is pretty overkill. https://www.friendlyarm.com/index.php?route=product/product&path=69&product_id=279&sort=p.price&order=ASC&limit=75 https://www.friendlyarm.com/index.php?route=product/product&...

heipei 6y ago

3 servers is always going to be the minimum recommended production configuration for any highly available system (container schedulers, databases, search indexes, etc). You can run Nomad perfectly fine just by itself on a single server, you don't even need Consul in some cases.