4 ms·

Rachel’s starting point here is a fair few steps ahead of ”it’s called chef”. For one, chef only fixes the things that you explicitly tell it to. When you have

by pdpi 9y ago

Rachel’s starting point here is a fair few steps ahead of ”it’s called chef”. For one, chef only fixes the things that you explicitly tell it to. When you have enough machines and enough people poking at them, you’re more or less guaranteed that someone somewhere will have gotten a machine or three in a state chef can’t recover from — hence her suggestion of reimaging.

isostatic 9y ago
Always pxeboot, then "re-imaging" is a matter of "turn it off and on again". If you have a hardware failure how would you rebuild? Of course this all takes time to implement and manage -- it's a tradeoff.
- pdpi 9y ago
  How do you manage the network boot server? How will pxeboot interact with hosting your rack switch images?
  - isostatic 9y ago
    High availability IP for your dhcp for "next-server", so you can boot one server from another one. Mikrotik routers and switches can boot from dhcp (or bootp?), but yes a typical switch can't. Not a network person, but I assume you can give fine grained enough control that you can't do a "copy running-config startup-config" on a cisco switch, so have a startup config that boots to a known basic state then tftps it's config from your HA dhcp server.
Fnoord 9y ago
> If you want to impress me, set up a system at your company that will reimage a box within 48 hours of someone logging in as root and/or doing something privileged with sudo (or its local equivalent). Why revert it in the first place? Why not deny the changes? Doesn't this indicate that the wrong people have root access?
- softawre 9y ago
  sometimes the best way to test a fix to something, before you make it permanent in the deployment-as-code, is to test it on the box. This just makes you sure actually go back and change the deploy code, else it breaks in ~2 days.
  - Fnoord 9y ago
    You could just have one or more specific development/test box(ex) to try changes on, and then implement those on production.

by pdpi 9y ago

isostatic 9y ago

Always pxeboot, then "re-imaging" is a matter of "turn it off and on again". If you have a hardware failure how would you rebuild? Of course this all takes time to implement and manage -- it's a tradeoff.

pdpi 9y ago
How do you manage the network boot server? How will pxeboot interact with hosting your rack switch images?
- isostatic 9y ago
  High availability IP for your dhcp for "next-server", so you can boot one server from another one. Mikrotik routers and switches can boot from dhcp (or bootp?), but yes a typical switch can't. Not a network person, but I assume you can give fine grained enough control that you can't do a "copy running-config startup-config" on a cisco switch, so have a startup config that boots to a known basic state then tftps it's config from your HA dhcp server.

Fnoord 9y ago

> If you want to impress me, set up a system at your company that will reimage a box within 48 hours of someone logging in as root and/or doing something privileged with sudo (or its local equivalent). Why revert it in the first place? Why not deny the changes? Doesn't this indicate that the wrong people have root access?

softawre 9y ago
sometimes the best way to test a fix to something, before you make it permanent in the deployment-as-code, is to test it on the box. This just makes you sure actually go back and change the deploy code, else it breaks in ~2 days.
- Fnoord 9y ago
  You could just have one or more specific development/test box(ex) to try changes on, and then implement those on production.