4 ms·

This seems wrong to me. This was the state of the art ~3 years ago. Now, I feel like all of the machines should be provisioned already with an OS, and a basic i

by sargun 11y ago

This seems wrong to me. This was the state of the art ~3 years ago. Now, I feel like all of the machines should be provisioned already with an OS, and a basic image, and a orchestration system like CoreOS / Mesos / Docker should specialize them. IMHO, requiring hardware, or the entire machine should be exception, not the rule.

packetized 11y ago
Wait, why? Are you advocating for the use of an abstraction layer where there isn't always a business case for using one?
detaro 11y ago
Just because you use Containers/VMs for most of your apps doesn't mean that the lower levels don't need attention: installing OSes in the first place, hardware testing (both initially and to identify defects later), ... And for important fileservers and databases you're going to run on specific hardware for a long time.
kbar13 11y ago
you realize the cloud runs on hardware right. machines don't magically come with everything installed and configured.
ownagefool 11y ago
Even if you were going to run CoreOS or Mesos on the machine, you'd still manage it booting your specific image, which you can change, rather than trusting the pre-installed dell verion and managing that relationship. Now there's probably some room for debate on whether these guys job should just be outsourced to Amazon, but github has some pretty good uptime and they seem to know what they're doing, thus they've probably already won that debate.
lwhalen 11y ago
Sometimes you have a workload that really isn't a fit for virtualization/containers/whatever the latest Rails hotness is, at all, and you just need to throw a couple of cargo trailers of insanely massively-spec'd servers at the problem. In those cases, your 'old school' server provisioning toolkit had better be on-point. It's easy to forget just how ridiculously powerful bare iron is these days. Go to Dell.com and see how much RAM you can cram into a U or three or four today in 2015. Or see how many IOPS a modern NetApp or Symmetrix (EMC) can push with 'flashcache' or million-dollar SSDs. It is ridiculous, and while a lot of those platforms are meant for 'building your own private cloud', etc, there's a non-trivial amount of workloads/projects where bare-iron is the best tool for the job.
- icebraining 11y ago
  Containers are just a namespacing tool, though; you're still running on bare metal (well, bare Linux). Docker in particular runs on AuFS, which is slow, but other containerization tools just use a chroot.
  - vidarh 11y ago
    Docker can run on any number of things, include btrfs and overlayfs+ext4, as well as devicemapper. E.g. CoreOS defaults to overlayfs.
- mariusmg 11y ago
  >Sometimes you have a workload that really isn't a fit for virtualization Yeah, any serious I/O load is unsuited for virtualization.
  - devonkim 11y ago
    A lot of other latency-sensitive applications tend to have so many adverse performance conditions (that can usually be remediated with a lot of blood sweat and tears) under virtualization that it becomes easier to just go bare metal and deal with physical infrastructure overhead.
mverwijs 11y ago
I'm surprised everyone is still installing to disk. In 2008/2009, we had a POC where we ran the OS from memory after booting over PXE. * boot a live image into memory * point LXD/RKT/Docker to /containers * ... * profit!
- iofj 11y ago
  The big issue I'm having with that is that it involves trusting vendors to get network boot right. Especially when it comes to the looping part of "loop until DHCP gets a response" it becomes a problem. One of the cheap vendor tries 30 times and then goes to a boot failed screen after trying the disk. Also, 1 time out of a 4-5000 or so network boot fails. Not sure why.
  - vidarh 11y ago
    If you have IPMI on the server this doesn't become such a big problem - you can reasonably trigger resets/reboots if it's not up after a given amount of time.
    - iofj 11y ago
      We buy the cheapest server that meets our needs, and buy it in somewhat larger quantities (often double what was originally envisioned for less than was originally budgeted). Much more efficient. But it does mean no IPMI. However I built a small circuit that sits on a power cable that can interrupt said power cable with a relais that sits on a bus plugged into our server, so we can do the reboot thing. I've been meaning to redo that power cable circuit using wifi as the linking technology, now that we have esp8266 available.
  - mverwijs 11y ago
    That's where iLO comes in. iLO is horrible, but you can ssh to it and set all manner of stuff. When we didn't have PXE, we had a script that told iLO to boot from CD, and that the CD was located at http://something/bootme.iso http://something/bootme.iso. iLO would always have network, and would pass the .iso magically to the server as device to boot from.
jon-wood 11y ago
I built something similar for a managed hosting provider ~10 years ago. That doesn't make it any less useful now, and this does a ton more than our tool did, far more elegantly. At some point someone needs to manage the actual hardware, whether that's you, or a middleman, and when you're handling hundreds or even thousands of devices its just not practical without automation.
jssjr 11y ago
GitHub's physical infrastructure team doesn't dictate what technologies our engineers can run on our hardware. We are interested in providing reliable server resources in a easily consumable manner. If someone wants to provision hardware to run docker containers or similar, that's great! We may eventually offer higher order infrastructure or platform services internally, but it's not our current focus.

by sargun 11y ago

packetized 11y ago

Wait, why? Are you advocating for the use of an abstraction layer where there isn't always a business case for using one?

detaro 11y ago

Just because you use Containers/VMs for most of your apps doesn't mean that the lower levels don't need attention: installing OSes in the first place, hardware testing (both initially and to identify defects later), ... And for important fileservers and databases you're going to run on specific hardware for a long time.

kbar13 11y ago

you realize the cloud runs on hardware right. machines don't magically come with everything installed and configured.

ownagefool 11y ago

Even if you were going to run CoreOS or Mesos on the machine, you'd still manage it booting your specific image, which you can change, rather than trusting the pre-installed dell verion and managing that relationship. Now there's probably some room for debate on whether these guys job should just be outsourced to Amazon, but github has some pretty good uptime and they seem to know what they're doing, thus they've probably already won that debate.

lwhalen 11y ago

Sometimes you have a workload that really isn't a fit for virtualization/containers/whatever the latest Rails hotness is, at all, and you just need to throw a couple of cargo trailers of insanely massively-spec'd servers at the problem. In those cases, your 'old school' server provisioning toolkit had better be on-point. It's easy to forget just how ridiculously powerful bare iron is these days. Go to Dell.com and see how much RAM you can cram into a U or three or four today in 2015. Or see how many IOPS a modern NetApp or Symmetrix (EMC) can push with 'flashcache' or million-dollar SSDs. It is ridiculous, and while a lot of those platforms are meant for 'building your own private cloud', etc, there's a non-trivial amount of workloads/projects where bare-iron is the best tool for the job.

icebraining 11y ago
Containers are just a namespacing tool, though; you're still running on bare metal (well, bare Linux). Docker in particular runs on AuFS, which is slow, but other containerization tools just use a chroot.
- vidarh 11y ago
  Docker can run on any number of things, include btrfs and overlayfs+ext4, as well as devicemapper. E.g. CoreOS defaults to overlayfs.
mariusmg 11y ago
>Sometimes you have a workload that really isn't a fit for virtualization Yeah, any serious I/O load is unsuited for virtualization.
- devonkim 11y ago
  A lot of other latency-sensitive applications tend to have so many adverse performance conditions (that can usually be remediated with a lot of blood sweat and tears) under virtualization that it becomes easier to just go bare metal and deal with physical infrastructure overhead.

mverwijs 11y ago

I'm surprised everyone is still installing to disk. In 2008/2009, we had a POC where we ran the OS from memory after booting over PXE. * boot a live image into memory * point LXD/RKT/Docker to /containers * ... * profit!

iofj 11y ago
The big issue I'm having with that is that it involves trusting vendors to get network boot right. Especially when it comes to the looping part of "loop until DHCP gets a response" it becomes a problem. One of the cheap vendor tries 30 times and then goes to a boot failed screen after trying the disk. Also, 1 time out of a 4-5000 or so network boot fails. Not sure why.
- vidarh 11y ago
  If you have IPMI on the server this doesn't become such a big problem - you can reasonably trigger resets/reboots if it's not up after a given amount of time.
  - iofj 11y ago
    We buy the cheapest server that meets our needs, and buy it in somewhat larger quantities (often double what was originally envisioned for less than was originally budgeted). Much more efficient. But it does mean no IPMI. However I built a small circuit that sits on a power cable that can interrupt said power cable with a relais that sits on a bus plugged into our server, so we can do the reboot thing. I've been meaning to redo that power cable circuit using wifi as the linking technology, now that we have esp8266 available.
- mverwijs 11y ago
  That's where iLO comes in. iLO is horrible, but you can ssh to it and set all manner of stuff. When we didn't have PXE, we had a script that told iLO to boot from CD, and that the CD was located at http://something/bootme.iso http://something/bootme.iso. iLO would always have network, and would pass the .iso magically to the server as device to boot from.

jon-wood 11y ago

I built something similar for a managed hosting provider ~10 years ago. That doesn't make it any less useful now, and this does a ton more than our tool did, far more elegantly. At some point someone needs to manage the actual hardware, whether that's you, or a middleman, and when you're handling hundreds or even thousands of devices its just not practical without automation.

jssjr 11y ago

GitHub's physical infrastructure team doesn't dictate what technologies our engineers can run on our hardware. We are interested in providing reliable server resources in a easily consumable manner. If someone wants to provision hardware to run docker containers or similar, that's great! We may eventually offer higher order infrastructure or platform services internally, but it's not our current focus.