19 ms·

""" One could imagine a very imperative “cluster operating system,” like the above, which exposed primitives like “allocate 5 CPUs worth of compute” or “create

""" One could imagine a very imperative “cluster operating system,” like the above, which exposed primitives like “allocate 5 CPUs worth of compute” or “create a new virtual network,” which in turn backed onto configuration changes either in the system’s internal abstractions or into calls into the EC2 API (or other underlying cloud provider). """ There is a name for this: Single System Image. There is no need to write the operating system layer - Linux works well. I built a SSI. Each grid can host several apps, 'gridapps'. As priorities change during the day, an operator can rebalance resources using a bash-like shell. Each gridapp has (1) core nodes, which need to keep running, and (2) scale nodes which can be started and stopped. Nodes can be configured so they are taskset-bound to an individual processor, or sharing a processor between several nodes. There are equivalent primitives for memory and persistent storage. Commercial, 1000 cores. The author contrasts this with K8s by saying that K8s is declarative. This is not the important distinction. My system had declarative configuration, and did a fitting at startup. The key difference is motive. The design above is the natural choice when you build a grid computer from first-principles. K8s is designed to solve a different problem: scaling up the traditional model of programming. Here, a developer writes an algorithm on their workstation. The deployment circumstances and interactions with other systems are an afterthought. Often this developer does not deploy the code themselves, this is done by another team who don't understand the internals of the code.

pjmlp 5y ago
We don't need Linux, a type 1 hypervisor for Cloud infrastructure is good enough.
- cturner 5y ago
  Question about drivers. If you had DMA-oriented/kernel-bypass network cards or wanted to resource-manage GPUs, would a type 1 hypervisor be able to support this?
  - pjmlp 5y ago
    I don't see why not, type 1 hypervisors are the modern version of what started on System/360, naturally depending on specific one the APIs for such stuff will be different.
carlmr 5y ago
>The author contrasts this with K8s by saying that K8s is declarative. This is not the important distinction. I feel like declarative is some kind of technical manual meme for the last few years. I don't know at this point how often I've read that something is better because it's modern and declarative. Why it's easy to understand because it's declarative, etc. There's a similar thing with "functional". In that case I at least understand the intuition why it's better, which is because if I/O behavior is constant and you don't have to think of an internal state, like in OOP. It is indeed much easier to divide and conquer when you can understand the parts of the whole individually. And as soon as you understand the parts you can piece them together. With "declarative" I still have to find out what's the intuition WHY it's supposedly better, until then it's kind of like the 90s, where the washing machine was "better" because it had "fuzzy logic".
- ghayes 5y ago
  Declarative here means you’re describing the end state and letting the system decide how to transition from the current state. For instance, you can declare that you need two X (e.g. containers, disks, …) and pass that configuration to the framework. If the framework sees you currently have 1 X in production, it allocates another. If there are three, the framework kills one. And if there are 2, it’s a no-op. The beauty is that if you express what you need declaratively, you’re shielded from deciding the state transition.
  - deleted 5y ago
    [deleted]
- maccard 5y ago
  > With "declarative" I still have to find out what's the intuition WHY it's supposedly better, Declarative is better because you write what you want to _have_ not what you want to change, which means you need implicit knowledge of the state of the system, and leave the complexities of figuring out the in between to the tool you're using.
  - jen20 5y ago
solatic 5y ago
> an operator can rebalance So, you're not talking about the same thing as OP, because one of the points being made here is that Kubernetes intentionally tries to get rid of human operators for day-to-day decision making. If you have a machine making operations decisions instead of humans, then a "bash-like shell" is worse than watch/subscribe mechanisms with full type safety, plus standard conventions for reporting events and communicating status.
- henrydark 5y ago
  If a human operator can make changes with a computer language, then a human programmer can make a computer operator that makes changes autonomously
  - robertlagrant 5y ago
    Theoretically yes (of course) but Bash can make it tricky.
    - cturner 5y ago
      The purpose of the shell is to allow users to interact with the grid. You can't do this in a conventional OS shell. Example: if you are using /bin/sh on linux-host-a and want to affect something running on linux-host-b, you can't. In a SSI shell, you can affect resources from a set of machines from a single shell. You could offer something like shell scripting on that, but my system did not and there are better options. For example: have the resource manager expose a command that offers a DSL for making system-wide changes.
      - hobo_mark 5y ago
        > a DSL for making system-wide changes Like... Kubernetes? :)
- cturner 5y ago
  The paper presented declarative configuration as a contrast between the SSI approach and the k8s approach. I don't think it is. It is useful to have both the fitted arrangements and operator interaction. In addition to giving operators power to make changes in production, the command line is useful for development. Devs can develop in a one-host grid and interact with it in a repl-like way. Contrast this to traditional workflow where the deployment environment is quite different to the development environment. This encourages devs to work with the grain of the platform and discourages the "it works on my machine" dynamic.
trhway 5y ago
>K8s is designed to solve a different problem: scaling up the traditional model of programming. Yep. K8s is a hell spawn of ITIL (a decades old IBM's take on BigCo/government IT infrastructure management). Google has been practicing ITIL, Borg comes out as a result, which takes K8s shape outside of Google. That is one of the main reasons why management likes K8s - ITIL was created by managers for managers. The approaches like SSI for example shift the balance toward developers/engineering and thus instinctively resisted by managers.
- rswail 5y ago
  ITIL configuration management has nothing to do with K8S cluster management. ITIL configuration management is process driven, whereas K8S is declarative. It's in the names of the processes, Incident Management, Problem Management, Change Management. K8S has controllers that are given a configuration and work out how to deploy that, taking into account current state. ie, it's functionally taking the state of the system now, S0 and applying functional modifications to return S1. The issue is that the modifications aren't "pure" in a functional sense, because there's a temporal aspect that means there can be difficulties in achieving an equilibrium to get to S1.
- menage 5y ago
  > Google has been practicing ITIL, Borg comes out as a result, Having been a member of the team that built Borg, I can assure you that those two things were unrelated. I can't say for sure whether the initial idea/impetus for Borg came from the management or the developer side, but once the project was started it was very much engineer-driven, and pretty organically so. The Infrastructure team at Google at the time (of which Borg was a part, along with several other related cloud projects) had a single manager for what grew to eventually 100+ engineers (mostly senior/staff level, and highly independent) before Google started creating/hiring managers more aggressively, so we were pretty much self-managing. The design and implementation of Borg was horizontal, both within the team, and coordinating with engineers (developers and operations teams) that would actually need to use the system. I'd say it was at least three years in before management really got involved in design decisions, by which time Borg was running most of Google's production services It was a bit of a Wild West, but compared to some later major from-scratch design/development projects I've been involved in since, it was a Belle Époque.
rr808 5y ago
Surely k8s benefits from being a standardized way to do this. Yes you can roll your own version but it'll be different in every company.
- deleted 5y ago
  [deleted]
zozbot234 5y ago
> There is a name for this: Single System Image. There is no need to write the operating system layer - Linux works well. Wouldn't a true SSI need to include some version of software-implemented distributed shared memory, which in turn would need to be achieved by involving the kernel? Linux now has process checkpoint and restore which will actually work seamlessly if all resources are properly namespaced ("containerized"), but not yet DSM, so far as I can tell. (The only alternative I can think of would involve hacking around the debug infrastructure to "attach" to other processes and somehow handle VM faults in userspace, which sounds positively terrible.)

pjmlp 5y ago

We don't need Linux, a type 1 hypervisor for Cloud infrastructure is good enough.

cturner 5y ago
Question about drivers. If you had DMA-oriented/kernel-bypass network cards or wanted to resource-manage GPUs, would a type 1 hypervisor be able to support this?
- pjmlp 5y ago
  I don't see why not, type 1 hypervisors are the modern version of what started on System/360, naturally depending on specific one the APIs for such stuff will be different.

carlmr 5y ago

>The author contrasts this with K8s by saying that K8s is declarative. This is not the important distinction. I feel like declarative is some kind of technical manual meme for the last few years. I don't know at this point how often I've read that something is better because it's modern and declarative. Why it's easy to understand because it's declarative, etc. There's a similar thing with "functional". In that case I at least understand the intuition why it's better, which is because if I/O behavior is constant and you don't have to think of an internal state, like in OOP. It is indeed much easier to divide and conquer when you can understand the parts of the whole individually. And as soon as you understand the parts you can piece them together. With "declarative" I still have to find out what's the intuition WHY it's supposedly better, until then it's kind of like the 90s, where the washing machine was "better" because it had "fuzzy logic".

ghayes 5y ago
Declarative here means you’re describing the end state and letting the system decide how to transition from the current state. For instance, you can declare that you need two X (e.g. containers, disks, …) and pass that configuration to the framework. If the framework sees you currently have 1 X in production, it allocates another. If there are three, the framework kills one. And if there are 2, it’s a no-op. The beauty is that if you express what you need declaratively, you’re shielded from deciding the state transition.
- deleted 5y ago
  [deleted]
maccard 5y ago
> With "declarative" I still have to find out what's the intuition WHY it's supposedly better, Declarative is better because you write what you want to _have_ not what you want to change, which means you need implicit knowledge of the state of the system, and leave the complexities of figuring out the in between to the tool you're using.
- jen20 5y ago

solatic 5y ago

> an operator can rebalance So, you're not talking about the same thing as OP, because one of the points being made here is that Kubernetes intentionally tries to get rid of human operators for day-to-day decision making. If you have a machine making operations decisions instead of humans, then a "bash-like shell" is worse than watch/subscribe mechanisms with full type safety, plus standard conventions for reporting events and communicating status.

henrydark 5y ago
If a human operator can make changes with a computer language, then a human programmer can make a computer operator that makes changes autonomously
- robertlagrant 5y ago
  Theoretically yes (of course) but Bash can make it tricky.
  - cturner 5y ago
    The purpose of the shell is to allow users to interact with the grid. You can't do this in a conventional OS shell. Example: if you are using /bin/sh on linux-host-a and want to affect something running on linux-host-b, you can't. In a SSI shell, you can affect resources from a set of machines from a single shell. You could offer something like shell scripting on that, but my system did not and there are better options. For example: have the resource manager expose a command that offers a DSL for making system-wide changes.
    - hobo_mark 5y ago
      > a DSL for making system-wide changes Like... Kubernetes? :)
cturner 5y ago
The paper presented declarative configuration as a contrast between the SSI approach and the k8s approach. I don't think it is. It is useful to have both the fitted arrangements and operator interaction. In addition to giving operators power to make changes in production, the command line is useful for development. Devs can develop in a one-host grid and interact with it in a repl-like way. Contrast this to traditional workflow where the deployment environment is quite different to the development environment. This encourages devs to work with the grain of the platform and discourages the "it works on my machine" dynamic.

trhway 5y ago

>K8s is designed to solve a different problem: scaling up the traditional model of programming. Yep. K8s is a hell spawn of ITIL (a decades old IBM's take on BigCo/government IT infrastructure management). Google has been practicing ITIL, Borg comes out as a result, which takes K8s shape outside of Google. That is one of the main reasons why management likes K8s - ITIL was created by managers for managers. The approaches like SSI for example shift the balance toward developers/engineering and thus instinctively resisted by managers.

rswail 5y ago
ITIL configuration management has nothing to do with K8S cluster management. ITIL configuration management is process driven, whereas K8S is declarative. It's in the names of the processes, Incident Management, Problem Management, Change Management. K8S has controllers that are given a configuration and work out how to deploy that, taking into account current state. ie, it's functionally taking the state of the system now, S0 and applying functional modifications to return S1. The issue is that the modifications aren't "pure" in a functional sense, because there's a temporal aspect that means there can be difficulties in achieving an equilibrium to get to S1.
menage 5y ago
> Google has been practicing ITIL, Borg comes out as a result, Having been a member of the team that built Borg, I can assure you that those two things were unrelated. I can't say for sure whether the initial idea/impetus for Borg came from the management or the developer side, but once the project was started it was very much engineer-driven, and pretty organically so. The Infrastructure team at Google at the time (of which Borg was a part, along with several other related cloud projects) had a single manager for what grew to eventually 100+ engineers (mostly senior/staff level, and highly independent) before Google started creating/hiring managers more aggressively, so we were pretty much self-managing. The design and implementation of Borg was horizontal, both within the team, and coordinating with engineers (developers and operations teams) that would actually need to use the system. I'd say it was at least three years in before management really got involved in design decisions, by which time Borg was running most of Google's production services It was a bit of a Wild West, but compared to some later major from-scratch design/development projects I've been involved in since, it was a Belle Époque.

rr808 5y ago

Surely k8s benefits from being a standardized way to do this. Yes you can roll your own version but it'll be different in every company.

deleted 5y ago
[deleted]

zozbot234 5y ago

> There is a name for this: Single System Image. There is no need to write the operating system layer - Linux works well. Wouldn't a true SSI need to include some version of software-implemented distributed shared memory, which in turn would need to be achieved by involving the kernel? Linux now has process checkpoint and restore which will actually work seamlessly if all resources are properly namespaced ("containerized"), but not yet DSM, so far as I can tell. (The only alternative I can think of would involve hacking around the debug infrastructure to "attach" to other processes and somehow handle VM faults in userspace, which sounds positively terrible.)