4 ms·

I'm involved in a project that uses Orleans. Conceptually and technically it's very interesting. Nodes automatically cluster together to form a big distributed

I'm involved in a project that uses Orleans. Conceptually and technically it's very interesting. Nodes automatically cluster together to form a big distributed system, and the concept of "virtual actors" make it possible to represent individual entities in the data model directly as actors, without worrying whether everything fits in memory at the same time. However, Orleans doesn't seem to play very well with Kubernetes. Kubernetes pods can be uncleanly terminated at any time for any reason. Orleans really doesn't like unclean shutdowns: it will leave behind dead entries in the membership tables, which then cause problems in the future because new nodes would contact dead members and run into errors. The clustering protocol also seems to assume that all members have stable identities and IP addresses, but that's obviously not the case in Kubernetes, where each new pod has a new identity and new IP address. This can cause members to fail uncleanly, which in turn pushes the clustering protocol into an infinite loop. Orleans was introduced by one person who has a background in distributed computing. Unfortunately, he's the only one in the organization with such a background. After he left, nobody could debug clustering problems.

reubenbond 6y ago
There is a new package which adds improved Kubernetes integration and fixes the issues you mentioned specifically. It's in beta right now, but we expect a stable release soon. Feel free to contact me and I can help you with running Orleans on Kubernetes
- bobuk 6y ago
  I'm interested, is there any urls to read about this package?
  - greyhoundracing 6y ago
    You can read more about it here: https://github.com/dotnet/orleans/pull/6707 https://github.com/dotnet/orleans/pull/6707
- deleted 6y ago
  [deleted]
smartbit 6y ago
Thanks for this insight. Skimming through the documentation [0], I find no ‘issues’ section that mentions this. IMHO Microsoft behaves untrustful by hiding an issue like this. > Early in the start-up process and at runtime, the silo will probe Kubernetes to find which silos do not have corresponding pods and mark those silos as dead [1] This is an experimental fix in a MR [1] that comes 6 years after Microsoft announced support for Kubernetes [2]. > Orleans was created at Microsoft Research and designed for use in the cloud. [0] I read: MS research developed the concepts of .NET Orleans 10 years ago without having Orchestrators like Kubernetes in mind. Now as an afterthought they have to come up with some patches for it not to cripple on short-lived Kubernetes nodes. A proper Operator & Helm chart was requested in nov 2017 [3] and the issue still is open. I get the impression that it is either not possible to write a proper Kubernetes Operator without a major rewrite of .NET Orleans or Microsoft has a commercial interest for .NET Orleans not being a first class citizen on Kubernetes. Whichever it is, I’m not going near a Kubernetes cluster hosting .NET Orleans. [0] http://dotnet.github.io/orleans/Documentation/index.html http://dotnet.github.io/orleans/Documentation/index.html [1] https://github.com/dotnet/orleans/pull/6707 https://github.com/dotnet/orleans/pull/6707 [2] https://azure.microsoft.com/en-us/blog/azure-collaboration-with-google-and-docker/ https://azure.microsoft.com/en-us/blog/azure-collaboration-w... [3] https://github.com/dotnet/orleans/issues/3692 https://github.com/dotnet/orleans/issues/3692
- reubenbond 6y ago
  Interesting take. We run Orleans on Kubernetes in production at Microsoft on multiple services. Other services run Orleans on Service Fabric in production - SF is similar to Kubernetes in terms of lifecycle. The project is open source, so things like Helm charts and k8s operators would be welcome contributions. If you want to discuss, feel free to message me on Twitter (https://twitter.com/reubenbond https://twitter.com/reubenbond) or Gitter (https://gitter.im/dotnet/orleans https://gitter.im/dotnet/orleans).
  - smartbit 6y ago
    Please feel free to explain here in HN - why it took up to 6 years for MS to release an experimental fix [1] to run on Kubernetes? - where I can find the outstanding issue in the release notes that [1] tries to fix? - why Microsoft doesn't write and release an Operator & Helm chart, instead asking the community? - if .NET Orleans runs fine on Kubernetes, why is there a need for an experimental fix?
csyszf 6y ago
> Kubernetes pods can be uncleanly terminated at any time for any reason Not true, pods can be gracefully shutdown, see https://cloudblog.withgoogle.com/products/gcp/kubernetes-best-practices-terminating-with-grace/amp/ https://cloudblog.withgoogle.com/products/gcp/kubernetes-bes... > The clustering protocol also seems to assume that all members have stable identities and IP addresses, but that's obviously not the case in Kubernetes, where each new pod has a new identity and new IP address. Both not true: Orleans cluster managed by a dynamic membership table which could be implemented by various tools(azure table, zookeeper, Kubernetes crds(etcd internally), etc), so nodes don't need static ip addresses also, in Kubernetes, you can deploy application as statefulset, by this way, pods will have stable identities > Orleans was introduced by one person who has a background in distributed computing. Unfortunately, he's the only one in the organization with such a background. After he left, nobody could debug clustering problems. Need source and proof for this one Please don't post ridiculous opinions about things you are not familiar with, both Kubernetes and Orleans. I have built successful products by Orleans & Kubernetes since two years ago, and now Orleans get better integration for Kubernetes than that time edit: formatting & typo

reubenbond 6y ago

There is a new package which adds improved Kubernetes integration and fixes the issues you mentioned specifically. It's in beta right now, but we expect a stable release soon. Feel free to contact me and I can help you with running Orleans on Kubernetes

bobuk 6y ago
I'm interested, is there any urls to read about this package?
- greyhoundracing 6y ago
  You can read more about it here: https://github.com/dotnet/orleans/pull/6707 https://github.com/dotnet/orleans/pull/6707
deleted 6y ago
[deleted]

smartbit 6y ago

Thanks for this insight. Skimming through the documentation [0], I find no ‘issues’ section that mentions this. IMHO Microsoft behaves untrustful by hiding an issue like this. > Early in the start-up process and at runtime, the silo will probe Kubernetes to find which silos do not have corresponding pods and mark those silos as dead [1] This is an experimental fix in a MR [1] that comes 6 years after Microsoft announced support for Kubernetes [2]. > Orleans was created at Microsoft Research and designed for use in the cloud. [0] I read: MS research developed the concepts of .NET Orleans 10 years ago without having Orchestrators like Kubernetes in mind. Now as an afterthought they have to come up with some patches for it not to cripple on short-lived Kubernetes nodes. A proper Operator & Helm chart was requested in nov 2017 [3] and the issue still is open. I get the impression that it is either not possible to write a proper Kubernetes Operator without a major rewrite of .NET Orleans or Microsoft has a commercial interest for .NET Orleans not being a first class citizen on Kubernetes. Whichever it is, I’m not going near a Kubernetes cluster hosting .NET Orleans. [0] http://dotnet.github.io/orleans/Documentation/index.html http://dotnet.github.io/orleans/Documentation/index.html [1] https://github.com/dotnet/orleans/pull/6707 https://github.com/dotnet/orleans/pull/6707 [2] https://azure.microsoft.com/en-us/blog/azure-collaboration-with-google-and-docker/ https://azure.microsoft.com/en-us/blog/azure-collaboration-w... [3] https://github.com/dotnet/orleans/issues/3692 https://github.com/dotnet/orleans/issues/3692

reubenbond 6y ago
Interesting take. We run Orleans on Kubernetes in production at Microsoft on multiple services. Other services run Orleans on Service Fabric in production - SF is similar to Kubernetes in terms of lifecycle. The project is open source, so things like Helm charts and k8s operators would be welcome contributions. If you want to discuss, feel free to message me on Twitter (https://twitter.com/reubenbond https://twitter.com/reubenbond) or Gitter (https://gitter.im/dotnet/orleans https://gitter.im/dotnet/orleans).
- smartbit 6y ago
  Please feel free to explain here in HN - why it took up to 6 years for MS to release an experimental fix [1] to run on Kubernetes? - where I can find the outstanding issue in the release notes that [1] tries to fix? - why Microsoft doesn't write and release an Operator & Helm chart, instead asking the community? - if .NET Orleans runs fine on Kubernetes, why is there a need for an experimental fix?

csyszf 6y ago

> Kubernetes pods can be uncleanly terminated at any time for any reason Not true, pods can be gracefully shutdown, see https://cloudblog.withgoogle.com/products/gcp/kubernetes-best-practices-terminating-with-grace/amp/ https://cloudblog.withgoogle.com/products/gcp/kubernetes-bes... > The clustering protocol also seems to assume that all members have stable identities and IP addresses, but that's obviously not the case in Kubernetes, where each new pod has a new identity and new IP address. Both not true: Orleans cluster managed by a dynamic membership table which could be implemented by various tools(azure table, zookeeper, Kubernetes crds(etcd internally), etc), so nodes don't need static ip addresses also, in Kubernetes, you can deploy application as statefulset, by this way, pods will have stable identities > Orleans was introduced by one person who has a background in distributed computing. Unfortunately, he's the only one in the organization with such a background. After he left, nobody could debug clustering problems. Need source and proof for this one Please don't post ridiculous opinions about things you are not familiar with, both Kubernetes and Orleans. I have built successful products by Orleans & Kubernetes since two years ago, and now Orleans get better integration for Kubernetes than that time edit: formatting & typo