5 ms·

If you don't use k8s and just run bespoke containers you still have to figure out how those containers find and talk to each other. Maybe you run some custom D

If you don't use k8s and just run bespoke containers you still have to figure out how those containers find and talk to each other. Maybe you run some custom DNS setup, maybe you run a purpose built service discovery thing like consul, etc. And you have to figure out how you'll do networking to support public/internet-facing workloads vs. private internal services (and how each can talk to the other). But... if you just use k8s you get things like basic service discovery, networking, ingress, etc. with it and don't have to figure out bespoke solutions (that you'll just chuck anyways once you move to k8s). I do agree though I would be very hesitant to immediately dive in running stateful workloads like databases, etc. on my own k8s cluster. The cloud hosted database services that every provider has is such a significant time and complexity saver, especially if your databases are just getting started and small-ish.

grogenaut 4y ago
Discovery can be quite simple, we've been doing it for years. On ECS/AWS we do it with just ALB and the cluster. Turns out DNS works really well, surprising for a fundamental internet technology. Tho if it's broken it's likely DNS that is the culprit. If I need to route to other things I just configure other dns entries with SSM Parameter store... or I just manipulate the local resolvers in the VPC. Very straightforward. But I'm generally running just my software, not other people's, and consuming aws datastores. This is for work.
- deleted 4y ago
  [deleted]
- staticassertion 4y ago
  Does k8s not just use dns? That's what Hashicorp Nomad uses - DNS via Consul.
  - qbasic_forever 4y ago
    It does, every pod, service, etc. in the cluster has a fully qualified DNS name resolved by a DNS server in the cluster.
  - KronisLV 4y ago
    Yep, K8s has managed DNS, much like Nomad allows you to have. But if that's a feature that you want, you can even use Docker Swarm which I mentioned in another comment here, which basically gives you the same thing, so that you can reference other containers by their names. Of course, it's a bit simpler than Nomad, so if you're already using Nomad then there's no point in moving away from it, though this still illustrates that there are plenty of simpler options than K8s!
  - grogenaut 4y ago
    My point is that people come up with very powerful and complex routing systems using kube and conflate it with with "discovery". We did this at my work. Consul was the "only solution" that would "work" for us (pre kube). Consul is a type of discovery, but people also lop in service mesh features which are well beyond "what IP address do I need to send packets at". 4 years later consul is gone in aws for us replaced by this simple dns/alb scheme. We haven't really needed all the features of advanced discovery. The key in general to "discovery" seems to be load balancers and dns in our approach (service protect thyself vs well behaved clients). Easy to use LBs on aws is very powerful feature. We don't have that on our on prem and it changes how we approach things. There discovery is harder.
- vinnymac 4y ago
  I am doing the same at my current company. We have a small team of 10 engineers, and DNS for discovery on AWS has been enough for us for the last 5 years. There hasn’t been a single issue for us in that time.
8n4vidtmkvmk 4y ago
I'm running a stateful db... it's like $5 to run myself vs $120/mo to have it managed. big difference. haven't had to touch it in years now but im also afraid to try upgrading it.
- 8organicbits 4y ago
  If you're afraid to upgrade yourself, you may eventually contract out for help. You'd need that contract to cost less than $1380 for each year of upgrades. Otherwise, you should have picked the managed solution. Doing it yourself is probably only cheaper if you never need maintenance or upgrades, security patching isn't needed, or your time is cheap.
- danjac 4y ago
  I would argue that if your spend is $5 a month your data is probably not very valuable to yourself or your customers. Which is totally OK for some projects, but $120 is not very much for that peace of mind and time-saving if your data is important to your company.
  - dgb23 4y ago
    It depends more on the volume, the frequency of changes and the complexity of implementation. You can have a robust solution for a few bucks infra cost, if those parameters are small.
    - danjac 4y ago
      OK, but how much time are you spending on managing these yourself? If it's non-zero then the cost between $5 and $120 for a company is trivial. For a personal side project, sure, but if we are talking about paying customers' data then even things like automated backup and recovery need to be taken into account.
      - dgb23 4y ago
        For backups I agree, they obviously should be automated, but recovery? Not in my experience, not at the scale we're talking about here. You want something that is simple and semi-automated. Backup recovery in a small, simple, robust system is very rare. Automated recovery is complex and there are quite a bunch of gotchas and perhaps project specific things to take into account. But automated backups are typically straightforward. You need to know when and how to do them and where to put them. I think there are tons of reasons to pay for an managed database if you need it. But it is not a baseline requirement for many projects.
closeparen 4y ago
If you have a complicated microservices topology, then yes you're going to need heavyweight infrastructure tools to manage the cross-service traffic, isolate and bin-pack heterogeneous workloads, etc. You probably don't need to have a complicated microservices topology though. A load balancer->stateless app tier->managed RDBMS architecture should be more than enough until you're at a point where you can afford to scale a serious infrastructure team.
slotrans 4y ago
Surely you recognize that we had services that could talk to each other before k8s existed? And therefore all of the things you point to are solved problems and don't require k8s in any way, shape, or form? Like, if I'm configuring the foo service and it needs to know how to contact the bar service... that's not a problem? I just put bar-service-lb.mycompany.net in a config file and I'm done. I don't understand why people think they need complex "service discovery". Just give things names! My cloud provider (or heaven forbid, wires in a data center! gasp!) already gives me perfectly good networking. Why layer anything on top of that? No one really needs "ingress". Your server program can listen on a port! Like we've been doing for 50 years! This is a completely self-inflicted problem. It's so depressing seeing the k8s generation completely forget the highly effective ways that we used to do things. All those things are still possible! You can choose to design, build, and run a simple system if you want to!
- KronisLV 4y ago
  > Like, if I'm configuring the foo service and it needs to know how to contact the bar service... that's not a problem? I just put bar-service-lb.mycompany.net in a config file and I'm done. I don't understand why people think they need complex "service discovery". Just give things names! What about when you're given a few servers for your dev/test environment, but don't have control over the DNS server and adding records to it has an annoying amount of red tape? Would this be a good option: - FOO_URL=http://300.0.0.1:3000 - BAR_URL=http://300.0.0.1:8080 - BAZ_URL=http://300.0.0.2:80 - X_URL=http://300.0.0.2:5601 - Y_URL=http://300.0.0.3:5602 Or would this be better: - FOO_URL=http://foo.dev.svc.cluster.local - BAR_URL=http://bar.dev.svc.cluster.local - BAZ_URL=http://baz.dev.svc.cluster.local - X_URL=http://x.dev.svc.cluster.local - Y_URL=http://y.dev.svc.cluster.local Sure, you can achieve the same with just modifying a hosts file, but what about when you need to propagate some changes to every computer and other server that needs to talk to these apps, while you are still not in control of the DNS server? I guess you can just run your own DNS server at that point as well, but that might raise some eyebrows in some places, vs just using whatever the orchestrator provides you with out of the box (and Kubernetes also has built in proxy functionality so people can connect to the cluster pretty easily and forward whichever services they want, which can take a bit of work otherwise, though i guess you'd get used to the proper order of arguments in the SSH tunnel commands too). > No one really needs "ingress". Your server program can listen on a port! Like we've been doing for 50 years! This is a completely self-inflicted problem. What if you want all of your services to have SSL/TLS certificates provisioned for them? What if you want to ensure that all of the inter-service communication is also encrypted? What about you wanting to add circuit breaking or rate limiting logic in there? Or maybe doing a MitM on yourself so you can debug the network traffic between the services better? Also, do you really want to configure the firewall rules for each of your internal services to be able to talk to others, or wouldn't it also be good to have it piggyback its traffic on some other overlay networking port that's open on all of them for cluster communications? Oh, also to avoid weirdness with CORS or other technologies, it might be nice to route requests based on domains and paths, instead of ports (e.g. my-app.com/analytics might be better than my-app.com:9000 in some contexts), though of course that is easily achievable with any reverse proxy, just saying that ports aren't always the best option. Personally, I think that there's a lot of benefit to using an ingress and a service mesh of some sort, as long as those don't add too much undue complexity. I mean, using our own Nginx/Apache/Caddy/Traefik instance instead of the Ingress abstraction was also perfectly passable, but there are certainly benefits to exploring new technologies and what they offer. In my experience, all of these container technologies providing you with their own DNS implementation out of the box is very useful, especially because recognizing that "300.0.0.1:3000" is the wrong URL is harder than looking at "foo.dev.svc.cluster.local" and seeing that your service probably doesn't intend to connect to foo. > It's so depressing seeing the k8s generation completely forget the highly effective ways that we used to do things. All those things are still possible! You can choose to design, build, and run a simple system if you want to! I do agree with this, though! There should be more resources out there to show off how to do basic and useful networking without getting too deeply into some of the fancier technologies.

grogenaut 4y ago

Discovery can be quite simple, we've been doing it for years. On ECS/AWS we do it with just ALB and the cluster. Turns out DNS works really well, surprising for a fundamental internet technology. Tho if it's broken it's likely DNS that is the culprit. If I need to route to other things I just configure other dns entries with SSM Parameter store... or I just manipulate the local resolvers in the VPC. Very straightforward. But I'm generally running just my software, not other people's, and consuming aws datastores. This is for work.

deleted 4y ago
[deleted]
staticassertion 4y ago
Does k8s not just use dns? That's what Hashicorp Nomad uses - DNS via Consul.
- qbasic_forever 4y ago
  It does, every pod, service, etc. in the cluster has a fully qualified DNS name resolved by a DNS server in the cluster.
- KronisLV 4y ago
  Yep, K8s has managed DNS, much like Nomad allows you to have. But if that's a feature that you want, you can even use Docker Swarm which I mentioned in another comment here, which basically gives you the same thing, so that you can reference other containers by their names. Of course, it's a bit simpler than Nomad, so if you're already using Nomad then there's no point in moving away from it, though this still illustrates that there are plenty of simpler options than K8s!
- grogenaut 4y ago
  My point is that people come up with very powerful and complex routing systems using kube and conflate it with with "discovery". We did this at my work. Consul was the "only solution" that would "work" for us (pre kube). Consul is a type of discovery, but people also lop in service mesh features which are well beyond "what IP address do I need to send packets at". 4 years later consul is gone in aws for us replaced by this simple dns/alb scheme. We haven't really needed all the features of advanced discovery. The key in general to "discovery" seems to be load balancers and dns in our approach (service protect thyself vs well behaved clients). Easy to use LBs on aws is very powerful feature. We don't have that on our on prem and it changes how we approach things. There discovery is harder.
vinnymac 4y ago
I am doing the same at my current company. We have a small team of 10 engineers, and DNS for discovery on AWS has been enough for us for the last 5 years. There hasn’t been a single issue for us in that time.

8n4vidtmkvmk 4y ago

I'm running a stateful db... it's like $5 to run myself vs $120/mo to have it managed. big difference. haven't had to touch it in years now but im also afraid to try upgrading it.

8organicbits 4y ago
If you're afraid to upgrade yourself, you may eventually contract out for help. You'd need that contract to cost less than $1380 for each year of upgrades. Otherwise, you should have picked the managed solution. Doing it yourself is probably only cheaper if you never need maintenance or upgrades, security patching isn't needed, or your time is cheap.
danjac 4y ago
I would argue that if your spend is $5 a month your data is probably not very valuable to yourself or your customers. Which is totally OK for some projects, but $120 is not very much for that peace of mind and time-saving if your data is important to your company.
- dgb23 4y ago
  It depends more on the volume, the frequency of changes and the complexity of implementation. You can have a robust solution for a few bucks infra cost, if those parameters are small.
  - danjac 4y ago
    OK, but how much time are you spending on managing these yourself? If it's non-zero then the cost between $5 and $120 for a company is trivial. For a personal side project, sure, but if we are talking about paying customers' data then even things like automated backup and recovery need to be taken into account.
    - dgb23 4y ago
      For backups I agree, they obviously should be automated, but recovery? Not in my experience, not at the scale we're talking about here. You want something that is simple and semi-automated. Backup recovery in a small, simple, robust system is very rare. Automated recovery is complex and there are quite a bunch of gotchas and perhaps project specific things to take into account. But automated backups are typically straightforward. You need to know when and how to do them and where to put them. I think there are tons of reasons to pay for an managed database if you need it. But it is not a baseline requirement for many projects.

closeparen 4y ago

If you have a complicated microservices topology, then yes you're going to need heavyweight infrastructure tools to manage the cross-service traffic, isolate and bin-pack heterogeneous workloads, etc. You probably don't need to have a complicated microservices topology though. A load balancer->stateless app tier->managed RDBMS architecture should be more than enough until you're at a point where you can afford to scale a serious infrastructure team.

slotrans 4y ago

Surely you recognize that we had services that could talk to each other before k8s existed? And therefore all of the things you point to are solved problems and don't require k8s in any way, shape, or form? Like, if I'm configuring the foo service and it needs to know how to contact the bar service... that's not a problem? I just put bar-service-lb.mycompany.net in a config file and I'm done. I don't understand why people think they need complex "service discovery". Just give things names! My cloud provider (or heaven forbid, wires in a data center! gasp!) already gives me perfectly good networking. Why layer anything on top of that? No one really needs "ingress". Your server program can listen on a port! Like we've been doing for 50 years! This is a completely self-inflicted problem. It's so depressing seeing the k8s generation completely forget the highly effective ways that we used to do things. All those things are still possible! You can choose to design, build, and run a simple system if you want to!

KronisLV 4y ago
> Like, if I'm configuring the foo service and it needs to know how to contact the bar service... that's not a problem? I just put bar-service-lb.mycompany.net in a config file and I'm done. I don't understand why people think they need complex "service discovery". Just give things names! What about when you're given a few servers for your dev/test environment, but don't have control over the DNS server and adding records to it has an annoying amount of red tape? Would this be a good option: - FOO_URL=http://300.0.0.1:3000 - BAR_URL=http://300.0.0.1:8080 - BAZ_URL=http://300.0.0.2:80 - X_URL=http://300.0.0.2:5601 - Y_URL=http://300.0.0.3:5602 Or would this be better: - FOO_URL=http://foo.dev.svc.cluster.local - BAR_URL=http://bar.dev.svc.cluster.local - BAZ_URL=http://baz.dev.svc.cluster.local - X_URL=http://x.dev.svc.cluster.local - Y_URL=http://y.dev.svc.cluster.local Sure, you can achieve the same with just modifying a hosts file, but what about when you need to propagate some changes to every computer and other server that needs to talk to these apps, while you are still not in control of the DNS server? I guess you can just run your own DNS server at that point as well, but that might raise some eyebrows in some places, vs just using whatever the orchestrator provides you with out of the box (and Kubernetes also has built in proxy functionality so people can connect to the cluster pretty easily and forward whichever services they want, which can take a bit of work otherwise, though i guess you'd get used to the proper order of arguments in the SSH tunnel commands too). > No one really needs "ingress". Your server program can listen on a port! Like we've been doing for 50 years! This is a completely self-inflicted problem. What if you want all of your services to have SSL/TLS certificates provisioned for them? What if you want to ensure that all of the inter-service communication is also encrypted? What about you wanting to add circuit breaking or rate limiting logic in there? Or maybe doing a MitM on yourself so you can debug the network traffic between the services better? Also, do you really want to configure the firewall rules for each of your internal services to be able to talk to others, or wouldn't it also be good to have it piggyback its traffic on some other overlay networking port that's open on all of them for cluster communications? Oh, also to avoid weirdness with CORS or other technologies, it might be nice to route requests based on domains and paths, instead of ports (e.g. my-app.com/analytics might be better than my-app.com:9000 in some contexts), though of course that is easily achievable with any reverse proxy, just saying that ports aren't always the best option. Personally, I think that there's a lot of benefit to using an ingress and a service mesh of some sort, as long as those don't add too much undue complexity. I mean, using our own Nginx/Apache/Caddy/Traefik instance instead of the Ingress abstraction was also perfectly passable, but there are certainly benefits to exploring new technologies and what they offer. In my experience, all of these container technologies providing you with their own DNS implementation out of the box is very useful, especially because recognizing that "300.0.0.1:3000" is the wrong URL is harder than looking at "foo.dev.svc.cluster.local" and seeing that your service probably doesn't intend to connect to foo. > It's so depressing seeing the k8s generation completely forget the highly effective ways that we used to do things. All those things are still possible! You can choose to design, build, and run a simple system if you want to! I do agree with this, though! There should be more resources out there to show off how to do basic and useful networking without getting too deeply into some of the fancier technologies.