6 ms·

Kubernetes is a solution worse than the problem. Microservices are unix pipes but slower. AI is linear algebra that costs x1000 because of the brand name. Th

Kubernetes is a solution worse than the problem. Microservices are unix pipes but slower. AI is linear algebra that costs x1000 because of the brand name. There is no revolution in software. We are just building crud as high and as fast as we can because the quality of developers tends to zero as the number of developers increases.

crtlaltdel 7y ago
^the account name that posted this comment is the single most honest thing i have seen on the internet today. well done!
abhs 7y ago
I like that username... :D
EdwardDiego 7y ago
> Kubernetes is a solution worse than the problem. I disagree. Maybe it's because we haven't gone "full Kubernetes" where every cronjob is now a Kubernetes cronjob, or what have you, but for all the complexity we've had to overcome, we've seen substantial benefits. For example, autoscaling instances of my data pipeline apps based on Kafka lag. Was it complex? Yes. We had to expose Kafka topic lag as a metric to Prometheus, then configure the k8s metrics server that HPAs (Horizontal Pod Autoscalers) use to scale to pull that in from Prometheus using k8s-prometheus-adapter as a custom metric, then set reasonable scaling limits in the data pipeline apps HPAs. Was it worth it? Fuck yes. We no longer have to worry about data arriving out of time at our data warehouse, because the scaling we've configured (in YAML, god rest its dirty soul) ensures that our data will always arrive at the data warehouse within 5 minutes, which is ideal for a near real time reporting product that greatly enables our end users. Is Kubernetes worth it? For us, definitely. For anyone else? Really depends on your use case, don't cargo cult this shit.
- pnako 7y ago
  How much data do your "data pipeline apps" process with Kafka, Kubernetes and probably other K-named things?
  - EdwardDiego 7y ago
    Several terabytes a day, but it varies, hence why I love the auto-scaling.
    - throwawaybbb 7y ago
      That's 10tb = 10000gb = 10000/24/60/60 = .11gb/s. My 2015 desktop could handle that. Where do you work? I'd like to pitch my radical idea of edge consolidated cloud computing.
      - fulafel 7y ago
        "Big data" frameworks are very good at throwing a lot of hardware at problems. See eg the classic big data vs laptop treatment: http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
      - EdwardDiego 7y ago
        The inevitable comment lol. I'm sure it could. To clarify, when the app scales, it's pods (container instances) that are scaled, not necessarily machines,that is managed separately by k8s. Depending on how much of the cluster is currently being used. Yep, we could handle it easily on one machine, if we gave it unfettered access to all the resource, but our throughput varies during the time of day, so dedicating a machine to it to handle the peak throughput wouldn't make financial sense at 3am at night. So it's far simpler to scale pods with known resource usage as needed. It also helps prevent issues when throughput suddenly doubles because of a business decision that you're left out of the loop on. So autoscaling pods is ideal for our use case.
        throwawaybbb 7y ago
- LaserToy 7y ago
  You don’t need kubernetes to do it, any cloud provider has api that will make it possible. Kube main point is declarative model, which comes with a big price (like sql had). Yaml - is error prone way to achieve anything. Will see if history repeats itself and we will see be imperative systems.
echelon 7y ago
> Kubernetes is a solution worse than the problem. Not if you're a 1,000+ engineer org with an SOA. > Microservices are unix pipes but slower. Humans are way more expensive than the machines running our code. You can always spend more on spinning up additional instances. In reality the overhead isn't that bad. The principles of immutable stateless deploys free up so much mental energy when it comes to thinking about how to get your code out and not break things that it more than pays for itself. > AI is linear algebra that costs x1000 because of the brand name. Tell that to my real time voice conversion system I just wrote. There's a lot of new magic happening right now. > There is no revolution in software. I'm on a fucking phone responding from hundreds of miles away. We're launching satellites that beam internet. > We are just building crud You can't speak for everyone here. Are you in adtech or a social spyware company? Maybe don't work like that then. Find meaningful work that speaks to you. > quality of developers tends to zero as the number of developers increases. I work with brilliant people.
- buzzkillington 7y ago
  >Not if you're a 1,000+ engineer org with an SOA. That's the problem. With a sane simple architecture instead of 40 years of crap on top of crap you wouldn't need thousands of engineers to spin up instances. >In reality the overhead isn't that bad. Over 1e2 times if the solution is in software, over 1e5 times if it's an fpga. Around 1e8 if you do it in silicon. You make the savings from not hiring the 1000s of engineers to deploy your bloated horror. >The principles of immutable stateless deploys Has nothing to do with k8s. Nixos and Guix are two systems that are as close to functional as possible and they sure as hell aren't 'stateless', even though they come closer than any of the sexy trending tech. >Tell that to my real time voice conversion system I just wrote. Thank you project librevox for proving a near infinite training corpus, thank you nvidia for gpus to do linear algebra 1e5 more efficiently than you can on a cpu. >I'm on a fucking phone responding from hundreds of miles away. We're launching satellites that beam internet. Like everything else you listed there this is a hardware improvement. A Casio watch from the 90s had the same order of magnitude computational capability as the first generation IMPs. Good luck getting it connected to the internet using any software you could write for it.
ummonk 7y ago
>AI is linear algebra that costs x1000 because of the brand name. Sure, but we've learnt to apply linear algebra to achieving things we could only dream of 10 (or even 5) years ago.
- sheeshkebab 7y ago
  Current AI is still mostly trash of the high school level project quality, linear algebra taken to the absolute possible level of insanity.
  - adrianN 7y ago
    I'm pretty sure that no high school project can beat the world's best Go player.
deleted 7y ago
[deleted]
Animats 7y ago
Well, arguably that's true in the webcrap space. The HTML->CSS->JS->Webasm line of development really is crap on top of crap. Trying to fix the design problems of one layer by putting another layer on top of it only sort of works. It gets worse over time. "The mold seeps through the wallpaper" (a comment I usually make about C++ collection classes trying, but only partially succeeding, to hide the problems of C arrays). I was doing "microservices" on QNX using MsgSend/MsgReceive for hard real time over 15 years ago. It doesn't have to suck; the mainstream is just using the wrong tools for doing it fast. The QNX people managed to shoot themselves in the foot by going closed-source -> open source -> closed source -> open source -> closed source, after which their developer community was fed up. Machine learning is not linear algebra. It's nonlinear. That's what makes it useful.
- ratww 7y ago
  > The HTML->CSS->JS->Webasm line of development really is crap on top of crap. I completely get the hate for modern web, but in this case I think it's unrelated. The whole Kubernetes/microservice world exists basically to serve JSON and similar protocols. Maybe rarely some static HTML. For HTML->CSS->JS->Webasm stuff it is more fashionable to compile at build time and service as static assets. That's the polar opposite of Kubernetes/microservices.
- cerberusss 7y ago
  > Well, arguably that's true in the webcrap space. (...) crap on top of crap Not that you're wrong per say, but personally, I think you should take a look at the HN guidelines. Like don't be snarky, and don't post shallow dismissals. Just categorizing the arguably most popular way of building software as "crap" doesn't seem like thoughtful or curious conversation.
smallcharleston 7y ago
“ AI is linear algebra that costs x1000 because of the brand name.” This doesn’t even make sense because it can be said about anything involving numerical work on a computer and it’s still roughly true. Also because all of the libraries are free anyway.

crtlaltdel 7y ago

^the account name that posted this comment is the single most honest thing i have seen on the internet today. well done!

abhs 7y ago

I like that username... :D

EdwardDiego 7y ago

> Kubernetes is a solution worse than the problem. I disagree. Maybe it's because we haven't gone "full Kubernetes" where every cronjob is now a Kubernetes cronjob, or what have you, but for all the complexity we've had to overcome, we've seen substantial benefits. For example, autoscaling instances of my data pipeline apps based on Kafka lag. Was it complex? Yes. We had to expose Kafka topic lag as a metric to Prometheus, then configure the k8s metrics server that HPAs (Horizontal Pod Autoscalers) use to scale to pull that in from Prometheus using k8s-prometheus-adapter as a custom metric, then set reasonable scaling limits in the data pipeline apps HPAs. Was it worth it? Fuck yes. We no longer have to worry about data arriving out of time at our data warehouse, because the scaling we've configured (in YAML, god rest its dirty soul) ensures that our data will always arrive at the data warehouse within 5 minutes, which is ideal for a near real time reporting product that greatly enables our end users. Is Kubernetes worth it? For us, definitely. For anyone else? Really depends on your use case, don't cargo cult this shit.

pnako 7y ago
How much data do your "data pipeline apps" process with Kafka, Kubernetes and probably other K-named things?
- EdwardDiego 7y ago
  Several terabytes a day, but it varies, hence why I love the auto-scaling.
  - throwawaybbb 7y ago
    That's 10tb = 10000gb = 10000/24/60/60 = .11gb/s. My 2015 desktop could handle that. Where do you work? I'd like to pitch my radical idea of edge consolidated cloud computing.
    - fulafel 7y ago
      "Big data" frameworks are very good at throwing a lot of hardware at problems. See eg the classic big data vs laptop treatment: http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
    - EdwardDiego 7y ago
      The inevitable comment lol. I'm sure it could. To clarify, when the app scales, it's pods (container instances) that are scaled, not necessarily machines,that is managed separately by k8s. Depending on how much of the cluster is currently being used. Yep, we could handle it easily on one machine, if we gave it unfettered access to all the resource, but our throughput varies during the time of day, so dedicating a machine to it to handle the peak throughput wouldn't make financial sense at 3am at night. So it's far simpler to scale pods with known resource usage as needed. It also helps prevent issues when throughput suddenly doubles because of a business decision that you're left out of the loop on. So autoscaling pods is ideal for our use case.
      - throwawaybbb 7y ago
LaserToy 7y ago
You don’t need kubernetes to do it, any cloud provider has api that will make it possible. Kube main point is declarative model, which comes with a big price (like sql had). Yaml - is error prone way to achieve anything. Will see if history repeats itself and we will see be imperative systems.

echelon 7y ago

> Kubernetes is a solution worse than the problem. Not if you're a 1,000+ engineer org with an SOA. > Microservices are unix pipes but slower. Humans are way more expensive than the machines running our code. You can always spend more on spinning up additional instances. In reality the overhead isn't that bad. The principles of immutable stateless deploys free up so much mental energy when it comes to thinking about how to get your code out and not break things that it more than pays for itself. > AI is linear algebra that costs x1000 because of the brand name. Tell that to my real time voice conversion system I just wrote. There's a lot of new magic happening right now. > There is no revolution in software. I'm on a fucking phone responding from hundreds of miles away. We're launching satellites that beam internet. > We are just building crud You can't speak for everyone here. Are you in adtech or a social spyware company? Maybe don't work like that then. Find meaningful work that speaks to you. > quality of developers tends to zero as the number of developers increases. I work with brilliant people.

buzzkillington 7y ago
>Not if you're a 1,000+ engineer org with an SOA. That's the problem. With a sane simple architecture instead of 40 years of crap on top of crap you wouldn't need thousands of engineers to spin up instances. >In reality the overhead isn't that bad. Over 1e2 times if the solution is in software, over 1e5 times if it's an fpga. Around 1e8 if you do it in silicon. You make the savings from not hiring the 1000s of engineers to deploy your bloated horror. >The principles of immutable stateless deploys Has nothing to do with k8s. Nixos and Guix are two systems that are as close to functional as possible and they sure as hell aren't 'stateless', even though they come closer than any of the sexy trending tech. >Tell that to my real time voice conversion system I just wrote. Thank you project librevox for proving a near infinite training corpus, thank you nvidia for gpus to do linear algebra 1e5 more efficiently than you can on a cpu. >I'm on a fucking phone responding from hundreds of miles away. We're launching satellites that beam internet. Like everything else you listed there this is a hardware improvement. A Casio watch from the 90s had the same order of magnitude computational capability as the first generation IMPs. Good luck getting it connected to the internet using any software you could write for it.

ummonk 7y ago

>AI is linear algebra that costs x1000 because of the brand name. Sure, but we've learnt to apply linear algebra to achieving things we could only dream of 10 (or even 5) years ago.

sheeshkebab 7y ago
Current AI is still mostly trash of the high school level project quality, linear algebra taken to the absolute possible level of insanity.
- adrianN 7y ago
  I'm pretty sure that no high school project can beat the world's best Go player.

deleted 7y ago

[deleted]

Animats 7y ago

Well, arguably that's true in the webcrap space. The HTML->CSS->JS->Webasm line of development really is crap on top of crap. Trying to fix the design problems of one layer by putting another layer on top of it only sort of works. It gets worse over time. "The mold seeps through the wallpaper" (a comment I usually make about C++ collection classes trying, but only partially succeeding, to hide the problems of C arrays). I was doing "microservices" on QNX using MsgSend/MsgReceive for hard real time over 15 years ago. It doesn't have to suck; the mainstream is just using the wrong tools for doing it fast. The QNX people managed to shoot themselves in the foot by going closed-source -> open source -> closed source -> open source -> closed source, after which their developer community was fed up. Machine learning is not linear algebra. It's nonlinear. That's what makes it useful.

ratww 7y ago
> The HTML->CSS->JS->Webasm line of development really is crap on top of crap. I completely get the hate for modern web, but in this case I think it's unrelated. The whole Kubernetes/microservice world exists basically to serve JSON and similar protocols. Maybe rarely some static HTML. For HTML->CSS->JS->Webasm stuff it is more fashionable to compile at build time and service as static assets. That's the polar opposite of Kubernetes/microservices.
cerberusss 7y ago
> Well, arguably that's true in the webcrap space. (...) crap on top of crap Not that you're wrong per say, but personally, I think you should take a look at the HN guidelines. Like don't be snarky, and don't post shallow dismissals. Just categorizing the arguably most popular way of building software as "crap" doesn't seem like thoughtful or curious conversation.

smallcharleston 7y ago

“ AI is linear algebra that costs x1000 because of the brand name.” This doesn’t even make sense because it can be said about anything involving numerical work on a computer and it’s still roughly true. Also because all of the libraries are free anyway.