12 ms·

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm increasingly coming to the view that there is a big split among "software developers" and AI is exacerbating it. There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes, and often just want to rent a VPS or at most a managed K8s solution. The other group (increasingly large) just wants to `git push` and be done with it, and they're willing to spend a lot of (usually their employer's) money to have that experience. They don't want to have to understand DNS, linux, or anything else beyond whatever framework they are using. A company like fly.io absolutely appeals to the latter. GPU instances at this point are very much appealing to the former. I think you have to treat these two markets very differently from a marketing and product perspective. Even though they both write code, they are otherwise radically different. You can sell the latter group a lot of abstractions and automations without them needing to know any details, but the former group will care very much about the details.

varenc 2y ago
Aren’t we just continually moving up layers of abstractions? Most of the increasingly small group doesn’t concern itself with voltages, manually setting jumpers, hand-rolling assembly for performance-critical code, cache line alignment, raw disk sector manipulation, etc. I agree it’s worthwhile to understand things more deeply but developers slowly moving up layers of abstractions seems like it’s been a long term trend.
- Moru 2y ago
  We certainly need abstractions for the first layer of the hardware. An abstraction of the abstraction can be useful if the first abstraction is very bad or very crude. But we are now at an abstraction of an abstraction x 8 or so. It's starting to get a bit over the top.
  - throwaway2037 2y ago
    I disgree with your sentiment. One thing that is constant in my experience as a computer programmer, there are always "old" computer programmers complaining that there are too many abstractions.
    - ehnto 2y ago
      You cannot see any way in which we run out of possible abstraction layers? I think in the past we have assumed that was natural language, but I think natural language is a pretty poor programming language. What people actually want when they say that, is for someone to read all the nuance out of their mind and codify it. I don't think we actually have been abstracting new layers over the past day 5-10 years anyway. Most of what I see is moving sideways, not up the stack. Covering more breadth not height or depth, of abstractions.
      - yazaddaruvala 2y ago
        We only run out of abstraction once there is stagnation and time to really bake. As long as some new thing is being invented in our industry, a new abstraction will be needed because the old one just can’t quite flex enough while being backwards compatible.
        immibis 2y ago
        One problem is that we're building abstractions on top of older abstractions when that isn't most efficient. Suppose we write an HTTP server that can call CGI programs. Then we implement PHP as a CGI program. Then we write a Lua interpreter in PHP. Then we write our website in Lua. Some of those levels are useful. Some of them are redundant. We should embed a Lua interpreter in our webserver and delete two levels of abstraction. (I'm not aware of any actual Lua interpreter written in PHP, but it's representative of the kinds of stacks that do exist out there)
        yazaddaruvala 2y ago
_dark_matter_ 2y ago
This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why do the details matter?
- lifeisstillgood 2y ago
  I used to think this, ut it only works if the abstractions hold - it’s like if we stopped random access memory and went back to tape drives suddenly abstractions matter. My comment elsewhere goes into but more detail but basically silicon stopped being able to make single threaded code faster in about 2012 - we just have been getting “more parallel cores” since. And now at wafer scale we see 900,000 cores on a “chip”. When 100% parallel coding runs 1 million times faster than your competitors, when following one software engineering path leads to code that can run 1M X, then we will find ways to use that excess capacity - and the engineers who can do it get to win. I’m not sure how LLMs face this problem.
  - jonas21 2y ago
    That's why abstractions like PyTorch exist. You can write a few lines of Python and get good utilization of all those GPU cores.
    - varelse 2y ago
      [dead]
  - dbcjv7vhxj 2y ago
    This. As soon as the abstractions leak or you run into an underlying issue you suddenly need to understand everything about the underlying system or you're SOOL. I'd rather have a simpler system I already understand all the proceeding abstractions about. The overhead of this is minimal when you keep things simple and avoid shiny things.
hypothesis 2y ago
> They don't want to have to understand DNS, linux, or anything else beyond whatever framework they are using. This is baffling. What’s value proposition here? At some point customer will be directly asking an AI agent to create an app for them and it will take care of coding/deployment for them..
- metaltyphoon 2y ago
  The value proposition is that you know how to fix something, when it eventually break, because you don’t fundamentally understand.
  - oarsinsync 2y ago
    GPT usually writes code that eventually (often immediately) breaks. When given the failure mode, it usually fixes the issue (and often creates a new one, GOTO 10)
- unification_fan 2y ago
  Some people became software developers because they like learning and knowing what they're doing, and why and how it works. Some people became software developers because they wanted to make easy money back when the industry was still advertising bootcamps (in order to drive down the cost of developers). Some people simply drifted into this profession by inertia. And everything in-between. From my experience there are a lot of developers who don't take pride in their work, and just do it because it pays the bills. I wouldn't want to be them but I get it. The thing is that by delegating all their knowledge to the tools they use, they are making themselves easy to replace, when the time comes. And if they have to fix something on their own, they can't. Because they don't understand why and how it works, and how and why it became what it is instead of something else. So they call me and ask me how that thing works...
  - jval43 2y ago
    This is my experience as well. I answer many such calls from devs as part of my work. I can usually tell at the end of a call which group they belong to. I've been wrong a few times too. As long as they don't waste my time I'm fine with everyone, some people just have other priorities in life. One thing I'd say is in my experience there are many competent and capable people in every group, but non-competent ones are extremely rare in the first group.
emilsedgh 2y ago
You described two points in an spectrum in which: One end is PaaS like Heroku, where you just git push. The other end is bare metal hosting. Every option you mentioned (VPS, Manages K8S, Self Hosted K8S, etc) they all fall somewhere between these two ends of the spectrum. If, a developer falls into any of these "groups" or has a preference/position on any of these solutions, they are just called juniors. Where you end up in this spectrum is a matter of cost benefit. Nothing else. And that calculation always changes. Those options only make sense where the cost of someone else managing it for you for a small premium gets higher than the opportunity/labor cost of you doing it yourself. So, as a business, you _should_ not have a preference to stick to. You should probably start with PaaS, and as you grow, if PaaS costs get too high, slowly graduate into more self-managed things. A company like fly.io is a PaaS. Their audience has always been, and will always be application developers who prefer to do nothing low-level. How did they forget this?
- tptacek 2y ago
  This is news to us. Our primary DX is a CLI. One of our defining features is hardware isolation. To use us, you have to manage Dockerfiles. Have you had the experience of teaching hundreds of Heroku refugees how to maintain a Dockerfile? We have had that experience. Have you ever successfully explained the distinction between "automated" Postgres and "managed" Postgres? We have not. You're not wrong that there's a PaaS/public-cloud dividing line, and that we're at an odd place between those two things. But I mean, no, it is not the case that our audience is strictly developers who do nothing low-level. I spent months of my life getting _UDP_ working for Fly apps!
  - emilsedgh 2y ago
    Ok, let me rephrase this: > Their audience has always been, and will always be application developers who prefer to do nothing low-level. How did they forget this? to this: Their audience has always been, and will always be application developers who prefer to do nothing except to build their main product. > Our primary DX is a CLI. One of our defining features is hardware isolation. To use us, you have to manage Dockerfiles. Have you had the experience of teaching hundreds of Heroku refugees how to maintain a Dockerfile? We have had that experience. Have you ever successfully explained the distinction between "automated" Postgres and "managed" Postgres? We have not. I'm pretty much sure an application developer in this day and age has to know all of them, yes. Just like git.
Yoric 2y ago
I'm not entirely certain. Or perhaps we're all part of both groups. I want to understand LLMs. I want to understand my compiler, my gc, my type system, my distributed systems. On the other hand, I don't really care about K8s or anything else, as long as I have something that works. Just let me `git push` and focus on making great things elsewhere.
- notatoad 2y ago
  >Or perhaps we're all part of both groups. this feels right to me. application development and platform development are both software development tasks, and lots of software devs do both. i like working on platform-level stuff, and i like building applications. but i like there to be a good distinction between the two, and when i'm working on application-level stuff, i don't want to have to think about the platform. services like fly.io do a good job of hiding all the platform level work and just giving you a place to deploy your application to, so when they start exposing tools like GPUs that are more about building platforms than building applications, it's messy.
SkyPuncher 2y ago
I enjoy the details, but I don’t get paid to tell my executives how we’re running things. I get paid to ship customer facing value. Particularly at startups, it’s almost always more cost effective to hit that “scale up” button from our hosting provider than do any sort of actual system engineering. Eventually, someone goes “hey we could save $$$$ by doing XYZ” so we send someone on a systems engineering journey for a week or two and cut our bill in half. None of it really matters, though. We’re racing against competition and runway. A few days less runway isn’t going to break a startup. Not shipping as fast as reasonable will.
- layoric 2y ago
  I’ve been in similar situations, but details matter. If your scale up button is heavily abstracted services, your choice starts to become very different as the cost of reimplementing what the service does might be high enough that you end up with a no win situation of your own making. The closer your “Scale up” button is referencing actual hardware, the less of a problem it is.
  - aeyes 2y ago
    That's the next problem startups should avoid at all cost. Don't do heavily abstracted services, just put it all in a monolith which will make it faster and easier to iterate. Don't overthink it, just get the feature out of the door. Chances are high that you won't get it right from the beginning, you can create these abstractions once you really understand the problem space with real world data. When you get to that point I have another pro tip: Don't refactor, just rewrite it and put all your learnings into the v2.
    - theoreticalmal 2y ago
      This is exactly, precisely what my experience has been.
    - alanmoraes 2y ago
      Why don't refactor just the parts the need it instead of rewriting everything?
anal_reactor 2y ago
> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. (...) The other group (increasingly large) just wants to `git push` and be done with it I think we're approaching the point where software development becomes a low-skilled job, because the automatic tools are good enough to serve business needs, while manual tools are too difficult to understand by anyone but a few chosen ones anyway.
conradev 2y ago
I feel like fly.io prioritizes a great developer experience and I think that appeals to engineers who both do and don't like magic. But the real reason I like fly.io is because it is a new thing that allows for new capabilities. It allows you to build your own Cloudflare by running full virtual machines colocated next to appliances in a global multicast network.
Diggsey 2y ago
I don't agree, I think you're just describing two sides of the same coin. As a software developer I want strong abstractions without bloat. LLMs are so successful in part because they are a really strong abstraction. You feed in text and you get back text. Depending on the model and other parameters your results may be better or worse, but changing from eg. Claude to ChatGPT is as simple as swapping out one request with another. If what I want is to run AI tasks, then GPUs are a poor abstraction. It's very complicated (as Fly have discovered) to share them securely. The amount of GPU you need could vary dramatically. You need to worry about drivers. You need to worry about all kinds of things. There is very little bloat to the ChatGPT-style abstraction, because the network overhead is a negligable part of the overall cost. If I say I don't want magic, what I really mean is that I don't trust the strength of the abstraction that is being offered. For example, when a distributed SQL database claims to be PostgreSQL compatible, it might just mean it's wire compatible, so none of my existing queries will actually work. It might have all the same functions but be missing support for stored procedures. The transaction isolation might be a lie. It's not that these databases are bad, it's that "PostgreSQL as a whole" cannot serve as a strong abstraction boundary - the API surface is simply too large and complex, and too many implementation details are exposed. It's the same reason people like containers: running your application on an existing system is a very poor abstraction. The API surface of a modern linux distro is huge, and includes everything from what libraries come pre-installed to the file-system layout. On the other hand the kernel API is (in comparison) small and stable, and so you can swap out either side without too much fear. K8S can be a very good abstraction if you deploy a lot of services to multiple VMs and need a lot of control over how they are scaled up and down. If you're deploying a single container to a VM, it's massively bloated. TLDR: Abstractions can be good and bad, both inherently, and depending on your use-case. Make the right choice based on your needs. Fly are probably correct that their GPU offering is a bad abstraction for many of their customer's needs.
nicoburns 2y ago
> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...
- sanderjd 2y ago
  Really? There are plenty of valid criticisms of kubernetes, but this doesn't strike me as one of them. It gives you tons of control over all of this. That's a big part of why it's so complex!
  - jldugger 2y ago
    IMO, it's rather hard to fully know all of kubernetes and what it's doing, and the kind of person who demands elegance in solutions will hate it.
  - sgarland 2y ago
    It gives you control via abstractions. That’s fine, and I like K8s personally, but if you don’t understand the underlying fundamentals that it’s controlling, you don’t understand what it’s doing.
  - PaulHoule 2y ago
    This mainframe system from the 1990s was so much simpler https://www.ibm.com/docs/en/cics-ts/6.x?topic=sysplex-parallel-principles https://www.ibm.com/docs/en/cics-ts/6.x?topic=sysplex-parall... even if it wasn't as scalable as Kube. One the other hand, a cluster of 32 CMOS mainframe could handle any commercial computing job that people were doing in the 1990s.
    - sanderjd 2y ago
      It's not simple but it's not opaque.
    - 2y ago
chrismarlow9 2y ago
I call this difference being a developer who is on call vs being a developer who is not on call
brightball 2y ago
I am the former. I also make cost benefit based decisions that involve time. Unless I have very specific configuration needs, the git push option lets me focus on what my users care about and gives me one less thing that I need to spend my time on. Increasingly, Fly even lets you dip into most complex configurations too. I’ve got no issue with using Tofu and Ansible to manage my own infrastructure but it takes time to get it right and it’s typically not worth the investment early on in the lifecycle.
teaearlgraycold 2y ago
It depends on the product you're building. At my last job we hosted bespoke controlnet-guided diffusion models. That means k8s+GPUs was a necessity. But I would have loved to use something simpler than k8s.
deleted 2y ago
[deleted]
danstewart_ 2y ago
I don’t think this comment does justice to fly.io. They have incredible defaults that can make it as simple as just running ‘git push’ but there isn’t really any magic happening, it’s all documented and configurable.
dirtbag__dad 2y ago
> they're willing to spend a lot of (usually their employer's) money May just be my naïveté, but I thought that something like ECS or EKS is much cheaper than an in-house k8 engineer.
- sgarland 2y ago
  If you don’t have staff to do that, you probably aren’t at the scale when you need them, and you’re needlessly adding complexity. It’s always baffling to me why people think that ECS or god forbid EKS is somehow easier than a few Linux boxes.
  - mdaniel 2y ago
    Because the force multiplier of a good DX way outweighs the occasional nonsense from having to do k8s upgrades or troubleshooting For example: how do you roll out a new release of your product? In sane setups, it's often $(helm upgrade --install ...), which is itself often run either in-cluster by watching a git managed descriptor, or in CI on merge to a release branch/tag How does your developer get logs? Maybe it's via Splunk/ELK/DataDog/whatever but I have never in my life seen a case where that's a replacement for viewing the logs How do you jump into the execution environment for your workload, to do more advanced debugging? I'm sure you're going to say ssh, which leads to the next questions of "how do you audit what was done, to prevent config drift" followed by "how do you authenticate the right developer at the right time with access to the right machine without putting root's public key file in a spreadsheet somewhere"
    - sethammons 2y ago
      > Splunk/ELK/DataDog/whatever but I have never in my life seen a case where that's a replacement for viewing the logs Uh, any time I run a distributed system and logs could appear on n nodes I need a log aggregator or I am tailing in n terminals. I almost only use Splunk. I tail logs in dev. Prod needs an aggregator. This has been my experience at 4 of my last 6 companies. The shit companies who had all the issues? Logs on cloudwatch or only on the node
Glyptodon 2y ago
I think it's true that engineers who want to understand every layer of everything in depth, or who want to have platform ownership, are not necessarily the same group as the more "product itself" focused sort who want to write something and just push it, I don't actually think I'm sold at all that any of these groups, in a vacuum, have substantial demand for GPU compute unless that's someone's area of interest for a pet project.
- flufluflufluffy 2y ago
  This. Personally, I’d want a GPU to self host whatever model, because I think that’s fun, plain and simple. Probably many people do too. But the business is not making money from people who are just thinking about fun.
the__alchemist 2y ago
Where does this dichotomy between Kubernetes, and superficial understanding come from? It is not consistent with my experience, and I don't have speculation its origin.
aaron695 2y ago
[dead]
startupsfail 2y ago
The view that developers just want LLMs is plain wrong. The age of AI is just starting.
jsemrau 2y ago
>who don't like "magic" and want to understand where their code is running and what it's doing. I just made this point in a post on my substack. Especially in regulated industries, you NEED to the able to explain your AI to the regulator. You can't have a situation where a human say "Well, gee I don't know. The AI told me to do it."
chii 2y ago
> They don't want to have to understand DNS, linux, or anything else beyond whatever framework they are using. tell me whether there's many brick layers who wants to understand the chemical composition of their bricks.
- steve_adams_86 2y ago
  I’ve never laid bricks but in other trades I’ve worked in, well, a lot of people understood basics of the chemistry of the products we used. It’s useful to understand how they work together safely, if they can be exposed to different environments, if they’re heat-safe, cold-safe, do they off-gas, etc. Paints, wood finishes, adhesives, oils, abrasives, you name it. You generally know at least a bit about what’s in it. I can’t say everyone I’ve worked with wanted to know, but it’s often intrinsic to what you’re doing and why. You don’t just pull a random product off a shelf and use it. You choose it, quite often, because of its chemical composition. I suspect it’s not always thought of this way, though. This is the same with a lot of artistic mediums as well. Ceramicists often know a lot more than you’d expect about what’s in their clay and glazes. It’s really cool. I’m not trying to be contrarian here. I know some people don’t care at all, and some people use products because it’s what they were told to do and they just go with it. But that wasn’t my experience most of the time. Maybe I got lucky, haha.
- anthk 2y ago
  In my country I think the vocational/trade degree -which might lie between HS and uni level- on car mechanics has basic physics, mechanics and maybe some chemistry. Ditto for the rest of technical voc degrees. If you think you can do IT without at least a trade degree on understanding the low level components interact, (and I'm not talking about CS level, concurrency with CSP, O-notation, linear+discrete algebra... but basic stuff such as networking protocols, basic SQL database normalizations, system administration, configuration, how the OS boots, how processes work -idle, active, waiting..., if you don't get that, you will be fired faster than anyone around.
lakomen 2y ago
[dead]
motorest 2y ago
> I'm increasingly coming to the view that there is a big split among "software developers" and AI is exacerbating it. I don't think this split exists, at least in the way you framed it. What does exist is workload, and problems that engineers are tasked with fixing. If you are tasked with fixing a problem or implementing a feature, you are not tasked with learning all the minute details or specifics of a technology. You are tasked with getting shit done, which might even turn out to not involve said technology. You are paid to be a problem-solver, not an academic expert on a specific module. What you tried to describe as "magic" is actually the balance between broad knowledge vs specialization, or being a generalist vs specialist. The bulk of the problems that your average engineer faces requires generalists, not specialists. Moreover, the tasks that actually require a specialist are rare, and when those surface the question is always whether it's worth to invest in a specialist. There are diminished returns on investment, and throwing a generalist at the problem will already get some results. You give a generalist access to a LLM and he'll cut down on the research time to deliver something close to what a specialist would deliver. So why bother? With this in mind, I would go as far as to frame a scenario backhandedly described as "want to understand where their code is running and what it's doing" (as if no engineer needs to have insight on how things work?) as opposed to the dismissive "just wants to `git push` and be done with it" scenario, can actually be classified as a form of incompetence. You,as an engineer, only have so many hours per day. Your day-to-day activities involve pushing new features and fixing new problems. To be effective, your main skillet is learn the system in a JIT way, dive in, fix it, and move on. You care about system traits, not low-level implementation details that may change tomorrow on a technology you may not even use tomorrow. If, instead, you feel the need to waste time on topics that are irrelevant to address the immediate needs of your role, you are failing to deliver value. I mean, if you frame yourself as a Kubernetes expert who even know commit hashes by heart, does that matter if someone asks you, say, why is a popup box showing off-center?
skrebbel 2y ago
"Enjoys doing linux sysadmin" is not the same as "Wants to understand how things work". It's weird to me that you group those two kinds of people in one bucket.
deleted 2y ago
[deleted]
pinoy420 2y ago
I was thinking about this just yesterday. I was advertised a device for an aircraft to geo-assist taxiing? (I have never flown so I don’t know why). The comments were the usual “old man shouts at cloud” angry that assistive devices make lives easier for people. I feel this is similar to what you are pointing out. Why _shouldn’t_ people be the “magic” users. When was the last time one of your average devs looked in to how esm loading? Or the python interpreter or v8? Or how it communicates with the OS and lower level hardware interfacing? This is the same thing. Only you are goalpost shifting.
andai 2y ago
I'd be in the latter group if my budget were infinite. Alas!
zwnow 2y ago
Also "They want LLMs" lol. I cant remember being asked. I dont want to use AI for coding.
black3r 2y ago
All professional developers want two things: Do their work as fast as possible and spend as little budget to make things work. That's the core operating principle of most companies. What's changing is that managed solutions are becoming increasingly easier to set up and increasingly cheaper on smaller scales. While I do personally enjoy understanding the entire stack, I can't justify self-hosting and managing an LLM until we run so many prompts a day that it becomes cheaper for us to run our own GPUs compared to just running APIs like OpenAI/Anthropic/Deepseek/...
_heimdall 2y ago
I agree that split exists, and that the former is more rare, but in my experience the split is less about avoid magic and more about keeping control of your system. Many, likely most, developers today don't care about controlling their system/network/hardware. There's nothing wrong with that necessarily, but it is a pretty fundamental difference. One concern I've had with building LLM features is whether my customers would be okay with me giving their data over to the LLM vendor. Say I'm building a tool for data analysis, is it really okay to a customer for me to give their table schemas or access to the data itself to OpenAI, for example? I rarely hear that concern raised though. Similarly when I was doing consulting recently, I wouldn't use copilot on client projects as I didn't want copilot servers accessing code that I don't actually own the rights to. Maybe its over protective though, I have never heard anyone raise that concern so maybe its just me.
- chasd00 2y ago
  I work for a major consulting firm and we’ve been threatened with fire and brimstone if any part of client info (code, docs, random email, anything) ever gets sent to an LLM. Even with permission from the client our attack lawyers prefer us not to use them. It’s a very sensitive topic. I still use LLMs from time to time but always starting with a blank prompt and the ask anonymized. (Heh I’m probably not even supposed to do that)
yungporko 2y ago
increasingly small is right. i'm definitely part of that former group but sadly more and more these days i just feel dumb for being this way. it usually just means that i'm less productive than my colleagues in practice as i'm spending time figuring out how things work while everybody else is pushing commits. maybe if we were put in a hypothetical locked room with no internet access i'd have a slightly easier time than them but that's not helpful to anybody. once upon a time i could have said that it's better this way and that everybody will be thankful when i'm the only person who can fix something, but at this point that isn't really true when anybody can just get an LLM to walk them through it if they need to understand what's going on under the hood. really i'm just a nerd and i need to understand if i want to sleep at night lol.
killerstorm 2y ago
I don't think you got this split right. I prefer to either manage software directly with no wrappers on top, or use a fully automated solution. K8S is something I'd rather avoid. Do you enjoy writing configuration for your automation layer?
IlikeKitties 2y ago
I don't think this is entirely correct. I'm working for a company that does IT Consulting and so I see many Teams working on many different Projects and one thing I have learned the hard way is that Companies and Teams that think they should do it all themselves are usually smaller companies and they often have a lot of Problems with that attitude. Just an example I recently came across: Working for a smaller company that uses Kubernetes and manages everything themselves with a small team. The result: They get hacked regularly and everything they run is constantly out of date because they don't have the capacity to actually manage it themselves. And it's not even cheaper in the long run because Developer Time is usually more expensive than just paying AWS to keep their EKS up to date. To be fair, in my home lab I also run everything bare metal and keep it updated but I run everything behind a VPN connection and run a security scanner every weekend that automatically kills any service it finds > Medium Level CVE and I fix it when I get the time to do it. As a small Team I can only fix so much and keep so much up to date before I get overwhelmed or the next customer Project gets forced upon me by Management with Priority 0, who cares about security updates. I'd strongly suggest to use as much managed service as you can and focus your effort as a team on what makes your Software Unique. Do you really need to hire 2-3 DevOps guys just to keep everything running when GCP Cloud Run "just werks"? Everything we do these days runs on so many levels of abstraction anyway, it's no shame to share cost of managing the lower levels of abstraction with others (using managed Service) and focus on your product instead. Unless you are large enough to pay for whole teams that deal with nothing but infrastructure to enable other teams to do Application Level Programming you are, in my limited experience, just going to shoot yourself in the foot. And again, just to emphasize it: I like to do everything myself because for privacy reasons I use as little services that aren't under my control as possible but I would not recommend this to a customer because it's neither economical nor does it work well in my, albeit limited, experience.
memhole 2y ago
I might be an outlier. I like to think I try for a deeper understanding of what I’m using. Like, fly uses firecracker vms afaik. Sometimes, especially for quick projects or testing ideas I just want to have it work without wrangling a bunch of AWS services. I’m typically evaluating is this the right tool or service and what is the price to convenience? For anything potentially long term, what’s the amount of lock in when or if I want to change providers?
ripped_britches 2y ago
The way I think about it is this: any individual engineer (or any individual team) has a limited complexity budget (in other words, how much can you fit in your meat brain). How you spend it is a strategic decision. Depending on your project, you may not want to waste it on infra so you can fit a lot of business logic complexity.
tarsinge 2y ago
It’s not about wanting, it’s about what the job asks for. As a self employed engineer I am paid to solve business problems in an efficient way. Most of the time it just make more business sense for the client and for me to pay to just have to git push if there is no performance challenges needing custom infrastructure.
EarthIsHome 2y ago
> The other group (increasingly large) just wants to `git push` and be done with it, and they're willing to spend a lot of (usually their employer's) money to have that experience. They don't want to have to understand DNS, linux, or anything else beyond whatever framework they are using. lol, even understanding git is hard for them. Increasingly, software engineers don't want to learn their craft.
- chasd00 2y ago
  I think the root of it is most people coming into the software engineering industry just want a good paying job. They don’t have any real interest in computers or networks or anything else. Whatever keeps the direct deposits coming is what they’ll do. And in their defense, the web dev industry is so large in breadth and depth and the pay/benefits are so generous it’s an attractive career path no matter what your passion is.
vagrantJin 2y ago
> I'm increasingly coming to the view that there is a big split among "software developers" and AI is exacerbating it This is admittedly low effort but the vast majority of devs are paid wages to "write CRUD, git push and magic" their way to the end of the month. The company does not afford them the time and privilege of sitting down and analyzing the code with a fine comb. An abstraction that works is good enough. The seasoned seniors get paid much more and afforded leeway to care about what is happening in the stack, since they are largely responsible for keeping things running. I'm just pointing out it might merely be a function of economics.
sudhirj 2y ago
This is context based dichotomy, not a person-based one. In my personal life, I’m curiosity-oriented, so I put my blog, side projects and mom’s chocolate shop on fully self hosted VPSs. At my job managing a team of 25 and servicing thousands of customers for millions in revenue, I’m very results-oriented. Anyone who tries to put a single line of code outside of a managed AWS service is going to be in a lot of trouble with me. In a results-oriented environment, I’m outsourcing a lot of devops work to AWS, and choosing to pay a premium because I need to use the people I hire to work on customer problems. Trying to conflate the two orientations with mindsets / personality / experience levels is inaccurate. It’s all about context.
antihero 2y ago
The former have the mentality of being independent, at the cost of their ability to produce a result as quickly. The latter are happy to be dependent, because the result is more important than the means. Obviously this is a spectrum.
hassleblad23 2y ago
You lost me at "Kubernetes".
seanp2k2 2y ago
Having worked with many of the latter and having had the displeasure of educating them on nix systems fundamentals: ugh, oof, I hate this timeline, yet I also feel a sense of job security. We used to joke about this a lot when Java devs would have memory issues and not know how to adjust the heap size in init scripts. So many “CS majors” who are completely oblivious to anything happening outside of the JVM, and plenty happening within it.
lovich 2y ago
> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. That problem started so long ago and has gotten so bad that I would be hard pressed to believe there is anyone on the planet who could take a modern consumer pc and explain what exactly is going on the machine without relying on any abstractions to understand the actual physical process. Given that, it’s only a matter of personal preference on where you draw the line for magic. As other commenters have pointed out, your line allowing for Kubernetes is already surprising to a lot of people
ianmcnaney 2y ago
Somebody who doesn’t want to understand DNS, Linux, or anything beyond their framework is a hazard. They’re not able to do a competent code review on the vomit that LLMs produce. (Am I biased much?)
khana 2y ago
[dead]
westurner 2y ago
IaaS or PaaS? Who owns and depreciates the logs, backups, GPUs, and the database(s)? K8s docs > Scheduling GPUs: https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/ https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus... : > Once you have installed the plugin, your cluster exposes a custom schedulable resource such as amd.com/gpu or nvidia.com/gpu. > You can consume these GPUs from your containers by requesting the custom GPU resource, the same way you request cpu or memory awesome-local-ai: Platforms / full solutions https://github.com/janhq/awesome-local-ai?platforms--full-solutions https://github.com/janhq/awesome-local-ai?platforms--full-so... But what about TPUs (Tensor Processing Units) and QPUs (Quantum Processing Units)? Quantum backends: https://github.com/tequilahub/tequila#quantum-backends https://github.com/tequilahub/tequila#quantum-backends Kubernetes Device Plugin examples: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#examples https://kubernetes.io/docs/concepts/extend-kubernetes/comput... Kubernetes Generic Device Plugin: https://github.com/squat/generic-device-plugin#kubernetes-generic-device-plugin https://github.com/squat/generic-device-plugin#kubernetes-ge... K8s GPU Operator: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html#pod-security-context-of-the-operator-and-operands https://docs.nvidia.com/datacenter/cloud-native/gpu-operator... Re: sunlight server and moonlight for 120 FPS 4K HDR access to GPU output over the Internet: https://github.com/kasmtech/KasmVNC/issues/305#issuecomment-2600828249 https://github.com/kasmtech/KasmVNC/issues/305#issuecomment-... : > Still hoping for SR-IOV in retail GPUs. > Not sure about vCPU functionality in GPUs Process isolation on vCPUs with or without SR-IOV is probably not as advanced as secure enclave approaches. Intel SGX is a secure enclave capability, which is cancelled on everything but Xeon. FWIU there is no SGX for timeshared GPUs. What executable loader reverifies the loaded executable in RAM after imit time ? What LLM loader reverifies the in-RAM model? Can Merkle hashes reduce that cost; of nn state verification? Can it be proven that a [chat AI] model hosted by someone else is what is claimed; that it's truly a response from "model abc v2025.02"? PaaS or IaaS
jrochkind1 2y ago
Eh, the way I see it the entire practice of computer science and software engineering is built on abstraction -- which can be described as the the ability to not have to understand lower levels -- to only have to understand the API and not the implementations of the lowest levels you are concerned with, and to have to pay even less attention to lower levels than that. I want to understand every possible detail about my framework and language and libraries. Like I think I understand more than many do, and I want to understand more, and find it fulfilling to learn more. I don't, it's true, care to understand the implementation details of, say, the OS. I want to know the affordances it offers me and the APIs that matter to me, I don't care about how it's implemented. I don't care to understand more about DNS than I need. I definitely don't care to spend my time futzing with kubernetes -- I see it as a tool, and if I can use a different tool (say heroku or fly.io) that lets me not have to learn as much -- so I have more time to learn every possible detail of my language and framework, so I can do what I really came to do, develop solutions as efficiently and maintainably as possible. You are apparently interested in lower levels of abstraction than I am. Which is fine! Perhaps you do ops/systems/sre and don't deal with the higher levels of abstraction as much as I do -- that is definitely lucrative these days, there are plenty of positions like that. Perhaps you deal with more levels of abstraction but don't go as deep as me -- or, and I totally know it's possible, you just have more brain space to go as deep or deeper on more levels of abstraction as me. But even you probably don't get into the implementation details of electrical engineering and CPU design? Or if you do, and also go deep on frameworks and languages, I think you belong to a very very small category! But I also know developers who, to me, dont' want to go to deep on any of the levels of abstraction. I admit I look down on them, as I think you do too, they seem like copy-paste coders who will never be as good at developing efficient maintainable soltuions. I started this post saying I think that's a different axis than what layers of abstraction one specializes in or how far down one wants to know the details. But as I get here, while I still think that's likely, I'm willing to consider that these developers I have not been respecting -- are just going really deep in even higher levels of abstraction than me? Some of them maybe, but honestly I don't think most of them, but I could be wrong!
hk1337 2y ago
The latter group sounds like they're more managers than software developers.
draw_down 2y ago
[dead]
samstave 2y ago
Wait until LLMS are deployed in edge FPGAs
garfieldnate 2y ago
It's been a while since I tried, but my experience trying to manually set up GPUs was atrocious, and with investigation generally ending at the closed-source NVidia drivers it's easy to feel disempowered pretty quickly. I think my biggest learning from trying to do DL on a manually set up computer was simply that GPU setup was awful and I never wanted to deal with it. It's not that I don't want to understand it, but with NVidia software you're essentially not allowed to understand it. If open source drivers or open GPU hardware were released, I would gladly learn how that works.

varenc 2y ago

Aren’t we just continually moving up layers of abstractions? Most of the increasingly small group doesn’t concern itself with voltages, manually setting jumpers, hand-rolling assembly for performance-critical code, cache line alignment, raw disk sector manipulation, etc. I agree it’s worthwhile to understand things more deeply but developers slowly moving up layers of abstractions seems like it’s been a long term trend.

Moru 2y ago
We certainly need abstractions for the first layer of the hardware. An abstraction of the abstraction can be useful if the first abstraction is very bad or very crude. But we are now at an abstraction of an abstraction x 8 or so. It's starting to get a bit over the top.
- throwaway2037 2y ago
  I disgree with your sentiment. One thing that is constant in my experience as a computer programmer, there are always "old" computer programmers complaining that there are too many abstractions.
  - ehnto 2y ago
    You cannot see any way in which we run out of possible abstraction layers? I think in the past we have assumed that was natural language, but I think natural language is a pretty poor programming language. What people actually want when they say that, is for someone to read all the nuance out of their mind and codify it. I don't think we actually have been abstracting new layers over the past day 5-10 years anyway. Most of what I see is moving sideways, not up the stack. Covering more breadth not height or depth, of abstractions.
    - yazaddaruvala 2y ago
      We only run out of abstraction once there is stagnation and time to really bake. As long as some new thing is being invented in our industry, a new abstraction will be needed because the old one just can’t quite flex enough while being backwards compatible.
      - immibis 2y ago
        One problem is that we're building abstractions on top of older abstractions when that isn't most efficient. Suppose we write an HTTP server that can call CGI programs. Then we implement PHP as a CGI program. Then we write a Lua interpreter in PHP. Then we write our website in Lua. Some of those levels are useful. Some of them are redundant. We should embed a Lua interpreter in our webserver and delete two levels of abstraction. (I'm not aware of any actual Lua interpreter written in PHP, but it's representative of the kinds of stacks that do exist out there)
        yazaddaruvala 2y ago

_dark_matter_ 2y ago

This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why do the details matter?

lifeisstillgood 2y ago
I used to think this, ut it only works if the abstractions hold - it’s like if we stopped random access memory and went back to tape drives suddenly abstractions matter. My comment elsewhere goes into but more detail but basically silicon stopped being able to make single threaded code faster in about 2012 - we just have been getting “more parallel cores” since. And now at wafer scale we see 900,000 cores on a “chip”. When 100% parallel coding runs 1 million times faster than your competitors, when following one software engineering path leads to code that can run 1M X, then we will find ways to use that excess capacity - and the engineers who can do it get to win. I’m not sure how LLMs face this problem.
- jonas21 2y ago
  That's why abstractions like PyTorch exist. You can write a few lines of Python and get good utilization of all those GPU cores.
  - varelse 2y ago
    [dead]
- dbcjv7vhxj 2y ago
  This. As soon as the abstractions leak or you run into an underlying issue you suddenly need to understand everything about the underlying system or you're SOOL. I'd rather have a simpler system I already understand all the proceeding abstractions about. The overhead of this is minimal when you keep things simple and avoid shiny things.

hypothesis 2y ago

> They don't want to have to understand DNS, linux, or anything else beyond whatever framework they are using. This is baffling. What’s value proposition here? At some point customer will be directly asking an AI agent to create an app for them and it will take care of coding/deployment for them..

metaltyphoon 2y ago
The value proposition is that you know how to fix something, when it eventually break, because you don’t fundamentally understand.
- oarsinsync 2y ago
  GPT usually writes code that eventually (often immediately) breaks. When given the failure mode, it usually fixes the issue (and often creates a new one, GOTO 10)
unification_fan 2y ago
Some people became software developers because they like learning and knowing what they're doing, and why and how it works. Some people became software developers because they wanted to make easy money back when the industry was still advertising bootcamps (in order to drive down the cost of developers). Some people simply drifted into this profession by inertia. And everything in-between. From my experience there are a lot of developers who don't take pride in their work, and just do it because it pays the bills. I wouldn't want to be them but I get it. The thing is that by delegating all their knowledge to the tools they use, they are making themselves easy to replace, when the time comes. And if they have to fix something on their own, they can't. Because they don't understand why and how it works, and how and why it became what it is instead of something else. So they call me and ask me how that thing works...
- jval43 2y ago
  This is my experience as well. I answer many such calls from devs as part of my work. I can usually tell at the end of a call which group they belong to. I've been wrong a few times too. As long as they don't waste my time I'm fine with everyone, some people just have other priorities in life. One thing I'd say is in my experience there are many competent and capable people in every group, but non-competent ones are extremely rare in the first group.

emilsedgh 2y ago

You described two points in an spectrum in which: One end is PaaS like Heroku, where you just git push. The other end is bare metal hosting. Every option you mentioned (VPS, Manages K8S, Self Hosted K8S, etc) they all fall somewhere between these two ends of the spectrum. If, a developer falls into any of these "groups" or has a preference/position on any of these solutions, they are just called juniors. Where you end up in this spectrum is a matter of cost benefit. Nothing else. And that calculation always changes. Those options only make sense where the cost of someone else managing it for you for a small premium gets higher than the opportunity/labor cost of you doing it yourself. So, as a business, you _should_ not have a preference to stick to. You should probably start with PaaS, and as you grow, if PaaS costs get too high, slowly graduate into more self-managed things. A company like fly.io is a PaaS. Their audience has always been, and will always be application developers who prefer to do nothing low-level. How did they forget this?

tptacek 2y ago
This is news to us. Our primary DX is a CLI. One of our defining features is hardware isolation. To use us, you have to manage Dockerfiles. Have you had the experience of teaching hundreds of Heroku refugees how to maintain a Dockerfile? We have had that experience. Have you ever successfully explained the distinction between "automated" Postgres and "managed" Postgres? We have not. You're not wrong that there's a PaaS/public-cloud dividing line, and that we're at an odd place between those two things. But I mean, no, it is not the case that our audience is strictly developers who do nothing low-level. I spent months of my life getting _UDP_ working for Fly apps!
- emilsedgh 2y ago
  Ok, let me rephrase this: > Their audience has always been, and will always be application developers who prefer to do nothing low-level. How did they forget this? to this: Their audience has always been, and will always be application developers who prefer to do nothing except to build their main product. > Our primary DX is a CLI. One of our defining features is hardware isolation. To use us, you have to manage Dockerfiles. Have you had the experience of teaching hundreds of Heroku refugees how to maintain a Dockerfile? We have had that experience. Have you ever successfully explained the distinction between "automated" Postgres and "managed" Postgres? We have not. I'm pretty much sure an application developer in this day and age has to know all of them, yes. Just like git.

Yoric 2y ago

I'm not entirely certain. Or perhaps we're all part of both groups. I want to understand LLMs. I want to understand my compiler, my gc, my type system, my distributed systems. On the other hand, I don't really care about K8s or anything else, as long as I have something that works. Just let me `git push` and focus on making great things elsewhere.

notatoad 2y ago
>Or perhaps we're all part of both groups. this feels right to me. application development and platform development are both software development tasks, and lots of software devs do both. i like working on platform-level stuff, and i like building applications. but i like there to be a good distinction between the two, and when i'm working on application-level stuff, i don't want to have to think about the platform. services like fly.io do a good job of hiding all the platform level work and just giving you a place to deploy your application to, so when they start exposing tools like GPUs that are more about building platforms than building applications, it's messy.

SkyPuncher 2y ago

I enjoy the details, but I don’t get paid to tell my executives how we’re running things. I get paid to ship customer facing value. Particularly at startups, it’s almost always more cost effective to hit that “scale up” button from our hosting provider than do any sort of actual system engineering. Eventually, someone goes “hey we could save $$$$ by doing XYZ” so we send someone on a systems engineering journey for a week or two and cut our bill in half. None of it really matters, though. We’re racing against competition and runway. A few days less runway isn’t going to break a startup. Not shipping as fast as reasonable will.

layoric 2y ago
I’ve been in similar situations, but details matter. If your scale up button is heavily abstracted services, your choice starts to become very different as the cost of reimplementing what the service does might be high enough that you end up with a no win situation of your own making. The closer your “Scale up” button is referencing actual hardware, the less of a problem it is.
- aeyes 2y ago
  That's the next problem startups should avoid at all cost. Don't do heavily abstracted services, just put it all in a monolith which will make it faster and easier to iterate. Don't overthink it, just get the feature out of the door. Chances are high that you won't get it right from the beginning, you can create these abstractions once you really understand the problem space with real world data. When you get to that point I have another pro tip: Don't refactor, just rewrite it and put all your learnings into the v2.
  - theoreticalmal 2y ago
    This is exactly, precisely what my experience has been.
  - alanmoraes 2y ago
    Why don't refactor just the parts the need it instead of rewriting everything?

anal_reactor 2y ago

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. (...) The other group (increasingly large) just wants to `git push` and be done with it I think we're approaching the point where software development becomes a low-skilled job, because the automatic tools are good enough to serve business needs, while manual tools are too difficult to understand by anyone but a few chosen ones anyway.

conradev 2y ago

I feel like fly.io prioritizes a great developer experience and I think that appeals to engineers who both do and don't like magic. But the real reason I like fly.io is because it is a new thing that allows for new capabilities. It allows you to build your own Cloudflare by running full virtual machines colocated next to appliances in a global multicast network.

Diggsey 2y ago

I don't agree, I think you're just describing two sides of the same coin. As a software developer I want strong abstractions without bloat. LLMs are so successful in part because they are a really strong abstraction. You feed in text and you get back text. Depending on the model and other parameters your results may be better or worse, but changing from eg. Claude to ChatGPT is as simple as swapping out one request with another. If what I want is to run AI tasks, then GPUs are a poor abstraction. It's very complicated (as Fly have discovered) to share them securely. The amount of GPU you need could vary dramatically. You need to worry about drivers. You need to worry about all kinds of things. There is very little bloat to the ChatGPT-style abstraction, because the network overhead is a negligable part of the overall cost. If I say I don't want magic, what I really mean is that I don't trust the strength of the abstraction that is being offered. For example, when a distributed SQL database claims to be PostgreSQL compatible, it might just mean it's wire compatible, so none of my existing queries will actually work. It might have all the same functions but be missing support for stored procedures. The transaction isolation might be a lie. It's not that these databases are bad, it's that "PostgreSQL as a whole" cannot serve as a strong abstraction boundary - the API surface is simply too large and complex, and too many implementation details are exposed. It's the same reason people like containers: running your application on an existing system is a very poor abstraction. The API surface of a modern linux distro is huge, and includes everything from what libraries come pre-installed to the file-system layout. On the other hand the kernel API is (in comparison) small and stable, and so you can swap out either side without too much fear. K8S can be a very good abstraction if you deploy a lot of services to multiple VMs and need a lot of control over how they are scaled up and down. If you're deploying a single container to a VM, it's massively bloated. TLDR: Abstractions can be good and bad, both inherently, and depending on your use-case. Make the right choice based on your needs. Fly are probably correct that their GPU offering is a bad abstraction for many of their customer's needs.

nicoburns 2y ago

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

sanderjd 2y ago
Really? There are plenty of valid criticisms of kubernetes, but this doesn't strike me as one of them. It gives you tons of control over all of this. That's a big part of why it's so complex!
- jldugger 2y ago
  IMO, it's rather hard to fully know all of kubernetes and what it's doing, and the kind of person who demands elegance in solutions will hate it.
- sgarland 2y ago
  It gives you control via abstractions. That’s fine, and I like K8s personally, but if you don’t understand the underlying fundamentals that it’s controlling, you don’t understand what it’s doing.
- PaulHoule 2y ago
  This mainframe system from the 1990s was so much simpler https://www.ibm.com/docs/en/cics-ts/6.x?topic=sysplex-parallel-principles https://www.ibm.com/docs/en/cics-ts/6.x?topic=sysplex-parall... even if it wasn't as scalable as Kube. One the other hand, a cluster of 32 CMOS mainframe could handle any commercial computing job that people were doing in the 1990s.
  - sanderjd 2y ago
    It's not simple but it's not opaque.
  - 2y ago

chrismarlow9 2y ago

I call this difference being a developer who is on call vs being a developer who is not on call

brightball 2y ago

I am the former. I also make cost benefit based decisions that involve time. Unless I have very specific configuration needs, the git push option lets me focus on what my users care about and gives me one less thing that I need to spend my time on. Increasingly, Fly even lets you dip into most complex configurations too. I’ve got no issue with using Tofu and Ansible to manage my own infrastructure but it takes time to get it right and it’s typically not worth the investment early on in the lifecycle.

teaearlgraycold 2y ago

It depends on the product you're building. At my last job we hosted bespoke controlnet-guided diffusion models. That means k8s+GPUs was a necessity. But I would have loved to use something simpler than k8s.

deleted 2y ago

[deleted]

danstewart_ 2y ago

I don’t think this comment does justice to fly.io. They have incredible defaults that can make it as simple as just running ‘git push’ but there isn’t really any magic happening, it’s all documented and configurable.

dirtbag__dad 2y ago

> they're willing to spend a lot of (usually their employer's) money May just be my naïveté, but I thought that something like ECS or EKS is much cheaper than an in-house k8 engineer.

sgarland 2y ago
If you don’t have staff to do that, you probably aren’t at the scale when you need them, and you’re needlessly adding complexity. It’s always baffling to me why people think that ECS or god forbid EKS is somehow easier than a few Linux boxes.
- mdaniel 2y ago
  Because the force multiplier of a good DX way outweighs the occasional nonsense from having to do k8s upgrades or troubleshooting For example: how do you roll out a new release of your product? In sane setups, it's often $(helm upgrade --install ...), which is itself often run either in-cluster by watching a git managed descriptor, or in CI on merge to a release branch/tag How does your developer get logs? Maybe it's via Splunk/ELK/DataDog/whatever but I have never in my life seen a case where that's a replacement for viewing the logs How do you jump into the execution environment for your workload, to do more advanced debugging? I'm sure you're going to say ssh, which leads to the next questions of "how do you audit what was done, to prevent config drift" followed by "how do you authenticate the right developer at the right time with access to the right machine without putting root's public key file in a spreadsheet somewhere"
  - sethammons 2y ago
    > Splunk/ELK/DataDog/whatever but I have never in my life seen a case where that's a replacement for viewing the logs Uh, any time I run a distributed system and logs could appear on n nodes I need a log aggregator or I am tailing in n terminals. I almost only use Splunk. I tail logs in dev. Prod needs an aggregator. This has been my experience at 4 of my last 6 companies. The shit companies who had all the issues? Logs on cloudwatch or only on the node

Glyptodon 2y ago

I think it's true that engineers who want to understand every layer of everything in depth, or who want to have platform ownership, are not necessarily the same group as the more "product itself" focused sort who want to write something and just push it, I don't actually think I'm sold at all that any of these groups, in a vacuum, have substantial demand for GPU compute unless that's someone's area of interest for a pet project.

flufluflufluffy 2y ago
This. Personally, I’d want a GPU to self host whatever model, because I think that’s fun, plain and simple. Probably many people do too. But the business is not making money from people who are just thinking about fun.

the__alchemist 2y ago

Where does this dichotomy between Kubernetes, and superficial understanding come from? It is not consistent with my experience, and I don't have speculation its origin.

aaron695 2y ago

[dead]

startupsfail 2y ago

The view that developers just want LLMs is plain wrong. The age of AI is just starting.

jsemrau 2y ago

>who don't like "magic" and want to understand where their code is running and what it's doing. I just made this point in a post on my substack. Especially in regulated industries, you NEED to the able to explain your AI to the regulator. You can't have a situation where a human say "Well, gee I don't know. The AI told me to do it."

chii 2y ago

> They don't want to have to understand DNS, linux, or anything else beyond whatever framework they are using. tell me whether there's many brick layers who wants to understand the chemical composition of their bricks.

steve_adams_86 2y ago
I’ve never laid bricks but in other trades I’ve worked in, well, a lot of people understood basics of the chemistry of the products we used. It’s useful to understand how they work together safely, if they can be exposed to different environments, if they’re heat-safe, cold-safe, do they off-gas, etc. Paints, wood finishes, adhesives, oils, abrasives, you name it. You generally know at least a bit about what’s in it. I can’t say everyone I’ve worked with wanted to know, but it’s often intrinsic to what you’re doing and why. You don’t just pull a random product off a shelf and use it. You choose it, quite often, because of its chemical composition. I suspect it’s not always thought of this way, though. This is the same with a lot of artistic mediums as well. Ceramicists often know a lot more than you’d expect about what’s in their clay and glazes. It’s really cool. I’m not trying to be contrarian here. I know some people don’t care at all, and some people use products because it’s what they were told to do and they just go with it. But that wasn’t my experience most of the time. Maybe I got lucky, haha.
anthk 2y ago
In my country I think the vocational/trade degree -which might lie between HS and uni level- on car mechanics has basic physics, mechanics and maybe some chemistry. Ditto for the rest of technical voc degrees. If you think you can do IT without at least a trade degree on understanding the low level components interact, (and I'm not talking about CS level, concurrency with CSP, O-notation, linear+discrete algebra... but basic stuff such as networking protocols, basic SQL database normalizations, system administration, configuration, how the OS boots, how processes work -idle, active, waiting..., if you don't get that, you will be fired faster than anyone around.

lakomen 2y ago

[dead]

motorest 2y ago

> I'm increasingly coming to the view that there is a big split among "software developers" and AI is exacerbating it. I don't think this split exists, at least in the way you framed it. What does exist is workload, and problems that engineers are tasked with fixing. If you are tasked with fixing a problem or implementing a feature, you are not tasked with learning all the minute details or specifics of a technology. You are tasked with getting shit done, which might even turn out to not involve said technology. You are paid to be a problem-solver, not an academic expert on a specific module. What you tried to describe as "magic" is actually the balance between broad knowledge vs specialization, or being a generalist vs specialist. The bulk of the problems that your average engineer faces requires generalists, not specialists. Moreover, the tasks that actually require a specialist are rare, and when those surface the question is always whether it's worth to invest in a specialist. There are diminished returns on investment, and throwing a generalist at the problem will already get some results. You give a generalist access to a LLM and he'll cut down on the research time to deliver something close to what a specialist would deliver. So why bother? With this in mind, I would go as far as to frame a scenario backhandedly described as "want to understand where their code is running and what it's doing" (as if no engineer needs to have insight on how things work?) as opposed to the dismissive "just wants to `git push` and be done with it" scenario, can actually be classified as a form of incompetence. You,as an engineer, only have so many hours per day. Your day-to-day activities involve pushing new features and fixing new problems. To be effective, your main skillet is learn the system in a JIT way, dive in, fix it, and move on. You care about system traits, not low-level implementation details that may change tomorrow on a technology you may not even use tomorrow. If, instead, you feel the need to waste time on topics that are irrelevant to address the immediate needs of your role, you are failing to deliver value. I mean, if you frame yourself as a Kubernetes expert who even know commit hashes by heart, does that matter if someone asks you, say, why is a popup box showing off-center?

skrebbel 2y ago

"Enjoys doing linux sysadmin" is not the same as "Wants to understand how things work". It's weird to me that you group those two kinds of people in one bucket.

deleted 2y ago

[deleted]

pinoy420 2y ago

I was thinking about this just yesterday. I was advertised a device for an aircraft to geo-assist taxiing? (I have never flown so I don’t know why). The comments were the usual “old man shouts at cloud” angry that assistive devices make lives easier for people. I feel this is similar to what you are pointing out. Why _shouldn’t_ people be the “magic” users. When was the last time one of your average devs looked in to how esm loading? Or the python interpreter or v8? Or how it communicates with the OS and lower level hardware interfacing? This is the same thing. Only you are goalpost shifting.

andai 2y ago

I'd be in the latter group if my budget were infinite. Alas!

zwnow 2y ago

Also "They want LLMs" lol. I cant remember being asked. I dont want to use AI for coding.

black3r 2y ago

All professional developers want two things: Do their work as fast as possible and spend as little budget to make things work. That's the core operating principle of most companies. What's changing is that managed solutions are becoming increasingly easier to set up and increasingly cheaper on smaller scales. While I do personally enjoy understanding the entire stack, I can't justify self-hosting and managing an LLM until we run so many prompts a day that it becomes cheaper for us to run our own GPUs compared to just running APIs like OpenAI/Anthropic/Deepseek/...

_heimdall 2y ago

I agree that split exists, and that the former is more rare, but in my experience the split is less about avoid magic and more about keeping control of your system. Many, likely most, developers today don't care about controlling their system/network/hardware. There's nothing wrong with that necessarily, but it is a pretty fundamental difference. One concern I've had with building LLM features is whether my customers would be okay with me giving their data over to the LLM vendor. Say I'm building a tool for data analysis, is it really okay to a customer for me to give their table schemas or access to the data itself to OpenAI, for example? I rarely hear that concern raised though. Similarly when I was doing consulting recently, I wouldn't use copilot on client projects as I didn't want copilot servers accessing code that I don't actually own the rights to. Maybe its over protective though, I have never heard anyone raise that concern so maybe its just me.

chasd00 2y ago
I work for a major consulting firm and we’ve been threatened with fire and brimstone if any part of client info (code, docs, random email, anything) ever gets sent to an LLM. Even with permission from the client our attack lawyers prefer us not to use them. It’s a very sensitive topic. I still use LLMs from time to time but always starting with a blank prompt and the ask anonymized. (Heh I’m probably not even supposed to do that)

yungporko 2y ago

increasingly small is right. i'm definitely part of that former group but sadly more and more these days i just feel dumb for being this way. it usually just means that i'm less productive than my colleagues in practice as i'm spending time figuring out how things work while everybody else is pushing commits. maybe if we were put in a hypothetical locked room with no internet access i'd have a slightly easier time than them but that's not helpful to anybody. once upon a time i could have said that it's better this way and that everybody will be thankful when i'm the only person who can fix something, but at this point that isn't really true when anybody can just get an LLM to walk them through it if they need to understand what's going on under the hood. really i'm just a nerd and i need to understand if i want to sleep at night lol.

killerstorm 2y ago

I don't think you got this split right. I prefer to either manage software directly with no wrappers on top, or use a fully automated solution. K8S is something I'd rather avoid. Do you enjoy writing configuration for your automation layer?

IlikeKitties 2y ago

I don't think this is entirely correct. I'm working for a company that does IT Consulting and so I see many Teams working on many different Projects and one thing I have learned the hard way is that Companies and Teams that think they should do it all themselves are usually smaller companies and they often have a lot of Problems with that attitude. Just an example I recently came across: Working for a smaller company that uses Kubernetes and manages everything themselves with a small team. The result: They get hacked regularly and everything they run is constantly out of date because they don't have the capacity to actually manage it themselves. And it's not even cheaper in the long run because Developer Time is usually more expensive than just paying AWS to keep their EKS up to date. To be fair, in my home lab I also run everything bare metal and keep it updated but I run everything behind a VPN connection and run a security scanner every weekend that automatically kills any service it finds > Medium Level CVE and I fix it when I get the time to do it. As a small Team I can only fix so much and keep so much up to date before I get overwhelmed or the next customer Project gets forced upon me by Management with Priority 0, who cares about security updates. I'd strongly suggest to use as much managed service as you can and focus your effort as a team on what makes your Software Unique. Do you really need to hire 2-3 DevOps guys just to keep everything running when GCP Cloud Run "just werks"? Everything we do these days runs on so many levels of abstraction anyway, it's no shame to share cost of managing the lower levels of abstraction with others (using managed Service) and focus on your product instead. Unless you are large enough to pay for whole teams that deal with nothing but infrastructure to enable other teams to do Application Level Programming you are, in my limited experience, just going to shoot yourself in the foot. And again, just to emphasize it: I like to do everything myself because for privacy reasons I use as little services that aren't under my control as possible but I would not recommend this to a customer because it's neither economical nor does it work well in my, albeit limited, experience.

memhole 2y ago

I might be an outlier. I like to think I try for a deeper understanding of what I’m using. Like, fly uses firecracker vms afaik. Sometimes, especially for quick projects or testing ideas I just want to have it work without wrangling a bunch of AWS services. I’m typically evaluating is this the right tool or service and what is the price to convenience? For anything potentially long term, what’s the amount of lock in when or if I want to change providers?

ripped_britches 2y ago

The way I think about it is this: any individual engineer (or any individual team) has a limited complexity budget (in other words, how much can you fit in your meat brain). How you spend it is a strategic decision. Depending on your project, you may not want to waste it on infra so you can fit a lot of business logic complexity.

tarsinge 2y ago

It’s not about wanting, it’s about what the job asks for. As a self employed engineer I am paid to solve business problems in an efficient way. Most of the time it just make more business sense for the client and for me to pay to just have to git push if there is no performance challenges needing custom infrastructure.

EarthIsHome 2y ago

> The other group (increasingly large) just wants to `git push` and be done with it, and they're willing to spend a lot of (usually their employer's) money to have that experience. They don't want to have to understand DNS, linux, or anything else beyond whatever framework they are using. lol, even understanding git is hard for them. Increasingly, software engineers don't want to learn their craft.

chasd00 2y ago
I think the root of it is most people coming into the software engineering industry just want a good paying job. They don’t have any real interest in computers or networks or anything else. Whatever keeps the direct deposits coming is what they’ll do. And in their defense, the web dev industry is so large in breadth and depth and the pay/benefits are so generous it’s an attractive career path no matter what your passion is.

vagrantJin 2y ago

> I'm increasingly coming to the view that there is a big split among "software developers" and AI is exacerbating it This is admittedly low effort but the vast majority of devs are paid wages to "write CRUD, git push and magic" their way to the end of the month. The company does not afford them the time and privilege of sitting down and analyzing the code with a fine comb. An abstraction that works is good enough. The seasoned seniors get paid much more and afforded leeway to care about what is happening in the stack, since they are largely responsible for keeping things running. I'm just pointing out it might merely be a function of economics.

sudhirj 2y ago

This is context based dichotomy, not a person-based one. In my personal life, I’m curiosity-oriented, so I put my blog, side projects and mom’s chocolate shop on fully self hosted VPSs. At my job managing a team of 25 and servicing thousands of customers for millions in revenue, I’m very results-oriented. Anyone who tries to put a single line of code outside of a managed AWS service is going to be in a lot of trouble with me. In a results-oriented environment, I’m outsourcing a lot of devops work to AWS, and choosing to pay a premium because I need to use the people I hire to work on customer problems. Trying to conflate the two orientations with mindsets / personality / experience levels is inaccurate. It’s all about context.

antihero 2y ago

The former have the mentality of being independent, at the cost of their ability to produce a result as quickly. The latter are happy to be dependent, because the result is more important than the means. Obviously this is a spectrum.

hassleblad23 2y ago

You lost me at "Kubernetes".

seanp2k2 2y ago

Having worked with many of the latter and having had the displeasure of educating them on nix systems fundamentals: ugh, oof, I hate this timeline, yet I also feel a sense of job security. We used to joke about this a lot when Java devs would have memory issues and not know how to adjust the heap size in init scripts. So many “CS majors” who are completely oblivious to anything happening outside of the JVM, and plenty happening within it.

lovich 2y ago

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. That problem started so long ago and has gotten so bad that I would be hard pressed to believe there is anyone on the planet who could take a modern consumer pc and explain what exactly is going on the machine without relying on any abstractions to understand the actual physical process. Given that, it’s only a matter of personal preference on where you draw the line for magic. As other commenters have pointed out, your line allowing for Kubernetes is already surprising to a lot of people

ianmcnaney 2y ago

Somebody who doesn’t want to understand DNS, Linux, or anything beyond their framework is a hazard. They’re not able to do a competent code review on the vomit that LLMs produce. (Am I biased much?)

khana 2y ago

[dead]

westurner 2y ago

IaaS or PaaS? Who owns and depreciates the logs, backups, GPUs, and the database(s)? K8s docs > Scheduling GPUs: https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/ https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus... : > Once you have installed the plugin, your cluster exposes a custom schedulable resource such as amd.com/gpu or nvidia.com/gpu. > You can consume these GPUs from your containers by requesting the custom GPU resource, the same way you request cpu or memory awesome-local-ai: Platforms / full solutions https://github.com/janhq/awesome-local-ai?platforms--full-solutions https://github.com/janhq/awesome-local-ai?platforms--full-so... But what about TPUs (Tensor Processing Units) and QPUs (Quantum Processing Units)? Quantum backends: https://github.com/tequilahub/tequila#quantum-backends https://github.com/tequilahub/tequila#quantum-backends Kubernetes Device Plugin examples: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#examples https://kubernetes.io/docs/concepts/extend-kubernetes/comput... Kubernetes Generic Device Plugin: https://github.com/squat/generic-device-plugin#kubernetes-generic-device-plugin https://github.com/squat/generic-device-plugin#kubernetes-ge... K8s GPU Operator: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html#pod-security-context-of-the-operator-and-operands https://docs.nvidia.com/datacenter/cloud-native/gpu-operator... Re: sunlight server and moonlight for 120 FPS 4K HDR access to GPU output over the Internet: https://github.com/kasmtech/KasmVNC/issues/305#issuecomment-2600828249 https://github.com/kasmtech/KasmVNC/issues/305#issuecomment-... : > Still hoping for SR-IOV in retail GPUs. > Not sure about vCPU functionality in GPUs Process isolation on vCPUs with or without SR-IOV is probably not as advanced as secure enclave approaches. Intel SGX is a secure enclave capability, which is cancelled on everything but Xeon. FWIU there is no SGX for timeshared GPUs. What executable loader reverifies the loaded executable in RAM after imit time ? What LLM loader reverifies the in-RAM model? Can Merkle hashes reduce that cost; of nn state verification? Can it be proven that a [chat AI] model hosted by someone else is what is claimed; that it's truly a response from "model abc v2025.02"? PaaS or IaaS

jrochkind1 2y ago

Eh, the way I see it the entire practice of computer science and software engineering is built on abstraction -- which can be described as the the ability to not have to understand lower levels -- to only have to understand the API and not the implementations of the lowest levels you are concerned with, and to have to pay even less attention to lower levels than that. I want to understand every possible detail about my framework and language and libraries. Like I think I understand more than many do, and I want to understand more, and find it fulfilling to learn more. I don't, it's true, care to understand the implementation details of, say, the OS. I want to know the affordances it offers me and the APIs that matter to me, I don't care about how it's implemented. I don't care to understand more about DNS than I need. I definitely don't care to spend my time futzing with kubernetes -- I see it as a tool, and if I can use a different tool (say heroku or fly.io) that lets me not have to learn as much -- so I have more time to learn every possible detail of my language and framework, so I can do what I really came to do, develop solutions as efficiently and maintainably as possible. You are apparently interested in lower levels of abstraction than I am. Which is fine! Perhaps you do ops/systems/sre and don't deal with the higher levels of abstraction as much as I do -- that is definitely lucrative these days, there are plenty of positions like that. Perhaps you deal with more levels of abstraction but don't go as deep as me -- or, and I totally know it's possible, you just have more brain space to go as deep or deeper on more levels of abstraction as me. But even you probably don't get into the implementation details of electrical engineering and CPU design? Or if you do, and also go deep on frameworks and languages, I think you belong to a very very small category! But I also know developers who, to me, dont' want to go to deep on any of the levels of abstraction. I admit I look down on them, as I think you do too, they seem like copy-paste coders who will never be as good at developing efficient maintainable soltuions. I started this post saying I think that's a different axis than what layers of abstraction one specializes in or how far down one wants to know the details. But as I get here, while I still think that's likely, I'm willing to consider that these developers I have not been respecting -- are just going really deep in even higher levels of abstraction than me? Some of them maybe, but honestly I don't think most of them, but I could be wrong!

hk1337 2y ago

The latter group sounds like they're more managers than software developers.

draw_down 2y ago

[dead]

samstave 2y ago

Wait until LLMS are deployed in edge FPGAs

garfieldnate 2y ago

It's been a while since I tried, but my experience trying to manually set up GPUs was atrocious, and with investigation generally ending at the closed-source NVidia drivers it's easy to feel disempowered pretty quickly. I think my biggest learning from trying to do DL on a manually set up computer was simply that GPU setup was awful and I never wanted to deal with it. It's not that I don't want to understand it, but with NVidia software you're essentially not allowed to understand it. If open source drivers or open GPU hardware were released, I would gladly learn how that works.