52 ms·
The decline of hardware knowledge in the era of cloud native compute
- mikerg87 3y agoIn other words, The cloud has no soul. Just as devs were partitioned into front and backend and full stack. Admins are going through a similar transformation.
- Almondsetat 3y agoWhen the industry will cry for it, universities will magically start offering (and heavily incentivize) Computer Engineering degrees instead of Computer Science ones
- orev 3y agoMost (good) computer people don’t learn about computers in university, they learn at home through tinkering. When everything comes in a container and recommends using AWS, there’s no need to bang your head on the wall trying to figure something out, which is what actually leads to learning and tinkering with hardware at home. A few bright spots, however, would be gaming (the constant need to pay attention to new GPUs), and smart home/arduino type stuff.
- spacecadet 3y agoSupport for home labs! Keep learning everyone.
- sgarland 3y agoFully agree. I love my rack, and getting to practice things in the safety of an isolated environment. Also, anecdotally, the best SREs I’ve worked with have had homelabs. I maintain it’s a good signal.
- lizknope 3y agoI feel like most of the stuff the author is complaining about is taught in computer engineering and not computer science. Computer engineering curriculums usually focus on lower level subjects and computer science on higher level abstraction. If the author wants people with the lower level skills then they should filter resumes based on the degree. But then they will probably be missing higher level skills. In the end there is not enough time to learn everything in 4 years. If you add a new class then an old class has to be dropped or modified. I'm in integrated circuit / semiconductor design and we aren't going to even look at someone with only a computer science degree. We look at people with electrical and computer engineering degrees. Even a bachelors degree is not really enough anymore. 90% of the new grads we hire have a masters degree and there is still a ton for them to learn on the job.
- tdeck 3y agoThe thing is, what the original article actually describes as "hardware knowledge" in their examples doesn't even touch on electrical engineering. At most it's stuff you'd expect to learn in an operating systems class, which many CS graduates will take. OP isn't close to asking "wtf is flip flop?" it's more like "wtf is select()?".
- lizknope 3y agoAll of which makes me wonder how the author is interviewing people. I've been working over 25 years and interviewed at least 100 people at all levels of experience. The hardest to interview are college students because they have so little real world experience. But that doesn't really matter. If the author thinks people should have 8-bit computer experience then ask the question during the interview. If the author wants to ask about operating system topics then ask those during the interview. I've worked with some interns and new grads that didn't have the skills I thought they should. So I added those things to my list of interview questions.
- computersuck 3y agoHas nothing to do with 8-bit. Deep OS level knowledge can and should be taught starting from 64 bit (where most OSs run) and maybe tiny speck of 16 bit because most BIOS still runs on 16 bit intel It's about exposure to syscalls, C, assembly, lower level debugging and just making sure people aren't afraid to touch that layer. Same goes with other foundational knowledge like packets and networking protocols
- klelatti 3y agoAre you suggesting that 'Deep OS knowledge', syscalls, C, assembly, packets and networking protocols should be taught in schools? What I think the author is getting at with '8-bit' is that presenting children with a complete machine that is simple enough to understand from top to bottom gives a foundation that they can build on when they encounter more complex systems later on.
- adrian_b 3y agoThe 32-bit ARMv7-M micro-controllers with Cortex-M4 or Cortex-M7 cores that are available for instance on $10 development boards from ST or from many other companies are conceptually simpler and easier to understand when programmed at the bare-metal level than the 8-bit computers of 40 to 50 years ago. They are a much better target for teaching children about hardware.
- FirmwareBurner 3y agoHard disagree. 8 bit micros are far simpler to understand from every perspective and depth than the 32 bit micros. One of the main benefits of teaching on 8 bit micros is they function like basic HW Harvard state machines, meaning their code execution is easily predictable and explainable form the diagrams on paper/whitrebaord, to the disassembly in the debugger in practice, as every CPU instruction takes exactly one clock cycle, there's no memory caching, no PLL and various multipliers and dividers for the 50 on-chip clock domains and IRQ tables, but they have on single clock from an external oscillator from which the CPU, RAM and FLASH memory, and all peripherals run on only that clock meaning it's easier to draw clock/signal diagrams to show "on this rising edge the CPU fetches the instruction, on the next clock edge, the ADC triggers the sampling, etc" Secondly, 8 bit micros come in DIP packages easier to breadboard and are 5V tolerant meaning you can just hook them up to USB, and most of the times they have high current pins with Schottky diodes meaning you can have them rectify AC voltage or direct drive LEDs without transistors. Thirdly, IMHO, 8bit ASM disassembly, like the Atmega AVR, is much easier to follow and understand in the debugger for beginners than the 32 bit ARM due to lower instruction count and simpler less powerful instructions that are always 8 bit, and less features that can confuse you like configurable IRQ tables or powerful instructions that cand do two operations at once per clock cycle for extra performance or optimized instructions that are 16 bit instead of 32 bit to save code space like the ARM Thumb. 8bit micros like the old Arduinos are far ore capable for learning and hobbyist tinkering than people give them credit for. People just needlessly fixate against them because "32 bit is bigger than 8 bit". My $0.02.
- madaxe_again 3y agoHonestly, maybe a majority of modern engineers not being familiar with anything but the pointy tip of the stack is a good thing. I come very much from the old world - I learned to code on graph paper, as that was how you saved your work, and being able to wield a scope and iron was kinda mandatory for being able to meaningfully use a computer. As tech grew up, so did I - and while it’s useful to be able to understand what is happening from clicking a button in a gui down to electrons tunnelling through a bandgap, particularly when it comes to debugging the truly arcane, I actually find that the level of abstraction I carry around in my head sometimes gets in the way. I look at newer techies bouncing around with their lofty pyramids of containerised cloud based abstracted infrastructures, and I almost envy the superficiality of their engagement - I can’t help but look at it and see immense complexity, because I see behind the simple and intuitive userland, and that makes me run for vim and some nice bare metal where I know what my hardware is doing. Maybe I’m just getting old.
- blueflow 3y agoYeah, it might look nice and easy. But this is when it works. When it doesn't they are unable to understand many kind of problems and have to resort and try&error.
- zvmaz 3y ago> I can’t help but look at it and see immense complexity, because I see behind the simple and intuitive userland, and that makes me run for vim and some nice bare metal where I know what my hardware is doing. But is it humanly possible to know details in every level of abstraction? There should be a balance between abstractions and knowing some details beneath some layers of abstraction. What do you think?
- spacecadet 3y agoWhy not? It's really not that many principles- to me, it is entities that add complexity by deviating standards or introducing esoteric nomenclature on-top of existing concepts.
- sgarland 3y ago
- deleted 3y ago[deleted]
- ljasdfwoy 3y agoI was only asked lc-medium/hard for my platform position. You really don't need to know anything anymore.
- sgarland 3y agoI had two markedly different experiences interviewing for DBRE roles recently. One was at a well-known SaaS “here’s a spreadsheet we call a database” company. I knew way more about Python than the interviewers, which became a problem because I used its stdlib to blow through the tasks they had. I then had a second interview, where they effectively asked me to code DFS. I am terrible with LC. I explained DFS perfectly, but struggled to turn it into code. The interviewer even commented that it had nothing to do with the position (you don’t say). The other was at a quant trading firm. I had multiple interviews, ranging from “can you use a programming language to accomplish tasks” (again, Python and its stdlib was quite helpful, and wasn’t met with pushback), to DB-specific questions, to Linux internals. All of it felt extremely on-task, and had they not required me to relocate for a remote position (?!) I would’ve taken the job in a heartbeat.
- gumby 3y agoI remember being shocked when I saw someone (on HN) use the phrase “running on the bare iron” to mean “in user space of a computer running a multitasking operating system”. They in turn were incredulous that it could mean anything else — apparently an OS is needed, right? Still, in some ways it’s a good thing. People can write good code without knowing what an ALU is, much less a flip flop. The bigger problem is that there is little incentive these days to write good code at all; those who do of necessity know more than just their tiny domain.
- bowsamic 3y agoIn my experience it is now such that even just writing and running a C++ program on a modern computer is considered "extremely low level". It's almost an extreme thing to do to decide to write a native program.
- gumby 3y agoHaving written crt0 and for that matter hardware bring up for a couple of chips, that made me laugh. But I haven’t tried to design a (extremely primitive) chip since I was in college, so there’s plenty for others to laugh at me for too.
- corethree 3y agoIts a domain thing. And one domain in the world is expanding. Everyone in the world is becoming a web developer. Actually it already happened. Throw a dart at a group of software developers most likely that dart will hit a web developer. Web developers can write shitty code because most of the time the bottleneck is in the database. You just need to write code thats faster then the database and you don't need to optimize beyond that. Now all the optimizations center around pointless refactorings to keep up with the newest technology trend.
- KaushikR2 3y agoOptimization has taken a back-seat now, I'd say. We focus more on developer productivity, and that extra mile optimization isn't deemed necessary if that effort and time can be put toward another project, squeezing out more value from the developer.
- nickdothutton 3y agoI had to explain the point of Finesse OS (used in the original NTI PIX firewalls[1]) to someone technical the other day, within the context of a single-purpose (in fact single service) OS concept. “But it was Linux right?” Was a cognitive hurdle it took several attempts to clear. [1] https://en.wikipedia.org/wiki/Cisco_PIX https://en.wikipedia.org/wiki/Cisco_PIX
- wslh 3y agoI really don't know how bad all of this is, but it's surprising that graduate engineers don't know how to configure a basic network without DHCP. On one hand, you can have a router at home from your ISP, and everything seems to work fine. On the other hand, there are a lot of existing misconfigurations and security issues with these routers, which ISPs used to update their firmware automatically.
- Havoc 3y agoIs it really needed though? If people can set up and operate a basic Debian server and a bit of networking that I suspect is more than enough for a cloud where anything lower is concealed anyway
- Spivak 3y agoI would love to see at minimum a basic "here's how datacenters are designed" even if the only use is to see how "the cloud" just maps those concepts to APIs. You'll imminently understand why AWS networking and EC2 have all the different concepts they have.
- Havoc 3y agoAgreed - Data center design would make a lot more sense than fk around with 8 bit stuff
- ddkto 3y agoSimon Wardley would like a word…in his model this is the natural order of things. As technology matures and standardized and a new generation of tools is built on top of new abstractions, and the details of that tech no longer need to be understood in order to use it. Subjects and skills that were requisite basics a generation* ago, become advanced, under the hood topics for specialists. The next generation of people need different skills in the day to day. This post is a great account of what that feels like from the inside, from the perspective of the newer generation learning these (now) ‘advanced’ topics. (Funnily enough, I don’t (yet) see anyone commenting "real men write assembler" - a skill that has long ago moved from required by all developers to super-specialized and not particularly useful to most people.) *I am using the word generation in the broadest sense as it relates to cycles of technology
- dahfizz 3y agoWhether or not this state of affairs is "natural", I do not think it is "good". Civil engineers still need to understand calculus and how to analyze structural integrity even though they can rely on modern computer modeling to do the heavy lifting. All engineers are expected to have some requisite level of knowledge and skill. Only in software do we accept engineers having the absolute bare minimum knowledge and skill to complete their specific job. Not that we shouldn't use modern tools, but having a generation of developers unable to do anything outside their chosen layer of abstraction is a sad state of affairs.
- galleywest200 3y ago> All engineers are expected to have some requisite level of knowledge and skill. Only in software do we accept engineers having the absolute bare minimum knowledge and skill to complete their specific job. Most software engineers just produce websites and nothing that impacts the safety of other humans. Other types of engineers have to ensure people do not die.
- bognition 3y agoWhere do you draw the line? Should a civil engineer also be an expert on material science? Likewise how much must a software engineer understand about how hardware works before they able to do a good job? At some point in time there is diminishing returns for someone to be truly “full stack”.
- ramoz 3y agoI mean it depends on your workloads. Any engineer faced with deploying AI systems at scale will inevitably break through abstractions (or fail). AI is further proof that our systems aren’t simply getting easier to build around.
- mrob 3y agoI'm not convinced that 8-bit computers are the ideal starting point for education. My reasoning is that Dijkstra had a point when he wrote: "It is practically impossible to teach good programming to students that have had a prior exposure to BASIC: as potential programmers they are mentally mutilated beyond hope of regeneration." This is of course exaggeration, and many great programmers did in fact start with BASIC, but more importantly, Dijkstra wrote this in 1975, when BASIC really did have some glaring flaws. '10 PRINT "HELLO WORLD"\n 20 GOTO 10' has appealing immediacy, but it sets students down the path to spaghetti code. A modern structured BASIC, or perhaps MicroPython, is a much better starting point. And that's problematic on an 8-bit machine. IMO, an educational computer should have a flat address space without paging. Paging isn't relevant to modern computers; it complicates things without teaching anything valuable. The natural address bus for an 8-bit machine is 16 bits wide, and that limits you to only 64KiB address space without paging. For simplicity, the frame buffer should live in this address space too, so you don't have much left for a modern programming language (ideally stored in ROM in the same address space), especially if you want things like user-friendly error reporting and built-in documentation, which you probably do (think of QBASIC). Therefore, I think a 16-bit computer is better. I'd prefer something like a 16-bit version of Ben Eater's breadboard CPU[0], using SMD components on a PCB to allow for speeds fast enough for interactive graphics. All chips should also be available in DIP for easy breadboard experimentation too. It should support fully static operation, with a pause switch and a single-step clock button, indicator LEDs for all the registers, and test points everywhere you might want to inspect with an oscilloscope. It should have a simple GPU that just converts a 320x240 1-bit monochrome frame buffer to VGA, with memory access interleaved with the CPU. The CPU can poll for vblank. Support for PS/2 or similar serial keyboard input, with a FIFO buffer so you don't lose keystrokes. A very primitive distinction between user space and kernel space, where some RAM is read only unless the program counter is in ROM. Although I generally like the PEEK/POKE anywhere freedom of the old 8-bit machines, I want this feature so it can support a memory monitor and disassembler in ROM, which is jumped to by triggering the single non-maskable interrupt. The monitor can copy all the registers and framebuffer to the kernel RAM on entry and and restore them on exit, so it's always possible to inspect things or debug any problems you might run into and continue afterwards. Data storage could be via a simple audio interface, like the cassette tape data storage on old home microcomputers. Advances in both software and audio recording hardware should make this substantially faster and more reliable than it was back them. This computer would have maximum observeability and ease of understanding, while still being advanced enough to be user friendly and interesting. It should even be powerful enough for games, which will provide some extra motivation. [0] https://www.youtube.com/playlist?list=PLowKtXNTBypGqImE405J2565dvjafglHU https://www.youtube.com/playlist?list=PLowKtXNTBypGqImE405J2...
- steelframe 3y agoWhen I first started university I had already been tinkering with computers and programming for about 10 years. I started in Computer Science, where I took a Java class from Urs Hölzle, who later went on to be Google's first VP of engineering. It was an amazing class, and I almost scored a 100% on every test, except for the one where I accidentally wrote "<=" as a less-than sign with a line under it and got dinged a point. However at the end of my first year I felt profoundly unsatisfied, like I was just learning superficial tricks to get computers to do what you wanted, but I didn't feel like I really understood what was actually happening. I switched schools and majors to a Computer Engineering course taught primarily by EE professors, hoping to learn about the lower-level stuff so I didn't feel so ignorant. I learned about logic gates, K-maps, BJT transistor characteristics, N/P-well doping, VLSI, and so forth. All was going swimmingly for me until it came time to take a physics course on crystal structures. At that point I realized that the rabbit hole goes very, very deep -- much deeper than I could ever hope to really fully "understand" from the quantum level all the way up to Java code. Recognizing that I only had so much time and mental capacity to learn about the whole system, I had to come to peace with knowing that I would have to choose a layer in which to increase my expertise and then largely stick to that. For me that started out being architecture-level operating system coding in C and some assembly, but I popped up the stack a smidge to then get to C and applied cryptography. I'm now one of the few people at my company that (IMO) understands operating systems at all. The organization is deathly afraid to do anything with the kernel that's being deployed in their Cloud instances, and most view it as some mystical "black magic" that just works if you don't touch it. Hardly anybody fully understands what a so-called "container" actually is, but they sure know all the API calls to "make the container work" and push container images around repositories. They're often surprised when things that I know will be issues with their containers ahead of time happen to them. Whenever they run into problems, they'll often open two dozen Stack Overflow tabs and try copying and pasting stuff until something seems to work. People approach me as some sort of mystical oracle who can bestow arcane knowledge of syscalls and what not when their Stack Overflow searches turn up dry. I feel like the pendulum perhaps has swung too far into the layers of abstraction in our universities, but I'm not sure what to do about that. I wonder what will happen as people like me ride off into the sunset.
- rochak 3y ago
- anthomtb 3y agoBelieve it or not, there are these things called books which contain all sorts of knowledge about things like low level C, system calls, assembly, even digital logic. I know because I have several of them sitting on my office shelf thanks to an EE oriented education. I could have picked up the same set of books and obtained the same knowledge with a more CS focused background. The hardware stuff is harder to pick up compared to software when self learning (for most - I know counter examples). But it’s by no means impossible. Point being, I don’t agree with the author. If you’re a cloud native engineer and you want to learn the lower levels of abstraction, you can. But the point of cloud native is that you don’t need hardware knowledge and you can focus on your product/users/hobbyist itch instead.
- deleted 3y ago[deleted]
- doubled112 3y agoHands on wise, playing with software is basically no/low cost. If I don't get it right, I can just fix it and try again. If I want to play with hardware, I have to buy it. I also risk having to replace it when I let the magic smoke out. Or at least that's always been my intuition about getting started.
- reactordev 3y agoOne can fund the other though. Besides, if you’re tinkering with hardware, it really is super cheap already. ATmega’s are less than a meal. Various capacitors and resistors on Amazon in variety packs for a couple 20s. A soldering iron station for $50. Power supply for $40. It’s the same cost of entry as an entry level laptop…
- logtempo 3y agoHow do you program your atmega? with a laptop. Also you won't go anywhere with few resistors and capacitors, you also need transistors, diodes, maybe LED, maybe some soldering, wires, breadboard, some sensors to have fun with, some ICs such as 555 timer and other op-amp. Eventually some voltage converters, a multimeter and logic gates. The list can be long and at the end, for software you need a laptop, for electronic you need a laptop + many things. And those things can cost (the shipping cost more, actually).
- more_corn 3y agoYou will never know the joy of diagnosing a bad mainboard over the course of serval hours or days. Just stop, start and the problem magically goes away. You’ll not realize that you could pay $1k for the server and run it yourself instead of renting forever. You’ll never learn that arranging your cables nicely and labeling everything takes five times as long and is 100% worth it. Or maybe you will. These things happen in waves.
- didgetmaster 3y agoThe problem is that there are so many layers between someone's code and the bare metal that optimization often gets completely lost. The mentality is that computing is cheap and developer time is expensive. Write an inefficient function that wastes several thousand compute cycles but saves an hour of developer time is given priority in most cases. This mentality works great when the function is only run a few million times (or much less) over its lifetime. But when it gets added to some popular library; distributed across millions of machines; and used thousands of times each hour on each machine; those wasted cycles can really add up. What percentage of worldwide compute (with its associated wasted electricity and heat) can be attributed to inefficient code? Spending a few hours optimizing some popular code might actually do more to help the environment that driving an EV for an entire year.
- logtempo 3y agoI agree. And it's common to any engineering topics: DO a prototype that achieve the thing it is designed for, and IF you're going to deploy it in mass, optimize it. ELSE you don't need to.
- giantg2 3y agoI used to be half decent at navigating and using Linux, writing ksh, etc. With cloud, I don't have any need to know that information anymore. I guess it's kind of good in general, but feels bad to have significant portions of my professional skills be essentially obsolete. Pretty soon developers will be measured by how fast they can tell an LLM what to build for them.
- deleted 3y ago[deleted]
- SebFender 3y agoAbsolutely. I've also witnessed a significant decline in networking knowledge. Example, a simple conversation on routing, web proxies and firewalls rapidly creates uncomfortable silences in most teams I interact with...
- arter4 3y agoI wouldn't call syscall (and strace) knowledge, "hardware knowledge". It's mostly about OS knowledge. You can go a long way with strace without knowing how a PCI bus work or other hardware details. Which brings me to another point. Virtualization ensured you could decouple (to an extent) the behavior of an operating system from the actual hardware used underneath. Therefore, some hardware knowledge is often ignored, but you still need to know about the OS. With containers, you are decoupling the application from the host operating system! Which means some OS knowledge is often ignored. That said, abstractions can be leaky. Most of the time and in the most common scenarios, you can ignore the lower level details. No one develops a simple website thinking about NUMA cores or C-state transitions. But if you want to really squeeze every ounce of performance, or if if you run very complicated systems, then you still need to peek at the lower levels from time to time. Which means looking at the hypervisors for virtual machines (assuming you can... on public cloud you cannot), or looking at the operating system for containers.
- deleted 3y ago[deleted]
- sgarland 3y ago> With containers, you are decoupling the application from the host operating system! Yes and no. The host’s kernel is still running the show, which catches people by surprise. Reference Node (or more accurately, libuv) incorrectly reporting CPU count by relying on /proc/cpuinfo, etc. > But if you want to really squeeze every ounce of performance Or if your C-suite orders a massive reduction in cloud cost. My company’s devs were recently told to cut vCPU allocation by 50%. Then, when they did so, they were told to do it again. My team was told to right-size DB instances. In fairness, some of them were grossly overprovisoned. ZIRP was quite a thing. Unsurprisingly, this has resulted in a lot of problems that were being papered over with massive headroom.
- arter4 3y agoSo you downsized databases? I get that you can, to an extent, scale databases horizontally (especially if you don't need JOINs or you just use them for read operations), but autoscaling databases sounds a bit risky