30 ms·

Where are the low level system engineers?

109 points by pyb 4y ago

dmitrygr 4y ago
Around, working on cool things at companies that can afford this increasingly rare skillset. Google employs a lot, Apple does too, Facebook does as well. Most startups cannot afford us, and just keep trying to sell us on "impact" and "get in early on equity".
tonfreed 4y ago
I did a lot of C, network wizardry and kernel hacking at uni. I couldn't get a job in that though, so I moved further up the stack and now do backend instead.
pclmulqdq 4y ago
Right here. I used to do FPGA stuff in finance and low-level stuff at a FAANG, got tired of all the BS and left to start up my own company. I'll let you know how it goes, but it's not looking great...
pyb 4y ago
IME : they had to move up the stack, in order to access better employment opportunities.
- markmark 4y ago
  I've got a friend who is a long-time embedded C developer, about 20 years of experience. Really knows his stuff. I just hired him to write python and he got a massive pay rise to come over, because all the embedded shops in our area pay shit.
- flyingfences 4y ago
  Seconded. I look around, at this week's "Who's Hiring" thread, for example, and the systems/firmware/embedded jobs are slim pickings. Filter out the super-senior positions and I can count them on one hand. How's a guy supposed to gain the experience to be a senior embedded dev when nobody's hiring junior embedded devs? I'm sticking at this, hoping for a needle in the haystack, because I like it, but most people in our position would just pick up some JavaScript and Leetcode and double their salary.
  - booboofixer 4y ago
    If you also take into consideration the years it takes to go from junior to senior/experienced in the systems area, salary becomes a much bigger deciding factor. It’s not just 20k a year less for one year, its 20/30/40k less for as long as you’re mastering your area when you could spend less time mastering something else for more money.
  - AlotOfReading 4y ago
    Part of the issue is that places like HN cater to smaller startups and places eager to seem 'hip'. Companies that employ a lot of embedded engineers are invariably either FAANG type companies that don't need to advertise that way, or hardware companies so institutionalized that they treat it like any other administrative job and only post on LinkedIn/indeed. Occasionally you'll get a smaller startup/unicorn that's trying to complete with SV tech giants, but that's uncommon. I'm trying to implement these sorts of changes (and many others) right now at my own place, but it's quite frustrating from the other side too.
  - uxp100 4y ago
    Hacker News is just the wrong place to look. This is essentially a web dev site. Just use linked in or zip recruiter or whatever. There are tons of positions for embedded now, same trouble hiring as everyone else, and often they’re open to anyone with software experience, you don’t need to know the difference between Vdd and Vcc or how to use an oscilloscope, though obviously employers would prefer you did. (Well, forget the first one, Vdd vs Vcc is essentially trivia)
    - flyingfences 4y ago
      I am looking on those other, more traditional sites as well, and junior-level embedded jobs (without moving to California or China) are still far less common.
  - spacemadness 4y ago
    As someone who has recently left embedded engineering to pursue other avenues, I got tired of: 1. Being paid much less than my similarly leveled colleagues 2. Needing to work with engineers well outside my timezone 3. Being hounded by recruiters to do bluetooth comms and nothing but bluetooth comms 4. Having to be very careful to not put myself in a position where I need to fly to China to deal with manufacturing issues on a moments notice 5. Lack of remote work opportunites for obvious reasons I do enjoy working on low level code but not enough to deal with the industry in its current state.
- ______-_-______ 4y ago
  I'm in webdev to access any employment opportunities. Moving to kernel/embedded would be fun but it almost feels like jumping to another industry. I have no connections and no idea how to find a good company (e.g. what is the Joel Test for hardware companies?) I have gone on a few job sites and sprayed/prayed my resume, which so far has gotten zero interest from anyone. OTOH I'm lucky to have landed where I did so it might be for the best that I stay in webdev.
  - buescher 4y ago
    Koopman's Better Embedded Software should give you some good things to ask about.
- dmitrygr 4y ago
  Not even remotely close. As OP posted: there are too few low-level people left. This skillset commands much better pay than "upstack"
  - ummonk 4y ago
    How much does it pay? From what I’ve seen, market pay for systems devs is substantially lower than for those up the stack, outside of FAANGs and HFT outfits.
- selfhoster11 4y ago
  I'd love to get paid doing low-level coding (or at least try my hand at that). Sadly, I am very passionate about being able to afford food, so DevOps it is (not that I'm complaining).
- flukus 4y ago
  They also won't give a second glance to those who would like to move down the stack, even in markets where these sorts of skills are practically non-existent.
- mads 4y ago
  In 30 years, I have moved all the way from the bottom of the stack doing assembly to the top, where I now do deployment pipelines and other DevOps related tasks.
Ambolia 4y ago
I guess in the microcontroller and IoT world there's still many? Though now there are alternatives like micropython to avoid diving down to the low level.
- fxtentacle 4y ago
  Plus most of these are hobbyists. I'm not sure if the market contains enough money to sustain people doing this for a living.
h2odragon 4y ago
Why design new, low level systems when our stacks are built on old, debugged foundations and work "well enough"? Who wants to think about cycles and bytes when everyone else is talking about gigabytes and "spin up another 1,000 instances" is the expected answer to every problem? Is anyone hiring these people? Look at the SoC world, the stuff that's being made with the expectation of taking advantage of the public code ecosystem... Would you even want to work at a place like that? "Make the minimal system that passes tests; actual functionality is secondary" is not a good motivation. Even the RasPi people have blobs of ugly and seekrit hardware; and they're the epitome of "open" systems engineering now, aren't they?
- pjc50 4y ago
  Downvoted for the blind cynicism. It's true that the frontend side is awash with money, and that there are nowhere near as many such engineers, but we tend to be a lot quieter about it. A lot of people went into IoT. The economics of the hardware market mean that "open" is even more unprofitable than it is for software, which already has a bit of a crisis in terms of the few people holding up the world not being paid properly.
  - evocatus 4y ago
    > a crisis in terms of the few people holding up the world not being paid properly. Who? The farmers and sweatshop workers in frontier or emerging economies?
- enneff 4y ago
  You care a lot about bytes and cycles especially when you’re going to spin up another 1000 instances. The larger the scale the more optimisations pay off. It’s actually small to medium operations that don’t care so much about performance.
evocatus 4y ago
Bootcamps and anti-intellectual sentiment spreading through the software development community have convinced newer developers that web APIs and other high-level frameworks/languages are actually the bedrock of abstraction, and that nothing important lies beneath (if they even recognize there IS something beneath). Nothing ever breaks down there, anyway. And if something does, it's not my problem to fix. 16 weeks at a bootcamp is enough to get you up to speed with the important parts of a rigorous 4 year education into the foundations of computing. No one needs to know CS or computer engineering anyway. It's just like math - useless. I'm a software developer. How does software work, you ask? Uh...
- taurath 4y ago
  Plenty of graduates specifically don't work in systems engineering for different reasons - its not like there is a dearth of people with degrees. The jobs in say embedded programming are fewer, harder to get, and tend to pay worse unless you're at the top of the market. If you're at Meta making an OS for VR, you're at the top of the market and getting paid very well. If you're at Cisco making router firmware, or F5 making DNS boxes, you're likely getting paid far worse than at equivalent experience to say a web dev - for which there is a lot more work. Employment mobility is far far lower when doing embedded work.
  - ThrowawayR2 4y ago
    Short term, yes, there's lots of demand for front-/back-end web developers and the lower barrier to entry makes it a solid choice. Long term, though, it's a race to the bottom as more people flood in and eventually web developers become easily replaceable commodities. (Don't take that the wrong way; it's not criticism of web development. It's just what generally happens when there's an oversupply of anything in a marketplace. We've been in a boom for web developers for a long time and what goes up must come down.)
    - taurath 4y ago
      But embedded engineers are treated more like commodities nowadays already (except at the top or in specific niches like legacy banking software or HFT), because the industries that drove their day are also commoditized and outsourced. It’s relatively easy to hire. Yes eventually web devs will be eventually commoditized, but that doesn’t mean the return of low level programmers persay.
unget 4y ago
They’re at FAANGs. I can say from experience that Amazon and Meta employ tons of systems software engineers working on kernels, languages, low-level Unix software, etc.
Unklejoe 4y ago
I’m right here, bombing Leetcode interviews because they ask me to write a program that sorts the columns in a CSV file in 15 minutes and I only know C.
- ericbarrett 4y ago
  Likewise. I've known C since the 90s, C++ since the 00s (with a refresher for C++11), and recently picked up Rust—have already written a few thousand lines, it's a great language. But I do DevOps because the pay is better and the interviews are practical.
  - deleted 4y ago
    [deleted]
- yolo69420 4y ago
  Maybe I'm missing something but unless the interviewer expects a 100% correct csv parser that handles all possible corner cases this is basically - read file - start at beginning; if you encounter separator, add new entry to an array. if you encounter newline, save entire current array as a new line to an array of lines. - do lexicographical sort of arrays (can just use qsort for this). - if the test q actually wants you to parse out numbers and shit like that, nothing about the structure changes, you just have to record some extra info when parsing values and then use a different comparison depending on the type in the function you pass to qsort I dare say someone programming algorithms in a kernel or writing code for custom file systems should be able to do this. OK 15 min may be tight if you're nervous but you should get there.
  - ericbarrett 4y ago
    You forgot quotes. Unfortunately we're moving forward with other candidates at this time. Good luck in your job search!
    - the_only_law 4y ago
      It gets very nasty at this point especially because I’ve used popular languages who’s CSV parsing libraries were unable to handle some weird quotes properly.
afr0ck 4y ago
Not so many employers. Very few places/teams in this world care about cache coherency or memory ordering in the go-lang runtime on ARMv9 CPUs. You can count them using one hand fingers. The entry barrier is also too high; you need too many years of serious CS education, usually a PhD. And then find a place at Google or Meta Kernel/Compiler teams, which is obviously super competitive.
- pkhuong 4y ago
  I've been employed for a variable mix of system design and low level performance work my whole career... except for Google. There are definitely more than five firms that would pay reasonably well for my expertise, and my PhD was in mathematical optimisation.
  - flyingfences 4y ago
    > There are definitely more than five firms that would pay reasonably well for my expertise There are definitely more than five firms that would pay reasonably well for an ex-Googler with a "whole career" and a PhD, yes. The barrier to entry is too high for the rest of us.
- ceeplusplus 4y ago
  Definitely not just one hand (ex: HFT), but perhaps you can count them using 3-4 hands. The nature of most compiler work is that you're pushing the boundary of what's been done, which in academia is almost exclusively done by the top 1% researchers. And HFT is obviously winner take all, which further favors top performers.
- saagarjha 4y ago
  A PhD is definitely not required to work on this kind of stuff.
  - ungamedplayer 4y ago
    But maybe it is to get hired ?
yolo69420 4y ago
The specific things he's mentioning are domain specific topics to (it sounds like) OS development that most low-level capable programmers could pick up no problem. I'm talking about programmers working on any kind of application in C/C++ (or other systems languages) with nontrivial performance and memory requirements. It's not like there are a ton of those either, but they're there. If for some reason some day some market popped up for OS devs that paid better than the other sub-fields those low level programmers are currently working in, nobody need be concerned about supply of programmers who can do the job.
- the_only_law 4y ago
  > I'm talking about programmers working on any kind of application in C/C++ (or other systems languages) with nontrivial performance and memory requirements. It's not like there are a ton of those either, but they're there. I tried finding a job somewhere around this realm. Half the companies were using Go and none of them wanted to talk to me.
- jpindar 4y ago
  Companies don't want someone who is capable of picking up x no problem. They only want someone who already has a job doing x.
  - the_only_law 4y ago
    Furthermore, a lot of these companies seem to only want somebody be with 10-20 years of experience already doing x.
  - ummonk 4y ago
    FAANGs are generally looking for the for the kind of person who is capable of picking up x no problem. That’s why the whiteboard interview so many on here complain about is the way it is.
    - yolo69420 4y ago
      people complain because those whiteboard / algo heavy interviews don't actually test for that ability, and it's trivially obvious. just the fact that people practice them before FAANG interviews and their correctness/success rate and speed at completing those tests suddenly gets better by an order or magnitude or more should tell you how little those tests actually filter for any more fundamental abilities. a proper test for the abilities you mentioned should not only be doable no problem by any working programmer (especially senior and above) walking into that test naked and unprepared, in fact preparing for the test in any reasonable manner before the interview should give no advantage at all.
alkaloid 4y ago
I'm a specialist in low-level telco code, lots of which is written in C/Perl. Recruiters are breaking down my door, asking me if I code in Ruby (Rails). No.
- Datenstrom 4y ago
  Similar story here. Have lots of experience with low-level scientific compute, ML, and network code in specialized environments like avionics and robotics. I have used Rust in production so it is listed on my skills, but mostly C/C++ and of course Python for prototyping. Recruiters saw Rust and I got blockchain spam. Nope. I've moved on to my own thing anyway, but I've still had to put an explicit "no crypto/blockchain" everywhere they might look.
  - jpindar 4y ago
    But now all the bots will see the words crypto and blockchain in your profile (or wherever) and you'll get even more spam.
    - crazysim 4y ago
      maybe there's a way to include those words like "bl0ckchain" and "crypt0" or even similar looking cyrillic letters eg: сrʏpto ʙloϲkchain from https://www.irongeek.com/homoglyph-attack-generator.php https://www.irongeek.com/homoglyph-attack-generator.php
omgmajk 4y ago
I'm here. QA/Dev integrated systems.
fxtentacle 4y ago
Here :) I maintain a C++ service running in production since 2015. I released Windows and Mac kernel drivers in 2012, both of which are still used in commercial products. My NAS runs a patched kernel because the stock dm-cache didn't offload burst writes onto SSD. I once built a virtual filesystem driver to run inside my PS2 so that I could exfiltrate in-game debug data over LAN. My most recent low-level project was PCB+Firmware for a digital camera. But: Ruby on Rails just has a much better bang for buck than C++ and customers know it. Powerful hardware is dirt-cheap these days, so nobody wants to pay for the extra +50% you get from C++ over JavaScript. And if you want to be paid well, call your multi-threaded C++ calculations something like "CPU-accelerated AI deployment optimization" ;) And lastly: Most embedded C/C++ jobs went to China! We used to design hardware and firmware in the west. But nowadays, you just book a Shenzhen shop to design, code, and produce your product. That's why nowadays almost every device has firmware with Chinese debug strings inside ^_^
- jurschreuder 4y ago
  We build our own cameras and write our own drivers here in Amsterdam, and sometimes get requests from other companies here that rather pay more than trying to explain their projects to Chinese manufacturers. Some industrial cameras we buy from Basler and the guy who builds them lives right around the block in a village close to Amsterdam. So maybe it's coming back around. As an engineer I have a really strong opinion about only finance being left in Europe and all the engineering being outsourced to cheaper countries. I personally think in tech the ladder to the top is made so you have to switch to management at some point in order to promote any further. I see managers and people that actually want to volunteer to become a manager instead of an engineer as the root of this problem. At my company we organize the hierarchy more like a hospital or a law firm, with the best engineers having the highest positions, just like partners at a law firm.
  - jhallenworld 4y ago
    I work at a product design firm here in the US, we're also organized like this. Engineers that can deliver end up getting more clients, so it does end up somewhat like a law firm (their hourly rates are still higher, which IMHO is crazy..). We also make cameras, I've become the MIPI expert. Our bread and butter includes mechanical design, PCB design and firmware / FPGA design. This maybe sounds like basic stuff, but includes things like RF and COMSOL / ANSYS physical and mechanical modeling and simulation. A startup company in India hired us: they had plenty of software developers, but wanted experts to de-risk the hardware and low level firmware. https://www.nklabs.com/ https://www.nklabs.com/
danrocks 4y ago
Right here. I was CTO of a hardware + software startup. Built a home automation device, couldn't get scale, sold to FAANG. Now I manage rEqUireMenTs, syNcHroniZaTiOn oF cRoSs fUnctIONal teams, building a myriad of control planes over open source software, make 5x more, and am 10x less happy. Bills to pay.
- the_only_law 4y ago
  My condolences, that sounds horrific.
jldugger 4y ago
> Code schools don't seem to be teaching this stuff and some universities have moved up the stack. ABET accredited CS degree programs still require students take courses including "Exposure to computer architecture and organization, information management, networking and communication, operating systems, and parallel and distributed computing." I'm not aware of any flagship state universities that don't have an ABET degree option, but maybe I just have massive selection bias. [1]: https://www.abet.org/accreditation/accreditation-criteria/criteria-for-accrediting-computing-programs-2020-2021/ https://www.abet.org/accreditation/accreditation-criteria/cr...
- morpheuskafka 4y ago
  UNC Chapel Hill doesn't, probably because we're not an engineering school (actually, we do show up on that page for the BME program but that is a joint degree with NC State, which is). I suspect there are a few other universities in that position that have reputable CS programs but otherwise no engineering--although CS itself can be said to not be engineering, I would argue that for the 90% of CS majors wanting to get a SWE job, what they are effectively seeking is an engineering degree not a science one. Still, there isn't any expectation on the part of employees that a degree is ABET accredited, as there is for engineering. C, basic digital logic, and MIPS assembler are part of the required intro sequence here.
- chrisseaton 4y ago
  Do students or employers care about ABET? I don’t think they do.
  - jldugger 4y ago
    Maybe the kind of person who complains on twitter we dont educate enough low level engineers does?
spicymaki 4y ago
I'm here brother! Low level for life!!! C/C++/weird bytecodes interpreted by FW. They don't give us the love like they used to, but can't build/optimize/secure/debug systems without us. Employers are leaving a lot of performance on the table and they don't even know why. I feel bad that we are not training the next generation, but at least I will be gainfully employed until I die.
- danrocks 4y ago
  > Employers are leaving a lot of performance on the table and they don't even know > why. This, 100 times. All performance discussions at my employer are at the upper layers of the stack ("hey look I'm using char vs. string in this C# method, 0.15% improvement!"), while nobody is looking at the lower layers where the real improvements can be made and benefit all platforms.
psyc 4y ago
Well I’m still here. But when I learned low level systems programming, it was just called ‘programming’
jcpst 4y ago
I started at a bootcamp-style program. After getting some experience, moving through junior, to mid, to senior level, there was this inevitable progression to learn what’s going on lower in the stack. Over time, there is more opportunity to run into performance problems, memory/ gc issues, deadlocks, null pointers… And when you don’t know what the hell is going on, you can end up going pretty deep down the stack before realizing you’re off course. Obviously, this doesn’t happen to every developer, but it does seem like there are always some devs that want to dig into this stuff. I’m now working on a team that builds foundational things like api templates, common libraries, patterns… It all seems like this long progression as I work towards lower level knowledge.
social_quotient 4y ago
I’ve been hunting a contractor for C and BPF/eBPF work. It’s been a challenge. Would love to know where to have better luck
- dmitrygr 4y ago
  If you are asking "where?", you are asking the wrong question, and that is likely why you are having troubles. You need to ask "at what cost?" Suddenly, your issues will disappear and you'll have no trouble hiring (at the proper cost) Generally, when you are unable to find X at cost Y, either settle for worse X, or raise your price-i-am-willing-to-pay (Y)
  - social_quotient 4y ago
    Haven’t even gotten to the cost part. But yes, I generally agree. People want unicorns without unicorn budgets.
- hcrean 4y ago
  What are you specifically looking for someone to do?
- pwnna 4y ago
  Still looking? What is the best way to contact you? Keybase?
deleted 4y ago
[deleted]
bamboozled 4y ago
They're bogged down fixing millions of issues.
QuikAccount 4y ago
Honestly I'd love to switch to systems programming from web CRUD. I just don't know how to get into that field.
- dmitrygr 4y ago
  Start with an arduino, maybe make a simple game. Then move on to some cortex-m style micros. Get comfy with those. That is already a hireable skill.
ssalka 4y ago
JavaScript 4 ever
BirAdam 4y ago
I actually learned this stuff in school, but I ended up not working in that field at all. I ended up first in computer repair, and then working in a NOC at datacenter, and then moving into management. Life is weird like that. I do more low level crap in hobby osdev on 8088 now that I don’t write software for a living in any capacity.
bacon_waffle 4y ago
I'm not on twitter... Working on consumer electronics, VoIP phone systems specifically. It's not sexy, but my boss is cool and it's reliable employment - have been at the same gig over 5 years. I occasionally feel jealous of my peers higher up the stack who get paid a bunch more and/or don't have to explain "how git works" weekly. Day-to-day, I deal with some tedious old school codebases that can be frustrating at times, but I occasionally get to play electronics engineer, and have an ongoing embedded Rust project to keep things fresh.
- kody 4y ago
  "How git works", especially demystifying merge conflicts, has been a major pain point at all my jobs from embedded to frontend web dev, unfortunately.
  - bacon_waffle 4y ago
    I'm a bit torn about it to be honest. On one hand, it's sad/frustrating to work with colleagues who can't be bothered to figure git out. On the other, it is sometimes nice to have a less technically-difficult distraction (or at least one in a different problem space) from what I actually get paid to do. Maybe the right thing is to lean in to it, and become a git consultant! I'm a bit scared to see if that's a thing - would be a sad commentary on the state of the software industry.
    - scruple 4y ago
      FWIW, I'm aware of a handful of people that work for a very large AAA game studio, nearby where I live, who are, in fact, git "experts" -- and, really, just extremely knowledgeable about version control systems, generally speaking, since they don't just use git -- for the company and they spend a large % of their time helping these teams deal with version control issues. As I understand it, they've written a lot of automation to deal with recurring problems like merge conflicts, perforce depot / workspace issues, etc. And they make good money for that expertise...
ramchip 4y ago
I don't get this take... there's plenty of low level system engineers graduating from computer/electronic engineering programs, and coding schools target a very different profile, so I don't think they compete with each other significantly.
eyberg 4y ago
Right here. https://nanovms.com https://nanovms.com If you like kernel level c I'm hiring. The best hackers want to work with the best hackers and we are charting a new course for the 2020s - not one from the 1960s. Despite what you might read in the main-stream dev news our work is extremely important and needs people just like you. You don't have to work at a boring corporate job -- you can instead work at the unikernel company NanoVMs - you just need to email me.
- _xrp0 4y ago
  Hey, can you speak a bit about how nanos compares to something Like unikraft? IIRC nanos doesn't eliminate syscalls in it's entirety like other unikernels.
  - eyberg 4y ago
    We don't think that's a defining characteristic no. We keep the kernel/user cause otherwise you can very very easily change page permissions which means all security goes out the window. You could instantly make rwx on all pages with a few instructions. If you rewind to ~ 2013 most unikernels were talking about trashing the context switch yet depending on the "context" that implies various things (kthread <> kthread, user proc <> user proc, user thread <> user thread, etc...) They all have different costs and while we definitely agree with the aggressive cost of various user process to user process the kernel <> user was one we felt was not worth trashing. Clearly though, because of massive adoption, we see 200% extra throughput on GCP and up to 300% on AWS - so the performance and security remain.
ryukoposting 4y ago
Here! I am a firmware architecture engineer at a Chicago-based startup. I get paid to do embedded firmware, but I like to make little kernels in my spare time.
zubiaur 4y ago
Garmin
- rsaxvc 4y ago
  It's a living.
walrus01 4y ago
> where are the low level system engineers working on operating systems and software loads for UAS flight controllers, some of them. stuff like an auterion FC is very limited in CPU and RAM.
binarynate 4y ago
I believe ECE (Electrical and Computer Engineering) programs still provide a strong background in low-level systems programming. At least, that was the case when I was at Purdue 9 years ago. My experience with low-level programming has been a boomerang: focused on embedded systems in college, went gradually further up the stack after school due to my interests, and then gradually drifted back down to low-level C++ as I began working on more complex problems.
jlokier 4y ago
I think a lot of people currently have "the knowledge" to do low level system engineering, but I agree with the comments that the job market drives what people pursue, and the job market isn't particularly favourable to low level system engineering. So those skills are getting rusty out of lack of use, and aren't being passed on in the workplace because that's not the sort of work people are doing in the workplace. "Low level" covers many things that don't have much overlap. In the employment market, I see a significant distinction between "embedded"/"firmware" and "server systems engineering". TLDR: Embedded pay is low compared with other software. As far as I can tell, from inbound recruiter mail and job ads, the pay for embedded and firmware work is consistently low compared with what's possible in webdev, cloud, and software generally. Even for "senior" embedded and firmware work. I suspect this is because it's adjacent to things like FPGAs and electronics, which also pay low compared with software engineering generally. By low, I mean senior embedded pay at about 60% of senior webdev pay. On the other hand, seasoned "system level" engineering such as kernels, compilers, database engines, GPU drivers, storage, crypto, and many other such things, especially things that run on servers, can pay well and seems to be in some demand, especially at big companies. (Device drivers can fall into either category. For example a GPU or storage subsystem driver is at a completely different level from a USB serial driver.)
tintedfireglass 4y ago
I am a college student who loves writing low level and reading this thread discourages me :(
- KerrAvon 4y ago
  Why? What do you want to work on? FAANGS are certainly hiring “low-level” engineers. Apple is hiring and has relatively low-bullshit interviews (depending on the team).
  - astrange 4y ago
    In fact, Apple has lots of open jobs for exactly the kind of low-level engineers this thread is about and you can find the team managers posting about them on Twitter.
    - tintedfireglass 4y ago
      Reading this thread made me feel like all the embedded jobs are low paying or moved to China. And whatever is left has fierce competition. That's why my parent comment.
      - astrange 4y ago
        I think most people on HN are web developers who only talk to other web developers and think the only good jobs in the world are at Facebook. There's a definite bias. Of course, there's a few more problems than low paying or moved to China - it might be on-site somewhere expensive (Apple), somewhere you don't want to live (probably a lot of these), or boring or unethical (a defense contractor).
  - guessmyname 4y ago
    > Apple is hiring and has relatively low-bullshit interviews (depending on the team). I agree. I passed interviews for a Systems Software Engineer position at Apple two months ago, and I can attest that the interview process was very down to Earth. No LeetCode questions. Instead, the interviews were mostly system design and and simple programming exercises where finding a solution was less important than the discussion with the interviewers. If anyone is interested, Apple is still hiring - https://jobs.apple.com/en-us/search?search=systems%20engineer&sort=relevance https://jobs.apple.com/en-us/search?search=systems%20enginee...
da39a3ee 4y ago
From what I gather it doesn't pay as well as other engineering subgenres, so presumably the market is telling us we don't actually need that many?
Matthias247 4y ago
Somewhat here. But to be honest, it's a tricky place to be. As an individual who is very proficient in this areas, I get rarely any reachouts about those skills. 90% of LinkedIn recuiter messages inquire about generic high level programming projects. Then there's 8% crypto related things, and a very tiny amount of other systems level engineering roles. On the other hand trying to hire someone for such a role is super hard. Most engineers don't deal with low level details and systems programming languages anymore. Compensation wise the domain does not be better than others. While one assumes it might be more specialized, that seems by far offset by the amount of lower opportunities and thereby less chances to negotiate compensation than in other software engineering parts.
- bibabaloo 4y ago
  I think another big reason for lower compensation is because a lot of companies working at a low level tend to be companies making physical electronic products. Profits for physical profits don't scale as exponentially as they do with software only products, so companies don't have endless amounts of cash to throw around at engineers. If you're after good compensation and want to work at a low level, then go for companies that sell both hardware and software. FAANGs are an obvious choice but B2B companies (usually IoT-ish) that have have both a software and hardware subscription component are another option worth looking at.
- scruple 4y ago
  > Compensation wise the domain does not be better than others. Yep. I moved into web development because it paid a lot more and carried a lot less stress. The 7 years I spent working on embedded systems paid off in a major way, too, because I showed up equipped to directly address performance problems. I'm still in web development and I'll probably never leave.
c_o_n_v_e_x 4y ago
Any recommended sites for finding/hiring embedded talent? I had A LOT of trouble trying to find someone for a driver development project a few years ago.
- gsempe 4y ago
  I’m doing Embedsys Weekly. With a job offer, you can post the opportunity on the job board jobs.embedsysweekly.com
rramadass 4y ago
The Software Industry is headed off a cliff. People who actually take care of the low-level plumbing are not being incentivized nor given their due. Expertise and Hard-won knowledge is being lost. There is also not much ground-breaking research going on in these domains which doesn't bode well for the future. As a final nail in the coffin, Colleges/Universities are not even creating/training the next generation of "Low-Level System Engineers". We are going to end-up with a handful of Companies holding the keys to the kingdom and holding Society-at-large hostage.
Balgair 4y ago
Since there are a lot of C/++ people in this thread, I figured I should swing this job posting at y'all: https://flagshipbio.bamboohr.com/jobs/view.php?id=79&source=flagshipbio https://flagshipbio.bamboohr.com/jobs/view.php?id=79&source=... I'm in no way affiliated with the company, but my friend is (honest!). They have had a heck of a time finding a person for the role. Comp maxes out at ~$120k, remote is just fine. If anyone is good with C++ and would like to transition to the bio side of things, this role may be for you.
- lacker 4y ago
  I mean, if comp maxes out at $120k for this job where you want an engineer with mid-level experience, and Google is giving $190k total comp to new graduates who have no experience, what do you expect? The compensation is way below market for a good, experienced C++ engineer.
- zwerdlds 4y ago
  >Basic Qualifications > • PhD degree in a field related to Image Analysis, or equivalent work experience I think I know why.
  - Balgair 4y ago
    Any other things that stand out to you? Impartial feedback may be good for my buddy to see.
sebazzz 4y ago
Well I'd love to write C++ as my day job, but it is not easy to get into a C++ job here in the Netherlands. There aren't a lot of native code positions, and those who are pay generally less than I get now or have an unclear offering. Once you take a career path, like C# web development in my case, you generally keep doing that unless you are attracted by a company which really needs you or believes in you. So my C++ experience is limited to side projects like Marlin based firmware and that's it.
jstream67 4y ago
I got excited because I thought this was talking about linux systems engineers and how that is a almost non-existent career path these days. I would call people who write c/c++ code 'programmers' - interesting to see that title is not universal. If you are wondering where all the c/c++ programmers (system engineers) are - they are in the defense industry - or at least all the ones that I know are.
glonq 4y ago
I work in a place that desperately needs a senior systems/embedded engineer, and our upper leadership is willing to throw so much money at that position that I'm jokingly suggesting that I should demote myself from Director back down to Sr Developer so that I can earn more pay and suffer less bullshit.

109 points by pyb 4y ago

Around, working on cool things at companies that can afford this increasingly rare skillset. Google employs a lot, Apple does too, Facebook does as well. Most startups cannot afford us, and just keep trying to sell us on "impact" and "get in early on equity".

tonfreed 4y ago

I did a lot of C, network wizardry and kernel hacking at uni. I couldn't get a job in that though, so I moved further up the stack and now do backend instead.

pclmulqdq 4y ago

Right here. I used to do FPGA stuff in finance and low-level stuff at a FAANG, got tired of all the BS and left to start up my own company. I'll let you know how it goes, but it's not looking great...

pyb 4y ago

IME : they had to move up the stack, in order to access better employment opportunities.

markmark 4y ago
I've got a friend who is a long-time embedded C developer, about 20 years of experience. Really knows his stuff. I just hired him to write python and he got a massive pay rise to come over, because all the embedded shops in our area pay shit.
flyingfences 4y ago
Seconded. I look around, at this week's "Who's Hiring" thread, for example, and the systems/firmware/embedded jobs are slim pickings. Filter out the super-senior positions and I can count them on one hand. How's a guy supposed to gain the experience to be a senior embedded dev when nobody's hiring junior embedded devs? I'm sticking at this, hoping for a needle in the haystack, because I like it, but most people in our position would just pick up some JavaScript and Leetcode and double their salary.
- booboofixer 4y ago
  If you also take into consideration the years it takes to go from junior to senior/experienced in the systems area, salary becomes a much bigger deciding factor. It’s not just 20k a year less for one year, its 20/30/40k less for as long as you’re mastering your area when you could spend less time mastering something else for more money.
- AlotOfReading 4y ago
  Part of the issue is that places like HN cater to smaller startups and places eager to seem 'hip'. Companies that employ a lot of embedded engineers are invariably either FAANG type companies that don't need to advertise that way, or hardware companies so institutionalized that they treat it like any other administrative job and only post on LinkedIn/indeed. Occasionally you'll get a smaller startup/unicorn that's trying to complete with SV tech giants, but that's uncommon. I'm trying to implement these sorts of changes (and many others) right now at my own place, but it's quite frustrating from the other side too.
- uxp100 4y ago
  Hacker News is just the wrong place to look. This is essentially a web dev site. Just use linked in or zip recruiter or whatever. There are tons of positions for embedded now, same trouble hiring as everyone else, and often they’re open to anyone with software experience, you don’t need to know the difference between Vdd and Vcc or how to use an oscilloscope, though obviously employers would prefer you did. (Well, forget the first one, Vdd vs Vcc is essentially trivia)
  - flyingfences 4y ago
    I am looking on those other, more traditional sites as well, and junior-level embedded jobs (without moving to California or China) are still far less common.
- spacemadness 4y ago
  As someone who has recently left embedded engineering to pursue other avenues, I got tired of: 1. Being paid much less than my similarly leveled colleagues 2. Needing to work with engineers well outside my timezone 3. Being hounded by recruiters to do bluetooth comms and nothing but bluetooth comms 4. Having to be very careful to not put myself in a position where I need to fly to China to deal with manufacturing issues on a moments notice 5. Lack of remote work opportunites for obvious reasons I do enjoy working on low level code but not enough to deal with the industry in its current state.
______-_-______ 4y ago
I'm in webdev to access any employment opportunities. Moving to kernel/embedded would be fun but it almost feels like jumping to another industry. I have no connections and no idea how to find a good company (e.g. what is the Joel Test for hardware companies?) I have gone on a few job sites and sprayed/prayed my resume, which so far has gotten zero interest from anyone. OTOH I'm lucky to have landed where I did so it might be for the best that I stay in webdev.
- buescher 4y ago
  Koopman's Better Embedded Software should give you some good things to ask about.
dmitrygr 4y ago
Not even remotely close. As OP posted: there are too few low-level people left. This skillset commands much better pay than "upstack"
- ummonk 4y ago
  How much does it pay? From what I’ve seen, market pay for systems devs is substantially lower than for those up the stack, outside of FAANGs and HFT outfits.
selfhoster11 4y ago
I'd love to get paid doing low-level coding (or at least try my hand at that). Sadly, I am very passionate about being able to afford food, so DevOps it is (not that I'm complaining).
flukus 4y ago
They also won't give a second glance to those who would like to move down the stack, even in markets where these sorts of skills are practically non-existent.
mads 4y ago
In 30 years, I have moved all the way from the bottom of the stack doing assembly to the top, where I now do deployment pipelines and other DevOps related tasks.

Ambolia 4y ago

I guess in the microcontroller and IoT world there's still many? Though now there are alternatives like micropython to avoid diving down to the low level.

fxtentacle 4y ago
Plus most of these are hobbyists. I'm not sure if the market contains enough money to sustain people doing this for a living.

h2odragon 4y ago

Why design new, low level systems when our stacks are built on old, debugged foundations and work "well enough"? Who wants to think about cycles and bytes when everyone else is talking about gigabytes and "spin up another 1,000 instances" is the expected answer to every problem? Is anyone hiring these people? Look at the SoC world, the stuff that's being made with the expectation of taking advantage of the public code ecosystem... Would you even want to work at a place like that? "Make the minimal system that passes tests; actual functionality is secondary" is not a good motivation. Even the RasPi people have blobs of ugly and seekrit hardware; and they're the epitome of "open" systems engineering now, aren't they?

pjc50 4y ago
Downvoted for the blind cynicism. It's true that the frontend side is awash with money, and that there are nowhere near as many such engineers, but we tend to be a lot quieter about it. A lot of people went into IoT. The economics of the hardware market mean that "open" is even more unprofitable than it is for software, which already has a bit of a crisis in terms of the few people holding up the world not being paid properly.
- evocatus 4y ago
  > a crisis in terms of the few people holding up the world not being paid properly. Who? The farmers and sweatshop workers in frontier or emerging economies?
enneff 4y ago
You care a lot about bytes and cycles especially when you’re going to spin up another 1000 instances. The larger the scale the more optimisations pay off. It’s actually small to medium operations that don’t care so much about performance.

evocatus 4y ago

Bootcamps and anti-intellectual sentiment spreading through the software development community have convinced newer developers that web APIs and other high-level frameworks/languages are actually the bedrock of abstraction, and that nothing important lies beneath (if they even recognize there IS something beneath). Nothing ever breaks down there, anyway. And if something does, it's not my problem to fix. 16 weeks at a bootcamp is enough to get you up to speed with the important parts of a rigorous 4 year education into the foundations of computing. No one needs to know CS or computer engineering anyway. It's just like math - useless. I'm a software developer. How does software work, you ask? Uh...

taurath 4y ago
Plenty of graduates specifically don't work in systems engineering for different reasons - its not like there is a dearth of people with degrees. The jobs in say embedded programming are fewer, harder to get, and tend to pay worse unless you're at the top of the market. If you're at Meta making an OS for VR, you're at the top of the market and getting paid very well. If you're at Cisco making router firmware, or F5 making DNS boxes, you're likely getting paid far worse than at equivalent experience to say a web dev - for which there is a lot more work. Employment mobility is far far lower when doing embedded work.
- ThrowawayR2 4y ago
  Short term, yes, there's lots of demand for front-/back-end web developers and the lower barrier to entry makes it a solid choice. Long term, though, it's a race to the bottom as more people flood in and eventually web developers become easily replaceable commodities. (Don't take that the wrong way; it's not criticism of web development. It's just what generally happens when there's an oversupply of anything in a marketplace. We've been in a boom for web developers for a long time and what goes up must come down.)
  - taurath 4y ago
    But embedded engineers are treated more like commodities nowadays already (except at the top or in specific niches like legacy banking software or HFT), because the industries that drove their day are also commoditized and outsourced. It’s relatively easy to hire. Yes eventually web devs will be eventually commoditized, but that doesn’t mean the return of low level programmers persay.

unget 4y ago

They’re at FAANGs. I can say from experience that Amazon and Meta employ tons of systems software engineers working on kernels, languages, low-level Unix software, etc.

Unklejoe 4y ago

I’m right here, bombing Leetcode interviews because they ask me to write a program that sorts the columns in a CSV file in 15 minutes and I only know C.

ericbarrett 4y ago
Likewise. I've known C since the 90s, C++ since the 00s (with a refresher for C++11), and recently picked up Rust—have already written a few thousand lines, it's a great language. But I do DevOps because the pay is better and the interviews are practical.
- deleted 4y ago
  [deleted]
yolo69420 4y ago
Maybe I'm missing something but unless the interviewer expects a 100% correct csv parser that handles all possible corner cases this is basically - read file - start at beginning; if you encounter separator, add new entry to an array. if you encounter newline, save entire current array as a new line to an array of lines. - do lexicographical sort of arrays (can just use qsort for this). - if the test q actually wants you to parse out numbers and shit like that, nothing about the structure changes, you just have to record some extra info when parsing values and then use a different comparison depending on the type in the function you pass to qsort I dare say someone programming algorithms in a kernel or writing code for custom file systems should be able to do this. OK 15 min may be tight if you're nervous but you should get there.
- ericbarrett 4y ago
  You forgot quotes. Unfortunately we're moving forward with other candidates at this time. Good luck in your job search!
  - the_only_law 4y ago
    It gets very nasty at this point especially because I’ve used popular languages who’s CSV parsing libraries were unable to handle some weird quotes properly.

afr0ck 4y ago

Not so many employers. Very few places/teams in this world care about cache coherency or memory ordering in the go-lang runtime on ARMv9 CPUs. You can count them using one hand fingers. The entry barrier is also too high; you need too many years of serious CS education, usually a PhD. And then find a place at Google or Meta Kernel/Compiler teams, which is obviously super competitive.

pkhuong 4y ago
I've been employed for a variable mix of system design and low level performance work my whole career... except for Google. There are definitely more than five firms that would pay reasonably well for my expertise, and my PhD was in mathematical optimisation.
- flyingfences 4y ago
  > There are definitely more than five firms that would pay reasonably well for my expertise There are definitely more than five firms that would pay reasonably well for an ex-Googler with a "whole career" and a PhD, yes. The barrier to entry is too high for the rest of us.
ceeplusplus 4y ago
Definitely not just one hand (ex: HFT), but perhaps you can count them using 3-4 hands. The nature of most compiler work is that you're pushing the boundary of what's been done, which in academia is almost exclusively done by the top 1% researchers. And HFT is obviously winner take all, which further favors top performers.
saagarjha 4y ago
A PhD is definitely not required to work on this kind of stuff.
- ungamedplayer 4y ago
  But maybe it is to get hired ?

yolo69420 4y ago

The specific things he's mentioning are domain specific topics to (it sounds like) OS development that most low-level capable programmers could pick up no problem. I'm talking about programmers working on any kind of application in C/C++ (or other systems languages) with nontrivial performance and memory requirements. It's not like there are a ton of those either, but they're there. If for some reason some day some market popped up for OS devs that paid better than the other sub-fields those low level programmers are currently working in, nobody need be concerned about supply of programmers who can do the job.

the_only_law 4y ago
> I'm talking about programmers working on any kind of application in C/C++ (or other systems languages) with nontrivial performance and memory requirements. It's not like there are a ton of those either, but they're there. I tried finding a job somewhere around this realm. Half the companies were using Go and none of them wanted to talk to me.
jpindar 4y ago
Companies don't want someone who is capable of picking up x no problem. They only want someone who already has a job doing x.
- the_only_law 4y ago
  Furthermore, a lot of these companies seem to only want somebody be with 10-20 years of experience already doing x.
- ummonk 4y ago
  FAANGs are generally looking for the for the kind of person who is capable of picking up x no problem. That’s why the whiteboard interview so many on here complain about is the way it is.
  - yolo69420 4y ago
    people complain because those whiteboard / algo heavy interviews don't actually test for that ability, and it's trivially obvious. just the fact that people practice them before FAANG interviews and their correctness/success rate and speed at completing those tests suddenly gets better by an order or magnitude or more should tell you how little those tests actually filter for any more fundamental abilities. a proper test for the abilities you mentioned should not only be doable no problem by any working programmer (especially senior and above) walking into that test naked and unprepared, in fact preparing for the test in any reasonable manner before the interview should give no advantage at all.

alkaloid 4y ago

I'm a specialist in low-level telco code, lots of which is written in C/Perl. Recruiters are breaking down my door, asking me if I code in Ruby (Rails). No.

Datenstrom 4y ago
Similar story here. Have lots of experience with low-level scientific compute, ML, and network code in specialized environments like avionics and robotics. I have used Rust in production so it is listed on my skills, but mostly C/C++ and of course Python for prototyping. Recruiters saw Rust and I got blockchain spam. Nope. I've moved on to my own thing anyway, but I've still had to put an explicit "no crypto/blockchain" everywhere they might look.
- jpindar 4y ago
  But now all the bots will see the words crypto and blockchain in your profile (or wherever) and you'll get even more spam.
  - crazysim 4y ago
    maybe there's a way to include those words like "bl0ckchain" and "crypt0" or even similar looking cyrillic letters eg: сrʏpto ʙloϲkchain from https://www.irongeek.com/homoglyph-attack-generator.php https://www.irongeek.com/homoglyph-attack-generator.php

omgmajk 4y ago

I'm here. QA/Dev integrated systems.

fxtentacle 4y ago

Here :) I maintain a C++ service running in production since 2015. I released Windows and Mac kernel drivers in 2012, both of which are still used in commercial products. My NAS runs a patched kernel because the stock dm-cache didn't offload burst writes onto SSD. I once built a virtual filesystem driver to run inside my PS2 so that I could exfiltrate in-game debug data over LAN. My most recent low-level project was PCB+Firmware for a digital camera. But: Ruby on Rails just has a much better bang for buck than C++ and customers know it. Powerful hardware is dirt-cheap these days, so nobody wants to pay for the extra +50% you get from C++ over JavaScript. And if you want to be paid well, call your multi-threaded C++ calculations something like "CPU-accelerated AI deployment optimization" ;) And lastly: Most embedded C/C++ jobs went to China! We used to design hardware and firmware in the west. But nowadays, you just book a Shenzhen shop to design, code, and produce your product. That's why nowadays almost every device has firmware with Chinese debug strings inside ^_^

jurschreuder 4y ago
We build our own cameras and write our own drivers here in Amsterdam, and sometimes get requests from other companies here that rather pay more than trying to explain their projects to Chinese manufacturers. Some industrial cameras we buy from Basler and the guy who builds them lives right around the block in a village close to Amsterdam. So maybe it's coming back around. As an engineer I have a really strong opinion about only finance being left in Europe and all the engineering being outsourced to cheaper countries. I personally think in tech the ladder to the top is made so you have to switch to management at some point in order to promote any further. I see managers and people that actually want to volunteer to become a manager instead of an engineer as the root of this problem. At my company we organize the hierarchy more like a hospital or a law firm, with the best engineers having the highest positions, just like partners at a law firm.
- jhallenworld 4y ago
  I work at a product design firm here in the US, we're also organized like this. Engineers that can deliver end up getting more clients, so it does end up somewhat like a law firm (their hourly rates are still higher, which IMHO is crazy..). We also make cameras, I've become the MIPI expert. Our bread and butter includes mechanical design, PCB design and firmware / FPGA design. This maybe sounds like basic stuff, but includes things like RF and COMSOL / ANSYS physical and mechanical modeling and simulation. A startup company in India hired us: they had plenty of software developers, but wanted experts to de-risk the hardware and low level firmware. https://www.nklabs.com/ https://www.nklabs.com/

danrocks 4y ago

Right here. I was CTO of a hardware + software startup. Built a home automation device, couldn't get scale, sold to FAANG. Now I manage rEqUireMenTs, syNcHroniZaTiOn oF cRoSs fUnctIONal teams, building a myriad of control planes over open source software, make 5x more, and am 10x less happy. Bills to pay.

the_only_law 4y ago
My condolences, that sounds horrific.

jldugger 4y ago

> Code schools don't seem to be teaching this stuff and some universities have moved up the stack. ABET accredited CS degree programs still require students take courses including "Exposure to computer architecture and organization, information management, networking and communication, operating systems, and parallel and distributed computing." I'm not aware of any flagship state universities that don't have an ABET degree option, but maybe I just have massive selection bias. [1]: https://www.abet.org/accreditation/accreditation-criteria/criteria-for-accrediting-computing-programs-2020-2021/ https://www.abet.org/accreditation/accreditation-criteria/cr...

morpheuskafka 4y ago
UNC Chapel Hill doesn't, probably because we're not an engineering school (actually, we do show up on that page for the BME program but that is a joint degree with NC State, which is). I suspect there are a few other universities in that position that have reputable CS programs but otherwise no engineering--although CS itself can be said to not be engineering, I would argue that for the 90% of CS majors wanting to get a SWE job, what they are effectively seeking is an engineering degree not a science one. Still, there isn't any expectation on the part of employees that a degree is ABET accredited, as there is for engineering. C, basic digital logic, and MIPS assembler are part of the required intro sequence here.
chrisseaton 4y ago
Do students or employers care about ABET? I don’t think they do.
- jldugger 4y ago
  Maybe the kind of person who complains on twitter we dont educate enough low level engineers does?

spicymaki 4y ago

I'm here brother! Low level for life!!! C/C++/weird bytecodes interpreted by FW. They don't give us the love like they used to, but can't build/optimize/secure/debug systems without us. Employers are leaving a lot of performance on the table and they don't even know why. I feel bad that we are not training the next generation, but at least I will be gainfully employed until I die.

danrocks 4y ago
> Employers are leaving a lot of performance on the table and they don't even know > why. This, 100 times. All performance discussions at my employer are at the upper layers of the stack ("hey look I'm using char vs. string in this C# method, 0.15% improvement!"), while nobody is looking at the lower layers where the real improvements can be made and benefit all platforms.

psyc 4y ago

Well I’m still here. But when I learned low level systems programming, it was just called ‘programming’

jcpst 4y ago

I started at a bootcamp-style program. After getting some experience, moving through junior, to mid, to senior level, there was this inevitable progression to learn what’s going on lower in the stack. Over time, there is more opportunity to run into performance problems, memory/ gc issues, deadlocks, null pointers… And when you don’t know what the hell is going on, you can end up going pretty deep down the stack before realizing you’re off course. Obviously, this doesn’t happen to every developer, but it does seem like there are always some devs that want to dig into this stuff. I’m now working on a team that builds foundational things like api templates, common libraries, patterns… It all seems like this long progression as I work towards lower level knowledge.

social_quotient 4y ago

I’ve been hunting a contractor for C and BPF/eBPF work. It’s been a challenge. Would love to know where to have better luck

dmitrygr 4y ago
If you are asking "where?", you are asking the wrong question, and that is likely why you are having troubles. You need to ask "at what cost?" Suddenly, your issues will disappear and you'll have no trouble hiring (at the proper cost) Generally, when you are unable to find X at cost Y, either settle for worse X, or raise your price-i-am-willing-to-pay (Y)
- social_quotient 4y ago
  Haven’t even gotten to the cost part. But yes, I generally agree. People want unicorns without unicorn budgets.
hcrean 4y ago
What are you specifically looking for someone to do?
pwnna 4y ago
Still looking? What is the best way to contact you? Keybase?

deleted 4y ago

[deleted]

bamboozled 4y ago

They're bogged down fixing millions of issues.

QuikAccount 4y ago

Honestly I'd love to switch to systems programming from web CRUD. I just don't know how to get into that field.

dmitrygr 4y ago
Start with an arduino, maybe make a simple game. Then move on to some cortex-m style micros. Get comfy with those. That is already a hireable skill.

ssalka 4y ago

JavaScript 4 ever

BirAdam 4y ago

I actually learned this stuff in school, but I ended up not working in that field at all. I ended up first in computer repair, and then working in a NOC at datacenter, and then moving into management. Life is weird like that. I do more low level crap in hobby osdev on 8088 now that I don’t write software for a living in any capacity.

bacon_waffle 4y ago

I'm not on twitter... Working on consumer electronics, VoIP phone systems specifically. It's not sexy, but my boss is cool and it's reliable employment - have been at the same gig over 5 years. I occasionally feel jealous of my peers higher up the stack who get paid a bunch more and/or don't have to explain "how git works" weekly. Day-to-day, I deal with some tedious old school codebases that can be frustrating at times, but I occasionally get to play electronics engineer, and have an ongoing embedded Rust project to keep things fresh.

kody 4y ago
"How git works", especially demystifying merge conflicts, has been a major pain point at all my jobs from embedded to frontend web dev, unfortunately.
- bacon_waffle 4y ago
  I'm a bit torn about it to be honest. On one hand, it's sad/frustrating to work with colleagues who can't be bothered to figure git out. On the other, it is sometimes nice to have a less technically-difficult distraction (or at least one in a different problem space) from what I actually get paid to do. Maybe the right thing is to lean in to it, and become a git consultant! I'm a bit scared to see if that's a thing - would be a sad commentary on the state of the software industry.
  - scruple 4y ago
    FWIW, I'm aware of a handful of people that work for a very large AAA game studio, nearby where I live, who are, in fact, git "experts" -- and, really, just extremely knowledgeable about version control systems, generally speaking, since they don't just use git -- for the company and they spend a large % of their time helping these teams deal with version control issues. As I understand it, they've written a lot of automation to deal with recurring problems like merge conflicts, perforce depot / workspace issues, etc. And they make good money for that expertise...

ramchip 4y ago

I don't get this take... there's plenty of low level system engineers graduating from computer/electronic engineering programs, and coding schools target a very different profile, so I don't think they compete with each other significantly.

eyberg 4y ago

Right here. https://nanovms.com https://nanovms.com If you like kernel level c I'm hiring. The best hackers want to work with the best hackers and we are charting a new course for the 2020s - not one from the 1960s. Despite what you might read in the main-stream dev news our work is extremely important and needs people just like you. You don't have to work at a boring corporate job -- you can instead work at the unikernel company NanoVMs - you just need to email me.

_xrp0 4y ago
Hey, can you speak a bit about how nanos compares to something Like unikraft? IIRC nanos doesn't eliminate syscalls in it's entirety like other unikernels.
- eyberg 4y ago
  We don't think that's a defining characteristic no. We keep the kernel/user cause otherwise you can very very easily change page permissions which means all security goes out the window. You could instantly make rwx on all pages with a few instructions. If you rewind to ~ 2013 most unikernels were talking about trashing the context switch yet depending on the "context" that implies various things (kthread <> kthread, user proc <> user proc, user thread <> user thread, etc...) They all have different costs and while we definitely agree with the aggressive cost of various user process to user process the kernel <> user was one we felt was not worth trashing. Clearly though, because of massive adoption, we see 200% extra throughput on GCP and up to 300% on AWS - so the performance and security remain.

ryukoposting 4y ago

Here! I am a firmware architecture engineer at a Chicago-based startup. I get paid to do embedded firmware, but I like to make little kernels in my spare time.

zubiaur 4y ago

Garmin

rsaxvc 4y ago
It's a living.

walrus01 4y ago

> where are the low level system engineers working on operating systems and software loads for UAS flight controllers, some of them. stuff like an auterion FC is very limited in CPU and RAM.

binarynate 4y ago

I believe ECE (Electrical and Computer Engineering) programs still provide a strong background in low-level systems programming. At least, that was the case when I was at Purdue 9 years ago. My experience with low-level programming has been a boomerang: focused on embedded systems in college, went gradually further up the stack after school due to my interests, and then gradually drifted back down to low-level C++ as I began working on more complex problems.

jlokier 4y ago

I think a lot of people currently have "the knowledge" to do low level system engineering, but I agree with the comments that the job market drives what people pursue, and the job market isn't particularly favourable to low level system engineering. So those skills are getting rusty out of lack of use, and aren't being passed on in the workplace because that's not the sort of work people are doing in the workplace. "Low level" covers many things that don't have much overlap. In the employment market, I see a significant distinction between "embedded"/"firmware" and "server systems engineering". TLDR: Embedded pay is low compared with other software. As far as I can tell, from inbound recruiter mail and job ads, the pay for embedded and firmware work is consistently low compared with what's possible in webdev, cloud, and software generally. Even for "senior" embedded and firmware work. I suspect this is because it's adjacent to things like FPGAs and electronics, which also pay low compared with software engineering generally. By low, I mean senior embedded pay at about 60% of senior webdev pay. On the other hand, seasoned "system level" engineering such as kernels, compilers, database engines, GPU drivers, storage, crypto, and many other such things, especially things that run on servers, can pay well and seems to be in some demand, especially at big companies. (Device drivers can fall into either category. For example a GPU or storage subsystem driver is at a completely different level from a USB serial driver.)

tintedfireglass 4y ago

I am a college student who loves writing low level and reading this thread discourages me :(

KerrAvon 4y ago
Why? What do you want to work on? FAANGS are certainly hiring “low-level” engineers. Apple is hiring and has relatively low-bullshit interviews (depending on the team).
- astrange 4y ago
  In fact, Apple has lots of open jobs for exactly the kind of low-level engineers this thread is about and you can find the team managers posting about them on Twitter.
  - tintedfireglass 4y ago
    Reading this thread made me feel like all the embedded jobs are low paying or moved to China. And whatever is left has fierce competition. That's why my parent comment.
    - astrange 4y ago
      I think most people on HN are web developers who only talk to other web developers and think the only good jobs in the world are at Facebook. There's a definite bias. Of course, there's a few more problems than low paying or moved to China - it might be on-site somewhere expensive (Apple), somewhere you don't want to live (probably a lot of these), or boring or unethical (a defense contractor).
- guessmyname 4y ago
  > Apple is hiring and has relatively low-bullshit interviews (depending on the team). I agree. I passed interviews for a Systems Software Engineer position at Apple two months ago, and I can attest that the interview process was very down to Earth. No LeetCode questions. Instead, the interviews were mostly system design and and simple programming exercises where finding a solution was less important than the discussion with the interviewers. If anyone is interested, Apple is still hiring - https://jobs.apple.com/en-us/search?search=systems%20engineer&sort=relevance https://jobs.apple.com/en-us/search?search=systems%20enginee...

da39a3ee 4y ago

From what I gather it doesn't pay as well as other engineering subgenres, so presumably the market is telling us we don't actually need that many?

Matthias247 4y ago

Somewhat here. But to be honest, it's a tricky place to be. As an individual who is very proficient in this areas, I get rarely any reachouts about those skills. 90% of LinkedIn recuiter messages inquire about generic high level programming projects. Then there's 8% crypto related things, and a very tiny amount of other systems level engineering roles. On the other hand trying to hire someone for such a role is super hard. Most engineers don't deal with low level details and systems programming languages anymore. Compensation wise the domain does not be better than others. While one assumes it might be more specialized, that seems by far offset by the amount of lower opportunities and thereby less chances to negotiate compensation than in other software engineering parts.

bibabaloo 4y ago
I think another big reason for lower compensation is because a lot of companies working at a low level tend to be companies making physical electronic products. Profits for physical profits don't scale as exponentially as they do with software only products, so companies don't have endless amounts of cash to throw around at engineers. If you're after good compensation and want to work at a low level, then go for companies that sell both hardware and software. FAANGs are an obvious choice but B2B companies (usually IoT-ish) that have have both a software and hardware subscription component are another option worth looking at.
scruple 4y ago
> Compensation wise the domain does not be better than others. Yep. I moved into web development because it paid a lot more and carried a lot less stress. The 7 years I spent working on embedded systems paid off in a major way, too, because I showed up equipped to directly address performance problems. I'm still in web development and I'll probably never leave.

c_o_n_v_e_x 4y ago

Any recommended sites for finding/hiring embedded talent? I had A LOT of trouble trying to find someone for a driver development project a few years ago.

gsempe 4y ago
I’m doing Embedsys Weekly. With a job offer, you can post the opportunity on the job board jobs.embedsysweekly.com

rramadass 4y ago

The Software Industry is headed off a cliff. People who actually take care of the low-level plumbing are not being incentivized nor given their due. Expertise and Hard-won knowledge is being lost. There is also not much ground-breaking research going on in these domains which doesn't bode well for the future. As a final nail in the coffin, Colleges/Universities are not even creating/training the next generation of "Low-Level System Engineers". We are going to end-up with a handful of Companies holding the keys to the kingdom and holding Society-at-large hostage.

Balgair 4y ago

Since there are a lot of C/++ people in this thread, I figured I should swing this job posting at y'all: https://flagshipbio.bamboohr.com/jobs/view.php?id=79&source=flagshipbio https://flagshipbio.bamboohr.com/jobs/view.php?id=79&source=... I'm in no way affiliated with the company, but my friend is (honest!). They have had a heck of a time finding a person for the role. Comp maxes out at ~$120k, remote is just fine. If anyone is good with C++ and would like to transition to the bio side of things, this role may be for you.

lacker 4y ago
I mean, if comp maxes out at $120k for this job where you want an engineer with mid-level experience, and Google is giving $190k total comp to new graduates who have no experience, what do you expect? The compensation is way below market for a good, experienced C++ engineer.
zwerdlds 4y ago
>Basic Qualifications > • PhD degree in a field related to Image Analysis, or equivalent work experience I think I know why.
- Balgair 4y ago
  Any other things that stand out to you? Impartial feedback may be good for my buddy to see.

sebazzz 4y ago

Well I'd love to write C++ as my day job, but it is not easy to get into a C++ job here in the Netherlands. There aren't a lot of native code positions, and those who are pay generally less than I get now or have an unclear offering. Once you take a career path, like C# web development in my case, you generally keep doing that unless you are attracted by a company which really needs you or believes in you. So my C++ experience is limited to side projects like Marlin based firmware and that's it.

jstream67 4y ago

I got excited because I thought this was talking about linux systems engineers and how that is a almost non-existent career path these days. I would call people who write c/c++ code 'programmers' - interesting to see that title is not universal. If you are wondering where all the c/c++ programmers (system engineers) are - they are in the defense industry - or at least all the ones that I know are.

glonq 4y ago

I work in a place that desperately needs a senior systems/embedded engineer, and our upper leadership is willing to throw so much money at that position that I'm jokingly suggesting that I should demote myself from Director back down to Sr Developer so that I can earn more pay and suffer less bullshit.