7 ms·

Last month I switched from a role working on a distributed system (FAANG) to a role working on embedded software which runs on cards in data center racks. I wa

• jasonjayr 2y ago
  > Whenever I would bring up this gap I would be told that we can't spent time and wait for people to create "magic tools". That sounds like an awful organizational ethos. 30hrs to make a "magic tool" to save 300hrs across the organization sounds like a no-brainer to anyone paying attention. It sounds like they didn't even want to invest in out-sourced "magic tools" to help either.
  • bsnnkv 2y ago
    The real kicker is that it wasn't even management saying this, it was "senior" developers on the team. I wonder if these roles tend to attract people who get the most job enjoyment and satisfaction out of the (manual) investigation aspect; it might explain some of the reluctance to adopting or creating more sophisticated observability tooling.
    • zelphirkalt 2y ago
      Senior doesn't always mean smarter or more experienced or anything really. It just all depends on the company and its culture. It can also mean "worked for longer" (which is not equal to more experienced, as you can famously have 10 times 1y experience, instead of 10y experience) and "more aligned with how management at the company acts".
      • bongodongobob 2y ago
        I'd probably take 10x 1y experience. Where I'm at now, everyone has been with the company 10-40 years. They think the way they do things is the only way because they've never seen anything else. I have many stories similar to the parent. They are a decade behind in their monitoring tooling, if it even exists at all. It's so frustrating when you know there are better ways.
        • HPsquared 2y ago
          "10x1y" means someone did the same thing for 10 years with no change or personal development. The learning stopped after the first year which then repeated Groundhog Day style.
          • bongodongobob 2y ago
            Ah, I misunderstood.
            • lazystar 2y ago
              another term for the phenomena is the "expert beginner" trap.
            • Nevermark 2y ago
              I see a dual. Between 10x1 workers and 1x10 workers working at 10x1 companies. Either way, doing the same kinds of things, the same kind of ways, more than a few times, is an automation/tool/practice improvement opportunity lost. I have yet to complete a single project I couldn't do much better, differently, if I were to do something similar again. Not everything is high creative, but software is such a complex balancing act/value terrain. Every project should deliver some new wisdom, however modest.
      • fuzztester 2y ago
        I have heard it as 20 versus 1, but it is the same thing. also called by some other names, including NIH syndrome, protecting your turf, we do it this way around here, our culture, etc.
    • zero_shift 2y ago
      Why would people who are good at [scarce, valuable skill] and get paid [many bananas] to practice it want to even imagine a world where that skill is now redundant? ;-)
      • filoleg 2y ago
        The real skill is “problem-solving”, not “doing lots of specific manual steps that could be automated and made easier.” Unfortunately, some people confuse the two and believe they are paid to do the latter, not the former, simply because others look at those steps and go “wtf, we could make that hell more pleasant and easier to deal with”. In the same vein, “creating perceived job security for yourself by willing to continuously deal with stupid bs that others rightfully aren’t interested in wasting time on.” Sadly, you are ultimately right though, as misguided self-interest often tends to win over well-meant proposals.
        • fc417fc802 2y ago
          If the goal is ensuring a future stream of bananas then can you really say the behavior is misguided?
    • Henchman21 2y ago
      IME, “senior” often means “who is left after the brain-drain & layoffs are done” when you’re at a medium sized company that isn’t prominent.
      • getnormality 2y ago
        I feel so seen.
    • the_sleaze_ 2y ago
      _To play devils advocate_: It could've sounded like the "new guy" came in and decided he needed to rewrite everything; bring in new xyx; steer the ship. New guy could even have been stepping directly on the toes of those senior developers who had fought and won wars to get were they are now. In my -very- humble opinion, you should wait at least a year before making big swinging changes or recommendations, most importantly in any big company.
      • jiggawatts 2y ago
        In my less humble opinion: the only honest and objective review you’ll get about a system is from a new hire for about a month. Measure the “what the fucks per hour” as a barometer of how bad your org is and how deep a hole it has dug itself into. After that honeymoon period, all but the most autistic people will learn the organisational politics, keep their head down, and “play the game” to be assigned trivial menial tasks in some unimportant corner of the system. At that point, only after two beers will they give their closest colleagues their true opinion. I’ve seen this play out over and over, organisation after organisation. The corollary is that you yourself are not immune to this effect and will grow accustomed to almost any amount of insanity. You too will find yourself saying sentences like “oh, it always has been like this” and “don’t try to change that” or “that’s the responsibility of another team” even though you know full well they’re barely even aware of what that thing is, let alone maintaining it in a responsible fashion. PS: This is my purpose in a nutshell as a consultant. I turn up and provide my unvarnished opinion, without being even aware of what I’m “not supposed to say” because “it upsets that psychotic manager”. I’ll be gone before I have any personal political consequences, but the report document will remain, pointing the finger at people that would normally try to bite it off.
        • disqard 2y ago
          This rings true in my experience across different orgs, teams, in the tech industry. FWIW, academia has off-the-charts levels of "wtf" that newcomers will point out, though it's even more ossified than corporate culture, and they don't hire consultants to come in and fix things :)
    • Jach 2y ago
      There's also immense resistance to figuring out how to code something if an approach isn't at once obvious. Hence "magic". Sometimes a "spike doc" can convince people. My favorite second-hand instance of this was a MS employee insisting that a fast rendering terminal emulator was so hard as to require "an entire doctoral research project in performant terminal emulation".
    • scottlamb 2y ago
      > I wonder if these roles tend to attract people who get the most job enjoyment and satisfaction out of the (manual) investigation aspect; it might explain some of the reluctance to adopting or creating more sophisticated observability tooling. That's weird. I love debugging, and so I'm always trying to learn new ways to do it better. I mean, how can it be any other way? How can someone love something and be that committed to sucking at it?
    • whstl 2y ago
      I saw a case like this recently, and the fact is that the team responsible was completely burned out and was just doing anything to avoid people from giving them more work, but they also didn't trust anyone else to do it. One of the engineers just quit on the spot for a better paid position, the other was demoted and is currently under heavy depression last I heard from him.
  • cmrdporcupine 2y ago
    Consider that there is a class of human motivation / work culture that considers "figuring it out" to be the point of the job and just accepts or embraces complexity as "that's what I'm paid to do" and gets an ego-satisfaction from it. Why admit weakness? I can read the logs by timestamp and resolve the confusions from the CAP theorem from there! Excessive drawing of boxes and lines, and the production of systems around them becomes a kind of Glass Bead Game. "I'm paid to build abstractions and then figure out how to keep them glued together!" Likewise, recomposing events in your head from logs, or from side effects -- that's somehow the marker of being good at your job. The same kind of motivation underlies people who eschew or disparage GUI debuggers (log statements should be good enough or you're not a real programmer), too. Investing in observability tools means admitting that the complexity might overwhelm you. As an older software engineer the complexity overwhelmed me a long time ago and I strongly believe in making the machines do analysis work so I don't have to. Observability is a huge part of that. Also many people need to be shown what observability tools / frameworks can do for them, as they may not have had prior exposure. And back to the topic of the whole thread, too: can we back up and admit that distributed systems is questionable as an end in itself? It's a means to an end, and distributing something should be considered only as an approach when a simpler, monolithic system (that is easier to reasona bout) no longer suffices. Finally I find that the original authors of systems are generally not the ones interested in building out observability hooks and tools because for them the way the system works (or doesn't work) is naturally intuitive because of their experience writing it.
• DaiPlusPlus 2y ago
  > I switched from a role working on a distributed system [...] to embedded software which runs on cards in data center racks Would you agree that, technically (or philosophically?) that both roles involved distributed systems (e.g. the world-wide-web of web-servers and web-browsers exists as a single distributed system) - unless your embedded boxes weren't doing any network IO at all? ...which makes me genuinely curious exactly what your aforementioned distributed-system role was about and what aspects of distributed-computing theory were involved.
• anonzzzies 2y ago
  I really love embedded work; at least it gives you the feeling that you have control over things. Not everything being confused and black boxed where you have to burn a goat to make it work, sometimes.
  • porridgeraisin 2y ago
    > where you have to burn a goat to make it work, sometimes. Or talk to a goat, sometimes https://modernfarmer.com/2014/05/successful-video-game-developer-owe-goats/ https://modernfarmer.com/2014/05/successful-video-game-devel...
• beoberha 2y ago
  Yep - I’ve very much been living the former for almost a decade now. It is especially difficult when the components stretch across organizations. It doesn’t quite address what the author here is getting at, but it does make me believe that this new programming model will come from academia and not industry.
• EtCepeyd 2y ago
  This resonates a lot with me. Distributed systems require insanely hard math at the bottom (paxos, raft, gossip, vector clocks, ...) It's not how the human brain works natively -- we can learn abstract thinking, but it's very hard. Embedded systems sometimes require the parallelization of some hot spots, but those are more like the exception AIUI, and you have a lot more control over things; everything is more local and sequential. Even data race free multi-threaded programming in modern C and C++ is incredibly annoying; I dislike dealing with both an explicit mesh of peers, and with a leaky abstraction that lies that threads are "symmetric" (as in SMP) while in reality there's a complicated messaging network underneath. Embedded is simpler, and it seems to require less that practitioners become advanced mathematicians for day to day work.
  • Thaxll 2y ago
    It does not requires any math because 99.9% of the time the issue is not in the low level implementation but in the business logic that the dev did. No one goes to review the transaction engine of Postgress.
    • EtCepeyd 2y ago
      I tend to disagree. - You work on postgres: you have to deal with the transaction engine's internals. - You work in enterprise application intergration (EAI): you have ten legacy systems that inevitably don't all interoperate with any one specific transaction manager product. Thus, you have to build adapters, message routing and propagation, gateways, at-least-once-but-idempotent delivery, and similar stuff, yourself. SQL business logic will be part of it, but it will not solve the hard problems, and you still have to dig through multiple log files on multiple servers, hoping that you can rely on unique request IDs end-to-end (and that the timestamps across those multiple servers won't be overly contradictory). In other words: same challenges at either end of the spectrum.
      • pfannkuchen 2y ago
        Yeah this is kind of an abstraction failure of the infrastructure. Ideally the surface visible to the user should be simple across the entire spectrum of use cases. In some very, very rare cases one necessarily has to spelunk under the facade and know something about the internals, but for some reason it seems to happen much more often in the real world. I think people often don't put enough effort into making their system model fit with the native model of the infrastructure, and instead torture the infrastructure interface (often including the "break glass" parts) to fit into their a priori system model.
• Scramblejams 2y ago
  I've often heard embedded is a nightmare of slapdashery. Any tips for finding shops that do it right?
  • api 2y ago
    A lot of times it is, but it's not your fault. It's the fault of vendors and/or third party code you have to use.
  • DanielHB 2y ago
    There is inherent complexity and self-inflicted complexity, they tend to go hand in hand but self-inflicted complexity can be exacerbated in bad projects. A lot of embedded software is just inherent complex, cars for example.
  • AlotOfReading 2y ago
    It's not foolproof, but I've found there's a strong correlation between product margin and the sanity of the dev experience.
• alabastervlog 2y ago
  I've found the rush to distributed computing when it's not strictly necessary kinda baffling. The costs in complexity are extreme. I can't imagine the median company doing this stuff is actually getting either better uptime or performance out of it—sure, it maybe recovers better if something breaks, maybe if you did everything right and regularly test that stuff (approximately nobody does though), but there's also so very much more crap that can break in the first place. Plus: far worse performance ("but it scales smoothly" OK but your max probable scale, which I'll admit does seem high on paper if you've not done much of this stuff before, can fit on one mid-size server, you've just forgotten how powerful computers are because you've been in cloud-land too long...) and crazy-high costs for related hardware(-equivalents), resources, and services. All because we're afraid to shell into an actual server and tail a log, I guess? I don't know what else it could be aside from some allergy to doing things the "old way"? I dunno man, seems way simpler and less likely to waste my whole day trying to figure out why, in fact, the logs I need weren't fucking collected in the first place, or got buried some damn corner of our Cloud I'll never find without writing a 20-line "log query" in some awful language I never use for anything else, in some shitty web dashboard. Fewer, or cheaper, personnel? I've never seen cloud transitions do anything but the opposite. It's like the whole industry went collectively insane at the same time. [EDIT] Oh, and I forgot, for everything you gain in cloud capabilities it seems like you lose two or three things that are feasible when you're running your own servers. Simple shit that's just "add two lines to the nginx config and do an apt-install" becomes three sprints of custom work or whatever, or just doesn't happen because it'd be too expensive. I don't get why someone would give that stuff up unless they really, really had to. [EDIT EDIT] I get that this rant is more about "the cloud" than distributed systems per se, but trying to build "cloud native" is the way that most orgs accidentally end up dealing with distributed systems in a much bigger way than they have to.
  • throwawaymaths 2y ago
    the minute you have a client (browser, e.g.) and a server you're doing a distributed system and you should be thinking a little bit about edge cases like loss of connection, incomplete tx. a lot of the goto protocols (tcp, http, even stuff like s3) are built with the complexities of distributed systems in mind so for most basic cases, a little thought goes a long way. but you get weird shit happening all the time (that may be tolerable) if you don't put any effort into it.
• bob1029 2y ago
  > Whenever I would bring up this gap I would be told that we can't spend time/money and wait for people to create "magic tools". I've never once been granted explicit permission to try a different path without being burdened by a mountain of constraints that ultimately render the effort pointless. If you want to try a new thing, just build it. No one is going to encourage you to shoot holes through things that they hang their own egos from.
  • DrFalkyn 2y ago
    Hope you can justify that during sprint planning / standup
    • bob1029 2y ago
      If you are going to just build it in the absence of explicit buy-in, you certainly shouldn't spend time on the standup talking about it. Wait until your idea is completely formed and then drop a 5 minute demo on the team. It can be challenging to push through to a completed demo without someone cheering you on every morning. I find this to be helpful more than hurtful if we are interested in the greater good. If you want to go against the grain (everyone else on the team), then you need to be really sure before you start wasting everyone else's time. Prove it to yourself first.
• im_down_w_otp 2y ago
  We built a bunch of tools & technology for leveraging observability (docs.auxon.io) to do V&V, stress testing, auto root-cause analysis, etc. in clusters of embedded development (all of it built in Rust too :waves: ), since the same challenges exist for folks building vehicle platforms, lunar rovers, drones, etc. Both within a single system as well as across fleets of systems. Many embedded developers are actually distributed systems developers... they just don't think of it that way. It's often quite a challenge to get that class of engineer to adopt things that give them visibility and data to track things down as well. Sometimes it's just a capability/experience gap and sometimes it's just over indexing on a perception of time getting to a solution vs. the time wasted on repeated problems and yak shavings.
• lumost 2y ago
  Anecdotally, I see a major under appreciation for just how fast and efficient modern hardware is in the distributed systems community. I’ve seen a great many engineers become so used to provisioning compute that they forget that the same “service” can be deployed in multiple places. Or jump to building an orchestration component when a simple single process job would do the trick.
• ithkuil 2y ago
  10 years ago I went on a similar journey. I left faang to work on a startup working on embedded firmware for esp8266. The lack of tooling was very frustrating. I ended up writing a gdb stub (before espressif released one) and a malloc debugger (via serial port) just to manage to get shit done.
• yolovoe 2y ago
  Is the “card” work EC2 Nitro by any chance? Sounds similar to what I used to do
• bryanlarsen 2y ago
  I think you were unlucky in your distributed system job and lucky in your embedded job. Embedded is filled with crappy 3rd party and in-house tooling, far more so than distributed, in my experience. That crappiness perhaps leads to a higher likelihood to spend time on them, but it doesn't have to. Embedded does give you a greater feeling of control. When things aren't working, it's much more likely to be your own fault.
• bagels 2y ago
  Which company? Doesn't sound like the infra org I was in at a FAANG
• englishspot 2y ago
  curious as to how you made that transition. seems like that'd be tough in today's job market.
• alfiedotwtf 2y ago
  I have talked to many people in the Embedded space doing Rust, and every single one of them had the biggest grin while talking about work. Sounds like you’ll have fun :)
• intelVISA 2y ago
  Distributed systems always ends up a dumping ground of failed tech solutions to deep org dysfunction. Weak tech leadership? Let's "fix" that with some microservices. Now it's FUBAR? Conceal it with some cloud native horrors, sacrifice a revolving door of 'smart' disempowered engineers to keep the theater going til you can jump to the next target. Funny because dis sys is pretty solved since Lamport, 40+ years ago.
  • rbjorklin 2y ago
    Would you mind sharing some more specific information/references to Lamport’s work?
    • vitus 2y ago
      The three big papers: clocks [0], Paxos [1], Byzantine generals [2]. [0] https://lamport.azurewebsites.net/pubs/time-clocks.pdf https://lamport.azurewebsites.net/pubs/time-clocks.pdf [1] https://lamport.azurewebsites.net/pubs/lamport-paxos.pdf https://lamport.azurewebsites.net/pubs/lamport-paxos.pdf [2] https://lamport.azurewebsites.net/pubs/byz.pdf https://lamport.azurewebsites.net/pubs/byz.pdf Or, if you prefer wiki articles: https://en.wikipedia.org/wiki/Lamport_timestamp https://en.wikipedia.org/wiki/Lamport_timestamp https://en.wikipedia.org/wiki/Paxos_(computer_science) https://en.wikipedia.org/wiki/Paxos_(computer_science) https://en.wikipedia.org/wiki/Byzantine_fault https://en.wikipedia.org/wiki/Byzantine_fault I don't know that I would call it "solved", but he certainly contributed a huge amount to the field.
    • madhadron 2y ago
      Lamport's website has his collected works. The paper to start with is "Time, clocks, and the ordering of events in a distributed system." Read it closely all the way to the end. Everyone seems to miss the last couple sections for some reason.
  • whstl 2y ago
    I suffered through this in two companies and man, it isn't easy. First one was a multi-billion-Unicorn had everything converted to microservices, with everything customized in Kubernetes. One day I even had to fix a few bugs in the service mesh because the guy who wrote it left and I was the only person not fighting fires able to write the language it was in. I left right after the backend-of-the-frontend failed to sustain traffic during a month where they literally had zero customers (Corona). At the second one there was a mandate to rewrite everything to microservices and it took another team 5 months to migrate a single 100-line class I wrote into a microservice. It just wasn't meant to be. Then the only guy who knows how the infrastructure works got burnout after being yelled at too many times and then got demoted, and last I heard is at home with depression. Weak leadership doesn't even begin to describe it, especially the second. But remembering it is a nice reminder that a job is just a means of getting a payment.
• sly010 2y ago
  I don't disagree, but funny that I recently made a point to someone that modern consumer embedded systems (with multiple MCUs connected with buses and sometimes shared memory) are basically small distributed systems, because partial restarts are common and the start/restart order of the MCUs is not very well defined. At least in the space I am working in. (Needless to say we use C, not rust)
• fons 2y ago
  Would you mind disclosing your current employer? I am also interested in moving to an embedded systems role.
• fra 2y ago
  As someone who builds observability tools for embedded software, I am flabbergasted that you're finding a more tools-friendly culture in embedded than in distributed systems! Most hardware companies have zero observability, and haven't yet seen the light ("our code doesn't really have bugs" is a quote I hear multiple times a week!).
  • whstl 2y ago
    It's probably a "grass is greener" situation. My experience with mid-size to enterprise is having lots of observability and observability-adjacent tools purchased but not properly configured. Or the completely wrong tools for the job being used. A few I've seen recently: Grafana running on local Docker of developers because of lack of permissions in the production version (the cherry on top: the CTO himself installed this on the PMs computers), Prometheus integration implemented by dev team but env variables still missing after a couple years, several thousand a month being paid to Datadog but nothing being done with the data nor with the dog. On startups it's surprisingly different, IME. But as soon as you "elect" a group to be administrator of a certain tool or some resource needed by those tools, you're doomed.
• lelanthran 2y ago
  I spent the majority of my career as an embedded dev. There are ... different ... challenges, and I'm not so sure that I would want to go back to it. It pays poorly, the tooling more often than not sucks (more than once I've had to do some sort of stub for an out-of-date gcc), observability is non-existent unless you're looking at a device on your desk, in which case your observability tool is an oscilloscope (or bus pirate type of device, if you're lucky in having the lower layers completely free of bugs). The datasheets/application notes are almost always incomplete, with errata (in a different document) telling you "Yeah, that application note is wrong, don't do that". The required math background can be strict as well: R/F, analog ... basically anything interesting you want to do requires a solid grounding in undergrad maths. I went independent about 2 years ago. You know what pays better and has less work? Line of business applications. I've delivered maybe two handfuls of LoB applications but only one embedded system, and my experience with doing that as a contractor is that I won't take an embedded contract anymore unless it's a client I've already done work for, or if the client is willing to pay 75% upfront, and they agree to a special hourly rate that takes into account my need for maintaining all my own equipment.
• junon 2y ago
  Can concur, I also switched mostly to firmware and have enjoyed it much more. Though Rust firmware jobs are hard to come by.
• fatnoah 2y ago
  > The technical aspect that contributed the most to this burnout was both the lack of observability tooling and the lack of organizational desire to invest in it. One of the most significant "triumphs" of my technical career came at a startup where I started as a Principal Engineer and left as the VP Engineering. When I started, we had nightly outages requiring Engineering on-call, and by the time I left, no one could remember a recent issue that required Engineers to wake up. It was a ton of work and required a strong investment in quality & resilience, but even bigger impact was from observability. We couldn't afford APM, so we took a very deliberate approach to what we logged and how, and stuffed it into an ELK stack for reporting. The immediate benefit was a drastic reduction in time to diagnose issues, and effectively let our small operations team triage issues and easily identify app vs. infra issues almost immediately. Additionally, it was much easier to identify and mitigate fragility in our code and infra. The net result was an increase in availability from 98.5% to 99.995%, and I think observability contributed to at least half of that.
• literallyroy 2y ago
  How did you make that transition/find a position? Were you already using Rust in a previous role?