12 ms·

I try to answer “how to become a systems engineer”

367 points by picture 3y ago

jspaetzel 3y ago
Run Linux as your desktop.
- harry8 3y ago
  Too easy nowadays. Pick an esoteric BSD?
  - nathants 3y ago
    it’s not about difficulty. it’s about building useful intuition to guide your curiosity the next time an eyebrow goes up. bsd is used in prod only slightly more than macos.
    - harry8 3y ago
      Nah it is. When something "just works" you don't learn much about how. When your graphics card driver shits itself, you learn something getting it working again. Console text editor at the minimum! It's not "misery is good for you" but having to solve random problems is an opportunity for learning the guts you otherwise don't get. Linux on a laptop used to fill this niche nicely. Does it still?
      - nathants 3y ago
        does still. run alpine for increased difficulty.
        harry8 3y ago
        running a mainstream distro & desktop env on a thinkpad t480s is significantly easier, more efficient and less fragile than any windows laptop I've ever used, seen or heard of. Less signficantly so but still easier than any apple laptop I've used. Genuinely so. YMMV.
        nathants 3y ago
        the primary difficulty isn’t supposed to be the os, but the software you’re building. though os difficulties will always come up, depending on workload. being on linux means you gain intuition debugging locally. whatever you might be debugging. to sum up my point: linux. never not be running it. to your point, linux is a better experience now than before. this is for great good, not a cause to search for a more brittle os and distro. at eod, os isn’t that cool. software is cool.
        harry8 3y ago
        Using an operating system you have to tinker with to get working right, and even fix when it breaks means you learn a lot about the low level of what an operating system is, and how this one does it. That information translates to being useful writing and fixing other software. The point is this, if you're a student, consider using something unusual that is more brittle. Maybe a sweet spot is running mainstream linux on your mainstream laptop (Lenovo T-series is one of the right answers there, dell is the wrong answer) and have a project using a pine rock64 SBC running linux to drive your tv screen or similar? Something that you want to use, that breaks at random in random places that you then have to investigate to work out how to fix. You really learn something keeping brittle systems functional. Linux laptops with nvidia gfx cards definitely used to qualify in ways a t480s really doesn't nowadays. When fixing things that break, strace, gdb, perf, ftrace, BPF, write or fix a kernel device driver, understand what the OS and compiler don't do for you, what they often will do for you that maybe you don't want them to.
        nathants 3y ago
        you may be right. i’m not sure i would recommend someone to pursue a brittle setup, but it might be more educational.
        nathants 3y ago
        my current brittle setup is postmarketos on oneplus 6t. definitely kind of works. the rest of my setups are non-brittle.
Balgair 3y ago
Aside: Can software based systems engineers please call themselves software systems engineer. I've been a 'traditional' systems engineer for many years and when searching for jobs (or with recruiters), it's always a mess trying to filter out the software only systems engineering gigs. I use a lot of software in my work too, it's just that I also have to do 'hardware' by the end of the work. It's not a big ask (I think), but man, would it make a lot of sense.
- 0xbadcafebee 3y ago
  We're still trying to convince hiring managers to stop listing positions for "DevOps Engineer", which is akin to "Culture Engineer".
ftxbro 3y ago
> malloc("HTTP")
- dilyevsky 3y ago
  Dont try this at home. Reminds me of james mickens pointer equals to 7 thing
  - vvanders 3y ago
    https://www.usenix.org/system/files/1311_05-08_mickens.pdf https://www.usenix.org/system/files/1311_05-08_mickens.pdf "... All of these people will play a role in my ultimate success as a dystopian warlord philosopher. However, the most important person in my gang will be a systems programmer." It's right there in the intro, truly a classic.
PaulCarrack 3y ago
> malloc(1213486160) is really malloc(0x48545450) is really malloc("HTTP") No it's not / that's not how C works. malloc("HTTP") is malloc of the address of a string literal in memory. If "HTTP" is at address 0x11223344 then it's malloc(0x11223344). So it's the address of where ever the "HTTP" string is in the binary not the characters themselves. It's kind of a weird way to say that you can convert decimal to hexadecimal in your head. I also wouldn't use malloc as the example to demonstrate that talent, it makes no sense that way. Perhaps becoming a systems engineer should involve learning C first... Don't always trust what you read.
- anonymoushn 3y ago
  malloc takes a size_t indicating the desired size of the allocation. If malloc is accidentally being called with a pointer instead that would be an error, but this story is about an instance where malloc was accidentally being called with the first bits of a string instead, apparently on a big-endian system where size_t is 32 bits. Hope that helps.
  - PaulCarrack 3y ago
    > If malloc is accidentally being called with a pointer instead that would be an error Not true, just tried it in MSVC and it compiled fine but with a warning on conflicting types. "HTTP" is of type const char* and sizeof(size_t) == sizeof(const char *). Doesn't matter if its 32 or 64, I'm not aware of a platform where that isn't the case. > but this story is about an instance where malloc was accidentally being called with the first bits of a string instead If that's the case, then it should have been something along the lines of *"HTTP" instead.
    - anonymoushn 3y ago
      > Not true, just tried it in MSVC and it compiled fine but with a warning on conflicting types. If I meant that it would cause the compiler to report an error then I would have written that. > If that's the case, then it should have been something along the lines of *"HTTP" instead. We can say that they really should have written memcpy to get the string literal into the size_t or we can accept that sometimes people use language imprecisely and expect readers to make an effort to understand rather than make an effort not to understand.
rukuu001 3y ago
I don't know if the article is a helpful answer to the question, but it's a very true answer I think. My path to 'doing' systems engineering for a time (it really wasn't my thing) was by maintaining software an actual sys. eng. had written, fixing problems with it, extending it and eventually becoming responsible for it. Best lol - API of a library we were using changed so that instead of taking a pointer to a struct to allocate memory for, it just wanted an amount of memory to allocate. We didn't notice, it passed all our tests (sheer luck) and dumped core on prod all day long. Funny.
- S_A_P 3y ago
  — We didn't notice, it passed all our tests (sheer luck) and dumped core on prod all day long. If this happened to me I’d not call that luck, I would want to know why my tests missed this.
  - erdo 3y ago
    That's sarcasm, it probably also wasn't actually Funny. Nothing about what they said suggests they didn't subsequently look into why their tests missed the issue
prpl 3y ago
I’ve worked with systems engineers, as in the modeling/SysML type. It is important to note that software is usually a part of a much bigger system, usually physical, and that systems engineering is kind of an engineering approach to project management to help in ensuring things get built to spec (even if the spec changes). The nice thing about that is you don’t get VPs/execs walking in the door saying what you’re going to do this quarter instead.
anotherhue 3y ago
I like the article at lot, I've got a similar background. My answer to the question would be: start explaining what you're asking the OS to do when you run your program. There's an entire category of developers who view the IDE and the language as a perfect hermetically sealed environment, until they meet a leaky abstraction. Some favourites: 1. Depleting the outbound TCP ports because ports aren't infinite and the code didn't know that. 2. Any nodejs BS that involves a precompiled C lib, where homebrew or some nonsense 'just works' on the dev machine but prod runs alpine/musl. NPM sneaking arch specific blobs in is really to blame here but what do we expect from the failed state of package managers. 3. Assuming latency is not a thing. Network calls everywhere. No management of timeouts / killed connections. 4. Memory, in all shapes and forms. Most typically thinking a GC prevents memory leaks, but more generally, having no clue whatsoever what the memory usage profile of the application is. A systems engineer (IMO) has much better answers to these questions than a regular developer, and it is absolutely mostly arcane nonsense, (ulimit anyone?), until it isn't. Personally I love OS tech so maybe I'm just cut out for it. I used to be able to tell what the machine was doing from the HDD whirrs and still have a 'feel' for system latency, e.g. when something completes too quickly.
- zorked 3y ago
  > I used to be able to tell what the machine was doing from the HDD whirrs I guess that dimension of debugging is now lost with SSDs. I didn't run an anti-virus and noticed by machine had caught a virus (in this case, DOS/Natas) because of the extra moments of whirrs during boot and when running applications.
  - beedeebeedee 3y ago
    I've been doing some audio work simulating audio equipment, and I bet you could pipe out some sound for disk writes and use that as a debug tool.
    - anotherhue 3y ago
      https://www.retrokits.de/index.php/hdd-clicker-hdd-sound-simulator-v0-1/ https://www.retrokits.de/index.php/hdd-clicker-hdd-sound-sim...
kodah 3y ago
I liked the article. It kind of taps into the complexity of what goes into Systems Engineering. I always describe the job as a software engineer who also understands OS internals, APIs, and tooling and about all the external things that they depend on.
analog31 3y ago
>>> I think it goes something like this: you start from the assumption that when you see something, you wonder why it is the way it is. Then maybe you observe it and maybe do a little research to figure out how it came to be the thing you see in front of you. This could go for just about anything: a telephone, a scale, a crusty old road surface, a forgotten grove of fruit trees, you name it. By research, I mean maybe you go poking around: try to open that scale with a screwdriver, get out of the car and walk down the old road, or turn over some of the dirt in the field to see if you can find any identifying marks. ... a scientist. My first job title at my current workplace was "systems engineer." My degree is in physics. I evolved into the role of being the person on the project who understands and can explain how the whole thing works. It doesn't mean just dictating a theory of operation, but also testing that theory. My job is in an area where products involve both hardware and software. I certainly don't claim to know every design detail. Part of the trick is figuring out what things I actually do need to know, and how to nail those things down.
- josephg 3y ago
  Yep. I found a bug the other week which was slowing some important code down by 20x. I found the problem because it was tweaking my intuition. It just didn’t make sense that this code was so slow. After a few nights lying in bed thinking about it, I did some back of the napkin math which agreed with my intuition. Eventually I took some time to track it down. It turned out a rogue debug assertion hidden in a big function was making it into production. The assertion did a deep scan of the system to check invariants, and that turned an O(n) into an O(n^2) in some cases. This sort of thing comes up a lot while programming. Something weird happens and you start to wonder - why does it behave like this? Answering these questions has led me all sorts of places - to reading specs, debugging into databases, compiling my own Firefox, all sorts of things. The chance this knowledge pays off sooner or later is wildly high. Computers are deterministic and debugable. You usually have the source code and everything it does was designed and written by people like you. A curious, scientific attitude is a beautiful thing to nurture in software.
  - alex-moon 3y ago
    I actually had a case like this earlyish in my career. I had a job where my main responsibility was maintaining and developing large ETL scripts for an online book store (so hundreds of millions of rows). We had one script which did the meat of the work which was running many hours each night. My job was to speed it up. Watching it run for a bit, it occurred to me to wonder how long it took to send each line of progress info (couple of lines each for every single row) to the log. I remember my boss at the time didn't believe me. "I think it's logging," I said, and turned logging off just for funsies. Boom, 100x faster.
scsteps 3y ago
Systems engineering is actually an overloaded term and that Wikipedia article is correct. The author’s definition is also correct of course. Just talking about different things.
- grepfru_it 3y ago
  What I thought hilarious, when the author was describing a series of tools that were maintained for a long period of time.. that was the complex system. No attempts were made to reduce the complexity, because the cost was not worth the increased value. Maybe that specific part, realizing the business driver of systems, is what makes a Systems Engineer. And even though your specific application won't EOL, there are many times where features are deprecated and that complex system must be altered. Maybe I'm just grizzled in my old age, but it seems like this author was writing about Systems Engineering while claiming not to know what it is..
__turbobrew__ 3y ago
The great part of “systems engineering” is that the fundamentals have not changed in the last like 30-40 years. If you have a basic university undergrad level understanding of operating systems you can work up from the fundamentals to understand what is going on and your knowledge will probably never go out of date in your career. It definitely isn’t for everyone and you need to be very persistent, but I enjoy not having to relearn tech stacks every 4 years and job security for Wizards tends to be quite good. The downside is you can spend 80+ hours tracking down a bug that results in a one line change (don’t ask me how I know).
- deleted 3y ago
  [deleted]
- gowld 3y ago
  I don't think this really holds. The bedrock layers may be pretty stable, but "systems" now have many more layers on top that needs more systems engineering attention.
- kentrf 3y ago
  > The downside is you can spend 80+ hours tracking down a bug that results in a one line change (don’t ask me how I know). We don't need to ask, we've all been there :)
faitswulff 3y ago
I keep plugging Oxide and Friends on this site, but the "Paths into Systems Programming" episode" is relevant and was a good discussion: https://oxide.computer/podcasts/oxide-and-friends/938256 https://oxide.computer/podcasts/oxide-and-friends/938256
throwawaaarrgh 3y ago
Systems Engineering is a bit discipline and industry specific; a nuclear systems engineer is a bit different from a candy factory systems engineer. But the basics are about the same. You don't need to understand the product management or biz management side, but it helps. A Senior needs to understand the basic phases of an engineering product's lifecycle from birth to death, and how to design for and complete each one. But a more standard systems engineer just needs to understand how systems work, how they are designed, implemented, and kept running. The rest comes over time. You don't have to be an expert in every part of the product. Like, you don't have to be an Operations Manager to engineer a product for production. The "boring" engineering work is all that's required. But it really helps to know a bit about everything else: how to find requirements, draft designs, liaise with stakeholders, make a POC, test it, implement components, tie them together into a system, test it, run it. Having a sunset plan is also great but not required. It definitely helps to have some experience in the industry you want to work in. If you wanna build a candy factory, having interned on the factory floor helps. Having built similar machines before helps.
vishnugupta 3y ago
Though I didn’t become a system engineer I was a part system admin earlier when I was about to start my professional career. The job was typical stuff, installing and maintaining DNS service, mail service, ensuring backups are done, and couple of times doing recovery etc. Along side I would troubleshoot issues like machine not being able to get onto LAN. It was really fantastic learning wise. I really got to see how things are implemented. I use those learnings even to this day. When a junior full stack developer runs into system level issues I often times see them go completely blank and I then gleefully roll up my sleeves and put on my susadmin hat, it’s quite fun.
- firesteelrain 3y ago
  Authors and yours definition of Systems Engineer are incorrect. I am actually getting my Masters in Systems Engineering and work for a large aerospace corporation - systems engineers are not admins and not even close. It’s simply not what they do.
  - fsociety 3y ago
    Systems Engineer means different things in different contexts. The fact that you are getting your Masters in Systems Engineering and work for a large aerospace corporation suggests you mean the tradition term, whereas the article and person you commented on means systems software engineer.
    - firesteelrain 3y ago
      That's my overall point. Both are wrong. Systems software engineer makes no sense.
  - vishnugupta 3y ago
    Without disagreeing with you, I was only chiming in with my experience in the same context as the author has written the article which is computer/software systems. Which is that being a part-time sysadmin is a great way to get into the computer systems field. I think within the software engineering field there's a general agreement about what "systems engineering" means and entails. FWIW here's an OCW course[1] [1] https://ocw.mit.edu/courses/6-033-computer-system-engineering-spring-2018/ https://ocw.mit.edu/courses/6-033-computer-system-engineerin...
pram 3y ago
I stopped using these titles because they’re just truly meaningless in the corporate world. It could literally be anything from a sysadmin to some programming job that actually requires super deep platform expertise. I’ve been a “Systems Analyst” and “Systems Engineer” at the same company and nothing about my tasks was different lol
julian_sark 3y ago
Just sign up at my company. Job done! My company used to title me: "System Architect". Yes, always singular, they're Germans and have no feel at all for English language. Then HR bought a pricy scheme from some consultant, and now I'm suddenly an "Advanced System Engineer". And also mightily pissed off for not being a senior title. It's like they roll dice around here. People went from being Data Analysts to being developers and vice versa. People were assigned titles that don't fit, purely so they could be outfited with certain benefits exclusively tied to those titles in a rigid scheme. Leave it to HR to screw over any industry naming conventions and to make my future job applications great fun, when I will explain why I'm not really that thing that's actually on my CV.
- bombolo 3y ago
  Surely you're aware that you can write whatever you want on your CV?
  - mejutoco 3y ago
    Not OP, but in Germany it is quite typical to receive a letter that officially states what were your duties and titles. Some employers expect you send them the previous job letters as a sort of background check. It is also easy to read between the lines in the letter to see if you did just the minimum or they were really happy with you. The fear of a bad letter can keep people accountable until they leave (best case) or be used as a threat by an unscrupulous employer (worst case). This affects international jobs a bit less, in my experience.
    - hef19898 3y ago
      That is slowly changing so. First because most employers are afraid of legal disputes and give roughly a 2 (based on the German grading system with 1 being the best 6 being the worst). The de-facto legal minimum, without hard reasons to deviate, is a 3. The second reason, nobody is asking for those anymore it seems. The last couple of jobs I took, nobody asked for any of my letters. Otherwise I agree, it takes some internal convincing to not care about them if your are used to them your whole life. Im short, those letters suck. And almost everybody knows it.
hardware2win 3y ago
> don't think I've ever taken anything through a whole "life cycle", whatever that even means for software. Im not talking about the author, but if you have ever heard phrase "seniors with 5 x (their first year) of experience" then not doing whole soft. dev. life cycle would be first sign of this. Jumping between projects every year and doing similar stuff sucks, yet people do that and are proud of the fact that they change jobs every 1 or 1.5 yr
kelnos 3y ago
> Nobody else saw something like this before, and so when you point it out and flip it back to sanity to restore the rest of the system, they look at you like you just pulled off some deep magic. and > you start from the assumption that when you see something, you wonder why it is the way it is. Then maybe you observe it and maybe do a little research to figure out how it came to be the thing you see in front of you. These both resonated with me deeply. I see a lot of people in tech who got into it for the money, or -- more charitably -- because they thought it might be cool, without much depth to that determination. I got into it in the late 80s, when I was 8 years old, because I was intensely curious about how things (first mechanical, and later digital) worked, and wanted to know how, and wanted to be able to make things that worked in the same way. I'm not saying the first category of people are deficient, but I definitely see people like that struggle with things that seem fairly obvious to me. Of course it's just because humans are pattern matching machines, and I've devoted a large chunk of my life to exploring these sorts of patterns, at many different levels of abstraction. I'm not trying to say I'm somehow so cool or a better human because of it, but I do think that "being a curious person", and having been acting on that curiosity all your life, really does give you a leg up when it comes to tech-focused fields (and presumably others, but I don't have experience with them, so I can't say). Most problems boil down to a much smaller set possible solutions, and being able to see the patterns that get you there can take more experience and time than many people are willing to devote.
- robertlagrant 3y ago
  > I got into it in the late 80s, when I was 8 years old I read this, and instinctively thought, "Wow - this person's old! This'll be a cool historical anecdote". Then I remembered I was also 8 in the late 80s. (I had an old BBC B Microcomputer I programmed on badly.)
  - zoom6628 3y ago
    LOL in late 80s I was in mid 20s and living and working in 3rd country. Started at NCR doing COBOL and NEAT/3 assembler for OS support and a manufacturing system. So I’m then terribly old. And I have lived a life driven by curiosity for everything. I’m no system engineer for sure but have self discovered a lot that sometimes proves useful and engineers and fellow product managers are like “ WTF how did you know that?!” The good old school system engineers knew how and why the software worked from the metal to the terminal and how to use the whole system productively. I’m in awe of such people and they are a privilege to work with. Oh and FTR started programming with punch cards when at high school. Used a local bank to compile Fortran.
TacticalCoder 3y ago
> Seeing four of them in a row in the same range with nothing going over 0x7f suggested SOME WORD IN ALL CAPS. It was. As opposed to some word in lower caps that would have had some bytes going over 0x7f? ('a' to 'z' go from 0x61 to 0x7a) That sentence got me confused!
a_c 3y ago
Zoom out the system bit to human level and you get entrepreneurship. Or just zoom a little bit out, you got product minded engineers. All of which have people actually using your code in the "thing that works" definition. Microservices, kubernetes, SPA, NoSQL, LLM-de-jour, architecture that I think better, space vs tabs, all matter way less than you think
ttr2021 3y ago
My simple answer: Don't. You're always hovering somewhere between a sysadmin and a software engineer, but never quite get the cred for developing software but still cop all the crud that sysadmins do.
hef19898 3y ago
>> I got some anonymous feedback a while back asking if I could do an article on how to become a systems engineer. I'm not entirely sure that I can, and part of that is the ambiguity in the request. To me, a "systems engineer" is a Real Engineer with actual certification and responsibilities to generally not be a clown. That's so far from the industry I work in that it's not even funny any more. Seriously though, if you look up "systems engineering" on Wikipedia, it talks about "how to design, integrate and manage complex systems over their life cycles". That's definitely not my personal slice of the world. I don't think I've ever taken anything through a whole "life cycle", whatever that even means for software. Just have to love the beginning!
NotYourLawyer 3y ago
It’s hard to imagine a more unhelpful answer. This is equal parts “look at how deeply I understand all this crusty old technology, yay me” and “everybody I work with is evil and stupid.” Zero parts actual advice about the question that was asked. So, par for the course on this blog.
- rramadass 3y ago
  You said it. The article just seems to be a series of ramblings. I am not quite sure why people are enamoured of it.
revskill 3y ago
How the tcp socket works is important. Because how can you make a process listen forever until you kill it. I think there's a serious issue in teaching style, where mentor often lost the bird eye view on things. System programming to me, is like "under the hood".
windi 3y ago
Interesting how systems engineering has two different definitions. We have the one being talked about it, and the 'traditional' definition. See https://en.wikipedia.org/wiki/Systems_engineering https://en.wikipedia.org/wiki/Systems_engineering. NASA even has a Systems Engineering Handbook: https://www.nasa.gov/seh/index.html https://www.nasa.gov/seh/index.html In short, 'systems engineering' as an engineering discipline has nothing to do with managing systems, servers, and networks.
- gowld 3y ago
  In computing, "systems" are inside and between the computers. Outside computing, "systems" include non-computer entities.
- ImPostingOnHN 3y ago
  the work I do as an ordinary [IT] systems engineer seems to fit that traditional definition just fine, with the work involving systems thinking about, and dealing with, how different parts of company systems interact (different groups' applications, network, service providers, customers, employees, each logical layer of each stack the software runs on, to varying degrees, down to bare metal, etc)
- twobitshifter 3y ago
  Right, in my experience you’ll run into ‘systems engineers’ in all fields, some are working on interfaces of the systems in that field, and others are thinking of the totality of the system and it’s maintainability, efficiency, and design. You always have to ask a follow up when someone says they’re a systems engineer.
- kejaed 3y ago
  I'll generally start with defining the "INCOSE" kind or the "MCSE" kind and most of the time people from one group or the other will know of the one they are close to and little to nothing about the other one.
svilen_dobrev 3y ago
systems engineering... is about curiosity. Asking how then why. AND Then knowledge+ability to build/tweak/fix/do-something-about the Thing. Not just observe. Whether the domain is software or mechanics or rocket science or group-psychology or (usualy) many of them together.. it's still same. Just takes time and accumulation of lots of rather-unconnected-unrelated-irrelevant little bits. When i hire, i look for DIY skills. Tinkering with stuff. Repairing broken pens, toys, door-frames, or UIs. That has proven to be the only relevant signal. Lots of people can apply formulas. Few question them. Seems Rare skill nowadays. Maybe because i'm DIY reverse-engineer-everything guy myself. You know, people hire their similars :) offtopic: some very-old DIY things of mine: https://svilendobrev.com/napravisam/ https://svilendobrev.com/napravisam/ pff, maybe i need to make a separate chapter on handles and hooks and such, i have (re)build dozens.. have fun
- FeepingCreature 3y ago
  Doesn't have to be curiosity, it also works with stubbornness. If you have the opinion that it "should ought to" work a certain way, and the system disagrees, then you will naturally have to dig into it, because you'll create your own breakage. :) I honestly don't give a damn how my system works, so long as it works. It's just that the way I use it tends to make that a very temporary state of affairs. It doesn't particularly matter how you come by the library of a thousand mostly useless low-level factoids about the system. So long as it accumulates, and you keep having to make use of it to work around the problems you yourself caused a few days prior, you won't be able to avoid becoming a systems person.
neilv 3y ago
This sounds like the familiar grizzled veteran who's grumpy about the reality of some field, but they're grumpy because they have hard-earned insight into the field. (And maybe not feeling as upbeat that day as they usually manage.) One idea a young grasshopper might keep in mind: maybe most all fields are such, that you'd be grumpy about a lot of things in them, once you have decades of understanding. So, if someone is telling you that the reality of field A has a lot of downside, but you're not hearing that about field B... maybe the situation is that you aren't hearing from someone knowledgeable/representative/honest enough about B, but you have a rare opportunity to learn from a straight-talking expert in A.
jwsteigerwalt 3y ago
I feel that the bridge between software engineer and systems architect comes through scale, maturation, and integration. Today a BS in computer science covers a wide range of topics outside of syntax and databases. On the day you graduate, you have a set of hard skills writing code that are likely immediately applicable to a job and you also have more abstract theoretical instruction related to real world scale, deployments. There are also types of problems that are very difficult to teach in a classroom setting. Those things where the core code works fine, but in a scaled production implementation, it’s not meeting the mark. Success there usually involves iterations over weeks or months to get a working solution. Being a really great systems architect involves mixing the hard technical skills with the softer qualitative skills learned along the way.
hyperliner 3y ago
[dead]
Rochus 3y ago
Systems engineering is like Esperanto; it would be very useful if more than a few people would "speak" it. It already starts with the fact that the term "systems engineering", which has been clearly defined in various standards since the sixties (MIL-STD 499, IEEE 1220, ISO 26702, ISO 24748-4, etc.) and backed by a comprehensive body of knowledge, is now used almost arbitrarily by anyone. Also in the article, the term is used with a different meaning than the original one. Of course it can be argued that language evolves like people, but after all, also in physics or mathematics we don't just replace or redefine terms ad libitum (you won't pass exams if you use your own terminology), and on systems engineering there are university courses and a well established scholarship (at least there were when I was a student many years ago).
- strobe999 3y ago
  Huh, in all these years I never suspected to find resources about SE in standards bodies. Would be nice to have more formal training about SE specific procedures instead of products and general project management. I found the INCOSE Systems Engineering Handbook which seems to be a prolific publication, is it worth studying as an IT SE? Any other recommendations?
  - Rochus 3y ago
    I wrote a couple of articles about the topic on Quora some years ago, see e.g. https://www.quora.com/Which-one-is-a-good-book-to-learn-Systems-Engineering/answer/Rochus-Keller https://www.quora.com/Which-one-is-a-good-book-to-learn-Syst...
- bionsystem 3y ago
  I remember introducing myself as such to a meeting and some manager didn't understand why I was there. Turns out he came from a previous experience where "systems engineer" was about designing nuclear combustion engines, which is quite a bit different than writing shell scripts ! Now I prefer SRE or simply sysadmin.
sylware 3y ago
Well, "engineer" means real maths/physics/chemistry applied to information systems. If not, it is "technician" (well, in my country). Most "engineers" actually work at a "technician" level. Where a systems technician may get in trouble: accurate and proven floating point calculations and where it comes from (minimal polynomial, range reduction, etc), real understanding of the maths of cryptography. Proofs of common performant algorithms (ring buffers synchronized with atomic variables, unification,etc). It may need a deep understanding of signal processing, depending on the field, etc etc. But since everything was already put in boxes (libs), a systems technician can re-use mostly all of them: the set of skills required is no higher than high school level. Basically, an engineer has a deeper undertanding of information systems, but work most of the time as a "technician", and has some background knowledge to help him/her ramp up on tricky stuff.
naillo 3y ago
Articles like this make me wonder if I'm cut out to be an engineer. I'm bored to tears by systems engineering and security. I just want to build interesting stuff.
- jjice 3y ago
  Of course you are, you just don't like systems engineering and that's fine. Someone's gotta build on top of the systems too. To you it's the boring stuff. To a systems engineer, you might do the boring stuff. We all complement each other well.
- eropple 3y ago
  "Be an engineer" and "write code" aren't the same thing. That this industry has decided to call everyone an "engineer" because they can write some Java or even some HTML has loaded these jobs with a set of implications that they don't ever really cash in on. Until something goes wrong and they now have a stick to hit you with. It is okay to be a "developer", it's not a naughty word, and it's okay to not be an "engineer" in the sense that Rachel means it. That said, there is considerable overlap between "doing engineering" and being an effective, skilled software craftsperson. Enumerate edge cases as exhaustively as possible, capture solutions for them. Ask "what if this breaks?". Have an answer. This isn't "being an engineer", engineers get in Actual Trouble when a bridge falls down, but it is being a skilled crafter of software, and they're not dissimilar questions. The level of rigor applied might differ, but if this isn't where your brain is going by default, it is a worthwhile and even necessary skill to cultivate to build something worth using. Of course, if you're building something people rely on, you'd better be working with and listening to people who are putting (or attempting to put; this is a young field, after wall) some kind of engineering rigor into practice.
  - 908B64B197 3y ago
    > Ask "what if this breaks?". Have an answer. This isn't "being an engineer", engineers get in Actual Trouble when a bridge falls down, but it is being a skilled crafter of software, and they're not dissimilar questions. Wait until you learn how much software is in a modern car!

367 points by picture 3y ago

jspaetzel 3y ago

Run Linux as your desktop.

harry8 3y ago
Too easy nowadays. Pick an esoteric BSD?
- nathants 3y ago
  it’s not about difficulty. it’s about building useful intuition to guide your curiosity the next time an eyebrow goes up. bsd is used in prod only slightly more than macos.
  - harry8 3y ago
    Nah it is. When something "just works" you don't learn much about how. When your graphics card driver shits itself, you learn something getting it working again. Console text editor at the minimum! It's not "misery is good for you" but having to solve random problems is an opportunity for learning the guts you otherwise don't get. Linux on a laptop used to fill this niche nicely. Does it still?
    - nathants 3y ago
      does still. run alpine for increased difficulty.
      - harry8 3y ago
        running a mainstream distro & desktop env on a thinkpad t480s is significantly easier, more efficient and less fragile than any windows laptop I've ever used, seen or heard of. Less signficantly so but still easier than any apple laptop I've used. Genuinely so. YMMV.
        nathants 3y ago
        the primary difficulty isn’t supposed to be the os, but the software you’re building. though os difficulties will always come up, depending on workload. being on linux means you gain intuition debugging locally. whatever you might be debugging. to sum up my point: linux. never not be running it. to your point, linux is a better experience now than before. this is for great good, not a cause to search for a more brittle os and distro. at eod, os isn’t that cool. software is cool.
        harry8 3y ago
        Using an operating system you have to tinker with to get working right, and even fix when it breaks means you learn a lot about the low level of what an operating system is, and how this one does it. That information translates to being useful writing and fixing other software. The point is this, if you're a student, consider using something unusual that is more brittle. Maybe a sweet spot is running mainstream linux on your mainstream laptop (Lenovo T-series is one of the right answers there, dell is the wrong answer) and have a project using a pine rock64 SBC running linux to drive your tv screen or similar? Something that you want to use, that breaks at random in random places that you then have to investigate to work out how to fix. You really learn something keeping brittle systems functional. Linux laptops with nvidia gfx cards definitely used to qualify in ways a t480s really doesn't nowadays. When fixing things that break, strace, gdb, perf, ftrace, BPF, write or fix a kernel device driver, understand what the OS and compiler don't do for you, what they often will do for you that maybe you don't want them to.
        nathants 3y ago
        you may be right. i’m not sure i would recommend someone to pursue a brittle setup, but it might be more educational.
        nathants 3y ago
        my current brittle setup is postmarketos on oneplus 6t. definitely kind of works. the rest of my setups are non-brittle.

Balgair 3y ago

Aside: Can software based systems engineers please call themselves software systems engineer. I've been a 'traditional' systems engineer for many years and when searching for jobs (or with recruiters), it's always a mess trying to filter out the software only systems engineering gigs. I use a lot of software in my work too, it's just that I also have to do 'hardware' by the end of the work. It's not a big ask (I think), but man, would it make a lot of sense.

0xbadcafebee 3y ago
We're still trying to convince hiring managers to stop listing positions for "DevOps Engineer", which is akin to "Culture Engineer".

ftxbro 3y ago

> malloc("HTTP")

dilyevsky 3y ago
Dont try this at home. Reminds me of james mickens pointer equals to 7 thing
- vvanders 3y ago
  https://www.usenix.org/system/files/1311_05-08_mickens.pdf https://www.usenix.org/system/files/1311_05-08_mickens.pdf "... All of these people will play a role in my ultimate success as a dystopian warlord philosopher. However, the most important person in my gang will be a systems programmer." It's right there in the intro, truly a classic.

PaulCarrack 3y ago

> malloc(1213486160) is really malloc(0x48545450) is really malloc("HTTP") No it's not / that's not how C works. malloc("HTTP") is malloc of the address of a string literal in memory. If "HTTP" is at address 0x11223344 then it's malloc(0x11223344). So it's the address of where ever the "HTTP" string is in the binary not the characters themselves. It's kind of a weird way to say that you can convert decimal to hexadecimal in your head. I also wouldn't use malloc as the example to demonstrate that talent, it makes no sense that way. Perhaps becoming a systems engineer should involve learning C first... Don't always trust what you read.

anonymoushn 3y ago
malloc takes a size_t indicating the desired size of the allocation. If malloc is accidentally being called with a pointer instead that would be an error, but this story is about an instance where malloc was accidentally being called with the first bits of a string instead, apparently on a big-endian system where size_t is 32 bits. Hope that helps.
- PaulCarrack 3y ago
  > If malloc is accidentally being called with a pointer instead that would be an error Not true, just tried it in MSVC and it compiled fine but with a warning on conflicting types. "HTTP" is of type const char* and sizeof(size_t) == sizeof(const char *). Doesn't matter if its 32 or 64, I'm not aware of a platform where that isn't the case. > but this story is about an instance where malloc was accidentally being called with the first bits of a string instead If that's the case, then it should have been something along the lines of *"HTTP" instead.
  - anonymoushn 3y ago
    > Not true, just tried it in MSVC and it compiled fine but with a warning on conflicting types. If I meant that it would cause the compiler to report an error then I would have written that. > If that's the case, then it should have been something along the lines of *"HTTP" instead. We can say that they really should have written memcpy to get the string literal into the size_t or we can accept that sometimes people use language imprecisely and expect readers to make an effort to understand rather than make an effort not to understand.

rukuu001 3y ago

I don't know if the article is a helpful answer to the question, but it's a very true answer I think. My path to 'doing' systems engineering for a time (it really wasn't my thing) was by maintaining software an actual sys. eng. had written, fixing problems with it, extending it and eventually becoming responsible for it. Best lol - API of a library we were using changed so that instead of taking a pointer to a struct to allocate memory for, it just wanted an amount of memory to allocate. We didn't notice, it passed all our tests (sheer luck) and dumped core on prod all day long. Funny.

S_A_P 3y ago
— We didn't notice, it passed all our tests (sheer luck) and dumped core on prod all day long. If this happened to me I’d not call that luck, I would want to know why my tests missed this.
- erdo 3y ago
  That's sarcasm, it probably also wasn't actually Funny. Nothing about what they said suggests they didn't subsequently look into why their tests missed the issue

prpl 3y ago

I’ve worked with systems engineers, as in the modeling/SysML type. It is important to note that software is usually a part of a much bigger system, usually physical, and that systems engineering is kind of an engineering approach to project management to help in ensuring things get built to spec (even if the spec changes). The nice thing about that is you don’t get VPs/execs walking in the door saying what you’re going to do this quarter instead.

anotherhue 3y ago

I like the article at lot, I've got a similar background. My answer to the question would be: start explaining what you're asking the OS to do when you run your program. There's an entire category of developers who view the IDE and the language as a perfect hermetically sealed environment, until they meet a leaky abstraction. Some favourites: 1. Depleting the outbound TCP ports because ports aren't infinite and the code didn't know that. 2. Any nodejs BS that involves a precompiled C lib, where homebrew or some nonsense 'just works' on the dev machine but prod runs alpine/musl. NPM sneaking arch specific blobs in is really to blame here but what do we expect from the failed state of package managers. 3. Assuming latency is not a thing. Network calls everywhere. No management of timeouts / killed connections. 4. Memory, in all shapes and forms. Most typically thinking a GC prevents memory leaks, but more generally, having no clue whatsoever what the memory usage profile of the application is. A systems engineer (IMO) has much better answers to these questions than a regular developer, and it is absolutely mostly arcane nonsense, (ulimit anyone?), until it isn't. Personally I love OS tech so maybe I'm just cut out for it. I used to be able to tell what the machine was doing from the HDD whirrs and still have a 'feel' for system latency, e.g. when something completes too quickly.

zorked 3y ago
> I used to be able to tell what the machine was doing from the HDD whirrs I guess that dimension of debugging is now lost with SSDs. I didn't run an anti-virus and noticed by machine had caught a virus (in this case, DOS/Natas) because of the extra moments of whirrs during boot and when running applications.
- beedeebeedee 3y ago
  I've been doing some audio work simulating audio equipment, and I bet you could pipe out some sound for disk writes and use that as a debug tool.
  - anotherhue 3y ago
    https://www.retrokits.de/index.php/hdd-clicker-hdd-sound-simulator-v0-1/ https://www.retrokits.de/index.php/hdd-clicker-hdd-sound-sim...

kodah 3y ago

I liked the article. It kind of taps into the complexity of what goes into Systems Engineering. I always describe the job as a software engineer who also understands OS internals, APIs, and tooling and about all the external things that they depend on.

analog31 3y ago

>>> I think it goes something like this: you start from the assumption that when you see something, you wonder why it is the way it is. Then maybe you observe it and maybe do a little research to figure out how it came to be the thing you see in front of you. This could go for just about anything: a telephone, a scale, a crusty old road surface, a forgotten grove of fruit trees, you name it. By research, I mean maybe you go poking around: try to open that scale with a screwdriver, get out of the car and walk down the old road, or turn over some of the dirt in the field to see if you can find any identifying marks. ... a scientist. My first job title at my current workplace was "systems engineer." My degree is in physics. I evolved into the role of being the person on the project who understands and can explain how the whole thing works. It doesn't mean just dictating a theory of operation, but also testing that theory. My job is in an area where products involve both hardware and software. I certainly don't claim to know every design detail. Part of the trick is figuring out what things I actually do need to know, and how to nail those things down.

josephg 3y ago
Yep. I found a bug the other week which was slowing some important code down by 20x. I found the problem because it was tweaking my intuition. It just didn’t make sense that this code was so slow. After a few nights lying in bed thinking about it, I did some back of the napkin math which agreed with my intuition. Eventually I took some time to track it down. It turned out a rogue debug assertion hidden in a big function was making it into production. The assertion did a deep scan of the system to check invariants, and that turned an O(n) into an O(n^2) in some cases. This sort of thing comes up a lot while programming. Something weird happens and you start to wonder - why does it behave like this? Answering these questions has led me all sorts of places - to reading specs, debugging into databases, compiling my own Firefox, all sorts of things. The chance this knowledge pays off sooner or later is wildly high. Computers are deterministic and debugable. You usually have the source code and everything it does was designed and written by people like you. A curious, scientific attitude is a beautiful thing to nurture in software.
- alex-moon 3y ago
  I actually had a case like this earlyish in my career. I had a job where my main responsibility was maintaining and developing large ETL scripts for an online book store (so hundreds of millions of rows). We had one script which did the meat of the work which was running many hours each night. My job was to speed it up. Watching it run for a bit, it occurred to me to wonder how long it took to send each line of progress info (couple of lines each for every single row) to the log. I remember my boss at the time didn't believe me. "I think it's logging," I said, and turned logging off just for funsies. Boom, 100x faster.

scsteps 3y ago

Systems engineering is actually an overloaded term and that Wikipedia article is correct. The author’s definition is also correct of course. Just talking about different things.

grepfru_it 3y ago
What I thought hilarious, when the author was describing a series of tools that were maintained for a long period of time.. that was the complex system. No attempts were made to reduce the complexity, because the cost was not worth the increased value. Maybe that specific part, realizing the business driver of systems, is what makes a Systems Engineer. And even though your specific application won't EOL, there are many times where features are deprecated and that complex system must be altered. Maybe I'm just grizzled in my old age, but it seems like this author was writing about Systems Engineering while claiming not to know what it is..

__turbobrew__ 3y ago

The great part of “systems engineering” is that the fundamentals have not changed in the last like 30-40 years. If you have a basic university undergrad level understanding of operating systems you can work up from the fundamentals to understand what is going on and your knowledge will probably never go out of date in your career. It definitely isn’t for everyone and you need to be very persistent, but I enjoy not having to relearn tech stacks every 4 years and job security for Wizards tends to be quite good. The downside is you can spend 80+ hours tracking down a bug that results in a one line change (don’t ask me how I know).

deleted 3y ago
[deleted]
gowld 3y ago
I don't think this really holds. The bedrock layers may be pretty stable, but "systems" now have many more layers on top that needs more systems engineering attention.
kentrf 3y ago
> The downside is you can spend 80+ hours tracking down a bug that results in a one line change (don’t ask me how I know). We don't need to ask, we've all been there :)

faitswulff 3y ago

I keep plugging Oxide and Friends on this site, but the "Paths into Systems Programming" episode" is relevant and was a good discussion: https://oxide.computer/podcasts/oxide-and-friends/938256 https://oxide.computer/podcasts/oxide-and-friends/938256

throwawaaarrgh 3y ago

Systems Engineering is a bit discipline and industry specific; a nuclear systems engineer is a bit different from a candy factory systems engineer. But the basics are about the same. You don't need to understand the product management or biz management side, but it helps. A Senior needs to understand the basic phases of an engineering product's lifecycle from birth to death, and how to design for and complete each one. But a more standard systems engineer just needs to understand how systems work, how they are designed, implemented, and kept running. The rest comes over time. You don't have to be an expert in every part of the product. Like, you don't have to be an Operations Manager to engineer a product for production. The "boring" engineering work is all that's required. But it really helps to know a bit about everything else: how to find requirements, draft designs, liaise with stakeholders, make a POC, test it, implement components, tie them together into a system, test it, run it. Having a sunset plan is also great but not required. It definitely helps to have some experience in the industry you want to work in. If you wanna build a candy factory, having interned on the factory floor helps. Having built similar machines before helps.

vishnugupta 3y ago

Though I didn’t become a system engineer I was a part system admin earlier when I was about to start my professional career. The job was typical stuff, installing and maintaining DNS service, mail service, ensuring backups are done, and couple of times doing recovery etc. Along side I would troubleshoot issues like machine not being able to get onto LAN. It was really fantastic learning wise. I really got to see how things are implemented. I use those learnings even to this day. When a junior full stack developer runs into system level issues I often times see them go completely blank and I then gleefully roll up my sleeves and put on my susadmin hat, it’s quite fun.

firesteelrain 3y ago
Authors and yours definition of Systems Engineer are incorrect. I am actually getting my Masters in Systems Engineering and work for a large aerospace corporation - systems engineers are not admins and not even close. It’s simply not what they do.
- fsociety 3y ago
  Systems Engineer means different things in different contexts. The fact that you are getting your Masters in Systems Engineering and work for a large aerospace corporation suggests you mean the tradition term, whereas the article and person you commented on means systems software engineer.
  - firesteelrain 3y ago
    That's my overall point. Both are wrong. Systems software engineer makes no sense.
- vishnugupta 3y ago
  Without disagreeing with you, I was only chiming in with my experience in the same context as the author has written the article which is computer/software systems. Which is that being a part-time sysadmin is a great way to get into the computer systems field. I think within the software engineering field there's a general agreement about what "systems engineering" means and entails. FWIW here's an OCW course[1] [1] https://ocw.mit.edu/courses/6-033-computer-system-engineering-spring-2018/ https://ocw.mit.edu/courses/6-033-computer-system-engineerin...

pram 3y ago

I stopped using these titles because they’re just truly meaningless in the corporate world. It could literally be anything from a sysadmin to some programming job that actually requires super deep platform expertise. I’ve been a “Systems Analyst” and “Systems Engineer” at the same company and nothing about my tasks was different lol

julian_sark 3y ago

Just sign up at my company. Job done! My company used to title me: "System Architect". Yes, always singular, they're Germans and have no feel at all for English language. Then HR bought a pricy scheme from some consultant, and now I'm suddenly an "Advanced System Engineer". And also mightily pissed off for not being a senior title. It's like they roll dice around here. People went from being Data Analysts to being developers and vice versa. People were assigned titles that don't fit, purely so they could be outfited with certain benefits exclusively tied to those titles in a rigid scheme. Leave it to HR to screw over any industry naming conventions and to make my future job applications great fun, when I will explain why I'm not really that thing that's actually on my CV.

bombolo 3y ago
Surely you're aware that you can write whatever you want on your CV?
- mejutoco 3y ago
  Not OP, but in Germany it is quite typical to receive a letter that officially states what were your duties and titles. Some employers expect you send them the previous job letters as a sort of background check. It is also easy to read between the lines in the letter to see if you did just the minimum or they were really happy with you. The fear of a bad letter can keep people accountable until they leave (best case) or be used as a threat by an unscrupulous employer (worst case). This affects international jobs a bit less, in my experience.
  - hef19898 3y ago
    That is slowly changing so. First because most employers are afraid of legal disputes and give roughly a 2 (based on the German grading system with 1 being the best 6 being the worst). The de-facto legal minimum, without hard reasons to deviate, is a 3. The second reason, nobody is asking for those anymore it seems. The last couple of jobs I took, nobody asked for any of my letters. Otherwise I agree, it takes some internal convincing to not care about them if your are used to them your whole life. Im short, those letters suck. And almost everybody knows it.

hardware2win 3y ago

> don't think I've ever taken anything through a whole "life cycle", whatever that even means for software. Im not talking about the author, but if you have ever heard phrase "seniors with 5 x (their first year) of experience" then not doing whole soft. dev. life cycle would be first sign of this. Jumping between projects every year and doing similar stuff sucks, yet people do that and are proud of the fact that they change jobs every 1 or 1.5 yr

kelnos 3y ago

> Nobody else saw something like this before, and so when you point it out and flip it back to sanity to restore the rest of the system, they look at you like you just pulled off some deep magic. and > you start from the assumption that when you see something, you wonder why it is the way it is. Then maybe you observe it and maybe do a little research to figure out how it came to be the thing you see in front of you. These both resonated with me deeply. I see a lot of people in tech who got into it for the money, or -- more charitably -- because they thought it might be cool, without much depth to that determination. I got into it in the late 80s, when I was 8 years old, because I was intensely curious about how things (first mechanical, and later digital) worked, and wanted to know how, and wanted to be able to make things that worked in the same way. I'm not saying the first category of people are deficient, but I definitely see people like that struggle with things that seem fairly obvious to me. Of course it's just because humans are pattern matching machines, and I've devoted a large chunk of my life to exploring these sorts of patterns, at many different levels of abstraction. I'm not trying to say I'm somehow so cool or a better human because of it, but I do think that "being a curious person", and having been acting on that curiosity all your life, really does give you a leg up when it comes to tech-focused fields (and presumably others, but I don't have experience with them, so I can't say). Most problems boil down to a much smaller set possible solutions, and being able to see the patterns that get you there can take more experience and time than many people are willing to devote.

robertlagrant 3y ago
> I got into it in the late 80s, when I was 8 years old I read this, and instinctively thought, "Wow - this person's old! This'll be a cool historical anecdote". Then I remembered I was also 8 in the late 80s. (I had an old BBC B Microcomputer I programmed on badly.)
- zoom6628 3y ago
  LOL in late 80s I was in mid 20s and living and working in 3rd country. Started at NCR doing COBOL and NEAT/3 assembler for OS support and a manufacturing system. So I’m then terribly old. And I have lived a life driven by curiosity for everything. I’m no system engineer for sure but have self discovered a lot that sometimes proves useful and engineers and fellow product managers are like “ WTF how did you know that?!” The good old school system engineers knew how and why the software worked from the metal to the terminal and how to use the whole system productively. I’m in awe of such people and they are a privilege to work with. Oh and FTR started programming with punch cards when at high school. Used a local bank to compile Fortran.

TacticalCoder 3y ago

> Seeing four of them in a row in the same range with nothing going over 0x7f suggested SOME WORD IN ALL CAPS. It was. As opposed to some word in lower caps that would have had some bytes going over 0x7f? ('a' to 'z' go from 0x61 to 0x7a) That sentence got me confused!

a_c 3y ago

Zoom out the system bit to human level and you get entrepreneurship. Or just zoom a little bit out, you got product minded engineers. All of which have people actually using your code in the "thing that works" definition. Microservices, kubernetes, SPA, NoSQL, LLM-de-jour, architecture that I think better, space vs tabs, all matter way less than you think

ttr2021 3y ago

My simple answer: Don't. You're always hovering somewhere between a sysadmin and a software engineer, but never quite get the cred for developing software but still cop all the crud that sysadmins do.

hef19898 3y ago

>> I got some anonymous feedback a while back asking if I could do an article on how to become a systems engineer. I'm not entirely sure that I can, and part of that is the ambiguity in the request. To me, a "systems engineer" is a Real Engineer with actual certification and responsibilities to generally not be a clown. That's so far from the industry I work in that it's not even funny any more. Seriously though, if you look up "systems engineering" on Wikipedia, it talks about "how to design, integrate and manage complex systems over their life cycles". That's definitely not my personal slice of the world. I don't think I've ever taken anything through a whole "life cycle", whatever that even means for software. Just have to love the beginning!

NotYourLawyer 3y ago

It’s hard to imagine a more unhelpful answer. This is equal parts “look at how deeply I understand all this crusty old technology, yay me” and “everybody I work with is evil and stupid.” Zero parts actual advice about the question that was asked. So, par for the course on this blog.

rramadass 3y ago
You said it. The article just seems to be a series of ramblings. I am not quite sure why people are enamoured of it.

revskill 3y ago

How the tcp socket works is important. Because how can you make a process listen forever until you kill it. I think there's a serious issue in teaching style, where mentor often lost the bird eye view on things. System programming to me, is like "under the hood".

windi 3y ago

Interesting how systems engineering has two different definitions. We have the one being talked about it, and the 'traditional' definition. See https://en.wikipedia.org/wiki/Systems_engineering https://en.wikipedia.org/wiki/Systems_engineering. NASA even has a Systems Engineering Handbook: https://www.nasa.gov/seh/index.html https://www.nasa.gov/seh/index.html In short, 'systems engineering' as an engineering discipline has nothing to do with managing systems, servers, and networks.

gowld 3y ago
In computing, "systems" are inside and between the computers. Outside computing, "systems" include non-computer entities.
ImPostingOnHN 3y ago
the work I do as an ordinary [IT] systems engineer seems to fit that traditional definition just fine, with the work involving systems thinking about, and dealing with, how different parts of company systems interact (different groups' applications, network, service providers, customers, employees, each logical layer of each stack the software runs on, to varying degrees, down to bare metal, etc)
twobitshifter 3y ago
Right, in my experience you’ll run into ‘systems engineers’ in all fields, some are working on interfaces of the systems in that field, and others are thinking of the totality of the system and it’s maintainability, efficiency, and design. You always have to ask a follow up when someone says they’re a systems engineer.
kejaed 3y ago
I'll generally start with defining the "INCOSE" kind or the "MCSE" kind and most of the time people from one group or the other will know of the one they are close to and little to nothing about the other one.

svilen_dobrev 3y ago

systems engineering... is about curiosity. Asking how then why. AND Then knowledge+ability to build/tweak/fix/do-something-about the Thing. Not just observe. Whether the domain is software or mechanics or rocket science or group-psychology or (usualy) many of them together.. it's still same. Just takes time and accumulation of lots of rather-unconnected-unrelated-irrelevant little bits. When i hire, i look for DIY skills. Tinkering with stuff. Repairing broken pens, toys, door-frames, or UIs. That has proven to be the only relevant signal. Lots of people can apply formulas. Few question them. Seems Rare skill nowadays. Maybe because i'm DIY reverse-engineer-everything guy myself. You know, people hire their similars :) offtopic: some very-old DIY things of mine: https://svilendobrev.com/napravisam/ https://svilendobrev.com/napravisam/ pff, maybe i need to make a separate chapter on handles and hooks and such, i have (re)build dozens.. have fun

FeepingCreature 3y ago
Doesn't have to be curiosity, it also works with stubbornness. If you have the opinion that it "should ought to" work a certain way, and the system disagrees, then you will naturally have to dig into it, because you'll create your own breakage. :) I honestly don't give a damn how my system works, so long as it works. It's just that the way I use it tends to make that a very temporary state of affairs. It doesn't particularly matter how you come by the library of a thousand mostly useless low-level factoids about the system. So long as it accumulates, and you keep having to make use of it to work around the problems you yourself caused a few days prior, you won't be able to avoid becoming a systems person.

neilv 3y ago

This sounds like the familiar grizzled veteran who's grumpy about the reality of some field, but they're grumpy because they have hard-earned insight into the field. (And maybe not feeling as upbeat that day as they usually manage.) One idea a young grasshopper might keep in mind: maybe most all fields are such, that you'd be grumpy about a lot of things in them, once you have decades of understanding. So, if someone is telling you that the reality of field A has a lot of downside, but you're not hearing that about field B... maybe the situation is that you aren't hearing from someone knowledgeable/representative/honest enough about B, but you have a rare opportunity to learn from a straight-talking expert in A.

jwsteigerwalt 3y ago

I feel that the bridge between software engineer and systems architect comes through scale, maturation, and integration. Today a BS in computer science covers a wide range of topics outside of syntax and databases. On the day you graduate, you have a set of hard skills writing code that are likely immediately applicable to a job and you also have more abstract theoretical instruction related to real world scale, deployments. There are also types of problems that are very difficult to teach in a classroom setting. Those things where the core code works fine, but in a scaled production implementation, it’s not meeting the mark. Success there usually involves iterations over weeks or months to get a working solution. Being a really great systems architect involves mixing the hard technical skills with the softer qualitative skills learned along the way.

hyperliner 3y ago

[dead]

Rochus 3y ago

Systems engineering is like Esperanto; it would be very useful if more than a few people would "speak" it. It already starts with the fact that the term "systems engineering", which has been clearly defined in various standards since the sixties (MIL-STD 499, IEEE 1220, ISO 26702, ISO 24748-4, etc.) and backed by a comprehensive body of knowledge, is now used almost arbitrarily by anyone. Also in the article, the term is used with a different meaning than the original one. Of course it can be argued that language evolves like people, but after all, also in physics or mathematics we don't just replace or redefine terms ad libitum (you won't pass exams if you use your own terminology), and on systems engineering there are university courses and a well established scholarship (at least there were when I was a student many years ago).

strobe999 3y ago
Huh, in all these years I never suspected to find resources about SE in standards bodies. Would be nice to have more formal training about SE specific procedures instead of products and general project management. I found the INCOSE Systems Engineering Handbook which seems to be a prolific publication, is it worth studying as an IT SE? Any other recommendations?
- Rochus 3y ago
  I wrote a couple of articles about the topic on Quora some years ago, see e.g. https://www.quora.com/Which-one-is-a-good-book-to-learn-Systems-Engineering/answer/Rochus-Keller https://www.quora.com/Which-one-is-a-good-book-to-learn-Syst...
bionsystem 3y ago
I remember introducing myself as such to a meeting and some manager didn't understand why I was there. Turns out he came from a previous experience where "systems engineer" was about designing nuclear combustion engines, which is quite a bit different than writing shell scripts ! Now I prefer SRE or simply sysadmin.

sylware 3y ago

Well, "engineer" means real maths/physics/chemistry applied to information systems. If not, it is "technician" (well, in my country). Most "engineers" actually work at a "technician" level. Where a systems technician may get in trouble: accurate and proven floating point calculations and where it comes from (minimal polynomial, range reduction, etc), real understanding of the maths of cryptography. Proofs of common performant algorithms (ring buffers synchronized with atomic variables, unification,etc). It may need a deep understanding of signal processing, depending on the field, etc etc. But since everything was already put in boxes (libs), a systems technician can re-use mostly all of them: the set of skills required is no higher than high school level. Basically, an engineer has a deeper undertanding of information systems, but work most of the time as a "technician", and has some background knowledge to help him/her ramp up on tricky stuff.

naillo 3y ago

Articles like this make me wonder if I'm cut out to be an engineer. I'm bored to tears by systems engineering and security. I just want to build interesting stuff.

jjice 3y ago
Of course you are, you just don't like systems engineering and that's fine. Someone's gotta build on top of the systems too. To you it's the boring stuff. To a systems engineer, you might do the boring stuff. We all complement each other well.
eropple 3y ago
"Be an engineer" and "write code" aren't the same thing. That this industry has decided to call everyone an "engineer" because they can write some Java or even some HTML has loaded these jobs with a set of implications that they don't ever really cash in on. Until something goes wrong and they now have a stick to hit you with. It is okay to be a "developer", it's not a naughty word, and it's okay to not be an "engineer" in the sense that Rachel means it. That said, there is considerable overlap between "doing engineering" and being an effective, skilled software craftsperson. Enumerate edge cases as exhaustively as possible, capture solutions for them. Ask "what if this breaks?". Have an answer. This isn't "being an engineer", engineers get in Actual Trouble when a bridge falls down, but it is being a skilled crafter of software, and they're not dissimilar questions. The level of rigor applied might differ, but if this isn't where your brain is going by default, it is a worthwhile and even necessary skill to cultivate to build something worth using. Of course, if you're building something people rely on, you'd better be working with and listening to people who are putting (or attempting to put; this is a young field, after wall) some kind of engineering rigor into practice.
- 908B64B197 3y ago
  > Ask "what if this breaks?". Have an answer. This isn't "being an engineer", engineers get in Actual Trouble when a bridge falls down, but it is being a skilled crafter of software, and they're not dissimilar questions. Wait until you learn how much software is in a modern car!