11 ms·

While writing a text editor, a compiler, an operating system, or a raytracer might make you a better programmer, it won't make you a better software engineer. I

by p-e-w 3y ago

While writing a text editor, a compiler, an operating system, or a raytracer might make you a better programmer, it won't make you a better software engineer. In fact, it might make you worse at software engineering, because it embodies the disastrous "Not Invented Here" doctrine. Hackers like to obsess about Big-O, data structures, HoTT, and other high-theory stuff, yet the following skills, essential for software engineering, are almost never discussed and even more rarely practiced: - Deciding what to write yourself and what to take from a library - Identifying high-quality libraries and frameworks that meet your project needs - Deciding where optimization is worth the effort and where it is not - Writing code that will still be readable to you (and others) a few years from now - Thinking about the project as a large-scale, complex system with software and non-software dependencies In that spirit, I offer the following alternative challenge: Create a web search engine. Don't bother with string matching algorithms etc., others have already done that for you. "Just" make a search engine (and crawler) that can actually work, even if it only supports a subset of the web and a single concurrent user at the beginning.

whateveracct 3y ago
[flagged]
- deleted 3y ago
  [deleted]
- p-e-w 3y ago
  I strongly disagree. Software engineering is simply the next higher abstraction level above programming. Just like programming is the level of abstraction above writing machine code, and writing machine code is the level above hardware manipulation. Almost all "programming" problems have been solved. Today's software is laughably inefficient not because we need better implementations of data structures, but because putting all those existing pieces together properly is a neglected discipline. The best place to see this is in Machine Learning. The vast majority of ML/AI software that is published (which often implements revolutionary techniques!) is of incredibly poor quality from a software engineering standpoint. As a result, practical AI applications are orders of magnitude slower than they could be.
  - nsguy 3y ago
    Not to split hairs but writing machine code is programming. I'm not sure what you're thinking about as "hardware manipulation". Programming is just one (and a very important one) part of what it takes to be a "software engineer" (or software developer, or whatever you want to call having skills to deliver, or be part of a team that delivers, major software projects).
  - ericjmorey 3y ago
    "The vast majority of ML/AI software that is published (which often implements revolutionary techniques!) is of incredibly poor quality from a software engineering standpoint. As a result, practical AI applications are orders of magnitude slower than they could be." That's quite the claim. Why should anyone take your word for it?
    - p-e-w 3y ago
      There are literally performance breakthroughs being made every few weeks (most recently, PowerInfer, which can speed up LLM inference by 10x or more) where the main improvement amounts to the application of caching/preloading techniques. If the "engineering" part of ML had kept up with the "science" part, I have no doubt that performance for typical use cases would be 100x-1000x higher than we are seeing today.
      - wiseowise 3y ago
        > If the "engineering" part of ML had kept up with the "science" part, I have no doubt that performance for typical use cases would be 100x-1000x higher than we are seeing today. Even more of a claim than previous one.
  - Xeamek 3y ago
    >Almost all "programming" problems have been solved. Today's software is laughably inefficient not because we need better implementations of data structures, but because putting all those existing pieces together properly is a neglected discipline. YES! But You learn that writing low level code, not high level. Writing Text editor, or even a compiler, you don't spend much time on algorithm optimization. You spend it on designing good abstractions. You are given basic tools, and have to come up with good architecture on your own. And because you are forced to use more levels then with pre-made libraries (that create the structure for you), you will see your mistakes much faster.
- slumberdisrupt 3y ago
  I genuinely love your sentiment. We have way too much faux asceticism and way too little self-mockery in this godforsaken industry that pays us all way too much money. Programming is wizardry. But being a professional one (a software engineer) is a ridiculous occupation we're coerced into. We all know this, otherwise Silicon Valley wouldn't have been a universally cathartic show. But GP is completely correct that all of the fun, magical projects in the OP article are antithetical to the profession.
  - whateveracct 3y ago
    > But GP is completely correct that all of the fun, magical projects in the OP article are antithetical to the profession. I guess I was emphasizing GP's point. The profession is the problem. Not the subject matter. I love computers.
Turing_Machine 3y ago
What if we don't want to be "software engineers"? I'm still not totally convinced that "software engineer" is even a thing, frankly.
- p-e-w 3y ago
  Who do you think puts systems like a web search engine together? It's certainly not people who are lost in details like string matching or how ACID is implemented in distributed databases. Of course you need all these things in order for the search engine to work – but if you write them yourself, even if you think about them excessively, you will never finish. Software engineering is knowing that you don't need to know everything, you only need to know where to find everything. "Programming" is an important part of it, but far from the only one.
  - deleted 3y ago
    [deleted]
  - steve_taylor 3y ago
    > Who do you think puts systems like a web search engine together? A really good programmer who is free from the tyranny of software “engineering”.
    - carom 3y ago
      I believe this more and more as I go through my career. There is usually one person carrying progress hard. If you can get three or four of those people and another one to coordinate them you can do truly amazing things. Usually though you just need one unencumbered by bureaucracy.
      - sfn42 3y ago
        Not sure what you guys think software engineering means, but it's definitely not the same as what I think it means.
samstave 3y ago
>> Writing code that will still be readable to you (and others) a few years from now - This will be a wonderful feature for copilots, to allow for the human to provide a verbal narrative of the logic/process or even the environment and of others in the room commenting on why a such-and-such is being done, and the copilot elegantly documenting the narrative. Wait until we have "Thick Code" (where an app comes with an AI generated Mini-Documentary-Series on the creation of the piece (as an optional function of your enterprise Copilot Agent (With the [Security Assistant] [Compliance Assistant] [Legal Assistant] [Media Assistant] and whatever Alignment Agents are required and inform the narrative of the project. EDIT: @Tao3300 - I do that all the time. You just need to be a better MeatPT and infer "like, my, opinion, man"
- Tao3300 3y ago
  I suffered a fatal error because I expected a few end parentheses there.
toast0 3y ago
I'm working on a hobby OS. It's mostly just a fun project, but there's certainly a lot of deciding what to write and what to use, and identifying quality libraries when you want to use something. You can start from nothing and build your own board and bring it up with no software you didn't write or you can use a commercially available board, bios or uefi boot, use an existing boot loader, use an existing libc, etc etc, and explore the specific thing you're interested in, while still earning a lot of systems knowledge. Deciding when to optimize isn't too hard when nobody is ever going to use it (only optimize run time if something really takes too long, otherwise don't do anything tremendously stupid, but optimize for less development time because this is taking forever) Readable code would be nice, because sometimes it takes a long time to get back to it, but I've also designed in a real reason to come back and retouch everything (x86 -> amd64; aarch64 as a stretch) and maybe I'll make things reasonable then... Although it's not my strongest skill. Even most simple OSes become complex. Maybe the dependency chain isn't too long though. Seems to tick all your boxes, IMHO.
short_throw 3y ago
If you want learn how things scale across a team and last years, read or contribute to open source code. It takes years for a single person to get a project to the point where it's a good learning ground for scaling and maintenance. Gluing a few libraries together is real software engineering but unless you're really invested in the outcome it's not that engaging and it's not that educational.
- p-e-w 3y ago
  > Gluing a few libraries together is real software engineering but unless you're really invested in the outcome it's not that engaging and it's not that educational. That's only true if complex systems don't interest you. Personally, I have always found the experience of "putting the pieces together" and orchestrating highly diverse systems into a coherent whole to be much more educational than learning about algorithmic details. I also generally find making things work well more interesting than making things work.
  - tonyg 3y ago
    > That's only true if complex systems don't interest you. Gluing a few libraries together will certainly produce a complex system.
  - exe34 3y ago
    So if I'm understanding your argument correctly, if you enjoy it, it's what everybody else should do. The other points are just post-hoc justification.
  - Xeamek 3y ago
    I think your view of low-level projects from scratch is simply wrong, if you describe them as 'algorithmic details'.
sarafiq 3y ago
There is a need for both "solving complex problems alone" and "learning how to work as a team developing a complex project within a timeline". The Software Engineering part is best learnt in industry under a mentor.
- DeathArrow 3y ago
  The tooth extraction skill is fundamental if you are a dentist. You learn it in school. Knowing how to market and sell that skill is valuable but you can learn it while working. CS I learned in school, engineering I learned while working.
Sparkyte 3y ago
I agree with this, only working on a project with certain expectations of delivery can make you a better software engineer.
bufio 3y ago
Gentlemen, you can't hack in here; this is Hacker News.
- arcanemachiner 3y ago
  I'm shocked, SHOCKED, to find that hacking is going on in here!
- deleted 3y ago
  [deleted]
anta40 3y ago
I think those who deal with OS kernel, compiler etc on daily basis also understand those points. They use various tools to make code review, time management etc easier. Not all hackers are overly obsessed with the most efficient Big O and various low level (sorry pun intended) details.
- DeathArrow 3y ago
  >Not all hackers are overly obsessed with the most efficient Big O and various low level (sorry pun intended) details. A NAND gate is low level. Big O is just theory that anyone should know if he's worth something as a programmer.
  - Arcanum-XIII 3y ago
    Most of the time, big O is not relevant if you're not writing new algorithm. For the usual CRUD app? Not so much. Heck even for more involved work, you will not touch them because your solution will be called or will call some external tools. The good news is that, while still valuable, you won't have to lose time creating a new broken wheel. The bad news is that this part of CompSci is now mostly fundamental research.
    - atq2119 3y ago
      And then you get needlessly inefficient CRUD apps that take minutes or even hours to generate a simple report because they're doing accidentally quadratic or even cubic work somewhere.
    - sfn42 3y ago
      This is just wrong. Understanding the difference between O(1), O(n) etc is essential for literally everyone who writes code. Every single programmer is better with this understanding than without it. You should know the complexity of the code you write - and most of the time that doesn't even require actively thinking about it. If you know the basics of complexity analysis it's just intuitive.
da4id 3y ago
Do you have any good resources for what you mentioned?
- yoyohello13 3y ago
  Have good critical thinking skills and you’re pretty much there.
- p-e-w 3y ago
  I wish. As you can plainly see in this thread, most "real programmers" consider such things beneath them, and the scarcity of relevant resources is a natural consequence.
  - wiseowise 3y ago
    Nobody considers them beneath them, you just found a straw man and keep doubling down on it.
johnnyanmac 3y ago
I figured that was the point here. It's nice to follow tutorials for a new language or framework or whatnot, but to really stress and test what you absorbed, you want some non-trivial project with minimal dependencies. the game section exemplies this the best: >The idea here is to implement a well-defined game from start to finish without getting bogged down on the other fun stuff (e.g., game design and art). Also, it is best if you use a barebones 2D graphics library (e.g., SDL, SFML, PyGame), not a big game engine that'll hide all of the interesting bits from you. This kind of game won't go on sale nor even be a portfolio piece for a non-junior engineer, but it's a good enough idea for someone like me who say, wanted to really wanted to solidify their Rust skills and show what my and the language's weaknesses are. After I can get that done, I may feel confident enough to move on to work on a toy renderer based on what I figured out in the game. And then later contributing to a proper engine that already has a lot of groundwork figured out. I'd just be getting in the way if I jumped straight into trying to throw anything but the simplest PRs into Bevy.
bee_rider 3y ago
I don’t really see how one could have any hope of performing engineering with any sort of rigor while throwing away big-O. Then again, big-O is useless in many cases because real computers have too many arbitrary performance thresholds. I suspect software engineering is impossible, or at least, nobody has made the model required to do it.
- sfn42 3y ago
  Going from O(n) to O(1) on an operation where n is 10 could be a performance downgrade. That doesn't mean it's useless, it just means there's more to it than the big O. Asymptotic complexity is about how things scale up. You should care about it when you're working with things that scale. Doesn't mean it's useless if your data is static, just means you need to understand when to go for the high scaling solution and when not to.
  - kragen 3y ago
    if your input data doesn't scale then either ① you're writing a real-time control program such as a motor controller and you care about not just the asymptotic complexity of your algorithms but their worst-case execution time, in microseconds; or ② you should just do the computation by hand with pencil and paper instead of writing and debugging a program to do it for you at the point that the input data becomes too big for ② to be an appealing option, n is at least 100, which means you probably care a lot about whether your program is O(n) or O(n⁴), at least if you wrote it in something slow like python maybe the time to start worrying about it is after you start the program debugged and running for the first time
  - bee_rider 3y ago
    It can be useful in general, but it is hard to provide the type of tolerances expressed in physical units you’d expect in an engineered solution when all the constants are ignored.
    - sfn42 3y ago
      I'm not sure what you mean. Sounds a bit like you might be misunderstanding what were talking about.
steve_taylor 3y ago
Notice it said programmer, not software engineer. Speaking for myself, software engineering is something I do because I have bills to pay. Programming is something I started doing because it’s fun. The projects in this article are the types of projects I would do for fun if I had the time.
- DeathArrow 3y ago
  >Speaking for myself, software engineering is something I do because I have bills to pay. Programming is something I started doing because it’s fun. Same for me. While I strive to do stuff fast, in a way that is easy to maintain, modify and extend for work, see the big picture, take the right time of shortcuts, understand business and customers, there is not were the fun is. I enjoy the time when I am alone with the PC, far from user stories and scrums. I love to program, to convince the PC to do the trick I tell it to do. What I enjoy also is CS, algorithms, data structures, abstract thinking.
- ixaxaar 3y ago
  Also, I'd add that doing HoTT or other such "high theory" stuff actually enhances my understanding of the fundamentals and gives me the machinery to handle complexities systematically that building a mere web search engine would not. I already have a job for building such commodity stuff everyday.
- tugberkk 3y ago
  I like watching Tsoding in my spare time and I love what he calls "recreational programming". That is what we need imho, more recreational programming.
- serial_dev 3y ago
  > Software engineering is something I do because I have bills to pay. Programming is something I started doing because it’s fun. Sudden clarity. I feel like printing it and putting it next to my workstation, so that I remember this both when doing something fun and when having to work so that my bills get paid.
  - 3y ago
DeathArrow 3y ago
>While writing a text editor, a compiler, an operating system, or a raytracer might make you a better programmer, it won't make you a better software engineer. To me learning the engineering part is easiest. And also not terribly valuable if you don't know CS fundamentals.
tehnub 3y ago
This is like telling a student learning an instrument never to practice scales or études because there are other skills necessary for playing in an orchestra. Those other skills are important, but you have to develop your chops at some point, and being a better a programmer will certainly help with basically all those skills you listed anyway.
- p-e-w 3y ago
  > This is like telling a student learning an instrument never to practice scales or études No, it's like recognizing that an orchestra conductor doesn't need to play every instrument (or even any instrument!) in order to make music. Or alternatively, that a violin player doesn't need to understand the physics of acoustic dispersion in order to be the best in the world at playing the violin.
  - tock 3y ago
    Yes a software engineer understanding data structures and runtime complexity is akin to a violin player understanding the physics of acoustic dispersion.
    - p-e-w 3y ago
      Is it not? I can't tell if you're being sarcastic or not.
      - Arch-TK 3y ago
        I would say, given how absurd the statement is, sarcastic.
      - khazhoux 3y ago
        The analogy would be a software engineer having to understand resistors and capacitors and transistors. Also, I don’t think you’ll find a single conductor on this planet who is not skilled in at least one instrument. Terrible analogy.
    - 3y ago
bmitc 3y ago
> In that spirit, I offer the following alternative challenge: Create a web search engine. Yea, no thanks. If I'm going to work on something in my spare time, I am going to work on something that I am interested in and find fun. I generally loathe the dirty work often involved in software engineering and debugging all of these dependencies all the time. It's exhausting. Doing things from scratch puts me in charge. And I mean, creating a search engine is the ultimate "not invented here" project. I'd just pay for and use Google, Bing, or whatever else API to do things for me. So it doesn't even make sense in the context of your objections.
- p-e-w 3y ago
  > And I mean, creating a search engine is the ultimate "not invented here" project. I'd just pay for and use Google, Bing, or whatever else API to do things for me. So it doesn't even make sense in the context of your objections. Except that none of the established search engines actually does full-text search (anymore). There is no way to prevent those engines from breaking apart and rewriting your query as they see fit. No matter how many quotation marks you put around it, and even if you enable "Verbatim" mode or whatever. So if you just implement the option to do that (which is actually the default if you leverage a standard FTS library), you already have something that AFAIK you can't get anywhere else. Then you can work on kicking the blogspam out of your index, and before long you will have a search engine of real value for yourself (and possibly others). Or, you can build a crappy clone of a text editor from the 70s.
  - exe34 3y ago
    This sounds like an excellent project! I look forward to following your work on GitHub. Everybody needs a crappy search engine from the 90s!
  - mkii 3y ago
    > and before long you will have a search engine of real value for yourself (and possibly others) How much do you think hosting would cost?
bobsomers 3y ago
I don't understand why you think a search engine requires the use of "real engineering skills" like picking and choosing libraries and identifying which opportunities will yield fruitful optimization. Literally all of the listed projects, text editors, compilers, operating systems, and ray tracers, can exercise the exact same activities. I'm more inclined to think that your comment is really more revealing about what kind of knowledge is in your own personal wheelhouse. Having worked on a ray tracer that was subsequently used on many feature films, I can assure you that all of your bullet points apply to that project, too. It's much more a matter of whether you want to do something small scale and fun, or whether you want to suck all the joy out of it by applying the same soul crushing constraints we already get paid to do in our day jobs. Bleh.
- PH95VuimJjqBqy 3y ago
  > I'm more inclined to think that your comment is really more revealing about what kind of knowledge is in your own personal wheelhouse. That's my read as well.
- p-e-w 3y ago
  > Literally all of the listed projects, text editors, compilers, operating systems, and ray tracers, can exercise the exact same activities. In the linked article, these projects are all explicitly described as opportunities to learn about low-level stuff like how to efficiently store editable text. The difference with a web search engine is that nobody today can build such a thing completely from scratch, therefore it forces you to give up the toxic NIH mentality, which in my experience is usually driven by elitism and ego (on full display in this comment thread), not by some lofty desire to learn something new.
  - 59nadir 3y ago
    > The difference with a web search engine is that nobody today can build such a thing completely from scratch I'm sorry, but can you substantiate this claim? I've seen no indication that a search engine is not buildable from scratch at all.
BlackjackCF 3y ago
I agree that these are all valuable things for a software engineer to learn, but in my experience, it’s valuable to at least take a crack at the projects listed in the post. I wasn’t under any illusion that I was going to build anything amazing, and gave me deeper appreciation of why you don’t try and build everything in-house.
auggierose 3y ago
This sounds like sensible advice. But there are a few problems with making software purely from ready-made building blocks. Here are two of them: 1) Much more often than not, the ready-made building blocks are crap. Your software will reflect that crappiness, and your life will consist not of writing software, but of maintaining and massaging crap. 2) Even if your ready-made building blocks are high-quality, they limit what kind of software you can write. Ideally, you imagine what you want your software to do, and then you write software to do it. You create the building blocks in that process. If you are only using ready-made building blocks, there is a lot of great software you just cannot write. And I think we all see that, because there isn't much great software around. These problems might not be a concern for you. If you write software that is pretty much just more of the same, 2) does not apply to you. And while 1) is still problematic, it will be just manageable because you use these building blocks mostly on their happy path. But I have bad news for you. This kind of software will soon be written not by you, but by an AI.
- p-e-w 3y ago
  > If you are only using ready-made building blocks, there is a lot of great software you just cannot write. And I think we all see that, because there isn't much great software around. I claim that the opposite is true: The reason there are so few software projects that are really great is because high-level skills, not low-level ones, are in short supply. Every CS graduate can implement Paxos. The number of people with the experience needed to take an implementation of Paxos, and many other components, and put them together into something that works really well is much, much smaller. > But I have bad news for you. This kind of software will soon be written not by you, but by an AI. AI is actually much better at writing low-level code than high-level code. If you tell it to implement a piece table or a Fibonacci heap, it will do so without problems, in any language. If you tell it to create a simple CRUD application using existing libraries, it will output something that doesn't actually work.
  - auggierose 3y ago
    I guess it all depends on your definition of low-level. High-level can also be low-level, just with different building blocks, if the building blocks are based on well-specified abstractions. I am actually currently writing a novel text editor, and I can guarantee you, my requirements mean that an AI cannot help me with that directly, because the underlying mechanisms I need don't exist yet. The AI is still helpful though when researching existing mechanisms, and to pinpoint why they don't work for me. > AI is actually much better at writing low-level code than high-level code. > If you tell it to create a simple CRUD application using existing libraries, it will output something that doesn't actually work. If your low-level code is just a copy or adaptation of something existing, then yes, AI is really good at that. Something novel though, AI cannot help you much here. On the other hand, simple CRUD applications will all be done by AI very soon. Probably not using existing libraries though, because these are crap. But there is certainly a role for humans currently in high-level design, I agree with you here. But this high-level design crucially relies on your ability for low-level design, with the ability to imagine and then implement (possibly with the help of AI) the building blocks you need, not just reuse the ones that are there.
hurril 3y ago
You're making a strawman here, based on a very much false dichotomy. The things you list here are all very important. But something you learn from writing a larger piece of software yourself is that you get to appreciate complexity and "bulk" (for lack of a better term) that you yourself created. You get to ask yourself questions like: Is all this code needed? (Why is it 400 lines to do X?!) Why did this get so involved/ complicated?! How do I progress (eat the elefant)? Can I (teach myself to) make small iterative units of progress every day, for instance? These kinds of katas are very very useful for a 5-7 year beginner (which I use as a label of respect - the longer we can remain beginners, the longer we learn with an open attitude!)
deleted 3y ago
[deleted]
cryogenicfire 3y ago
I don't understand why you believe low-level "intrusive" programming and general software engineering are mutually exclusive. There is a big, open field for high quality, low-level software to be written well bearing good practices and practical decisions. 1. To be faif, these kinds of "hacking" projects aren't technically meant for learning software engineering in the context of a job, but they are really interesting and a treasure trove of new knowledge for anyone willing to put in the effort. I don't think it's a bad thing. A person can learn to be a good software developer and simultaneously have deep knowledge of the systems they are working with. 2. The goal of this project is not to research and implement some novel algorithm to replace existing works. The goal is to learn how things work. You are still going to use libraries, procedures and algorithms designed by other people and compile them into a cohesive system, it is merely the depth and complexity of the problem that makes it an interesting learning challenge. It's not about some absurd NIH philosophy, it's about curiosity. 3. Software development is not as straightforward as you make it seem. It's not just about best practices, management and finding an optimal way to combine existing libraries into your project. Sometimes you run into unforeseen issues, strange niches that your libraries may not account for. Sometimes you need to dive into the specifics of these libraries, understand at a lower level why they do not function for your use case. You might need to "hack" these libraries. Fork and customise them to truly fit in your project and produce the desired results. And the only thing that will help you then is deep knowledge and the ability to work with low level software.
- tester756 3y ago
  >I don't understand why you believe low-level "intrusive" programming and general software engineering are mutually exclusive. Hmm, while I don't fully agree with his comment, then your comment reminded me how I hate to discuss programming languages and programming ecosystems with C people. It feels like C/low lvl people often refer to some "standard" that they treat as a bible of programming languages even when nobody mentions C and they treat GCC/CLANG as a perfect reference for compilers. So, while high level devs dont understand low lvl stuff, then I believe low lvl programmers aren't creative at very abstract level, very abstract system modeling cuz they're too used to modeling everything with integers in C. And they're too used to being hurt by C ecosystem that they believe it is "normal". High level people (web developers) spend a lot of time arguing about their buzzwords like microservices, ddd, oop vs fp, etc.
melagonster 3y ago
Is not this the main point of the editor war? programmers built tool for enhancing programming abilities, but they argued with which tool is best one...
PedroBatista 3y ago
I could not disagree more. If it is not not invented here, on the limit is not invented anywhere because nobody knows how to build these things, which we have been on that path more or less. Perhaps you were visualizing that guy on your job who creates his own libraries and obscure systems within the system before telling anyone and that is generally a bad peer. But the Universe is much bigger and diverse than that, people who programmed their own things but do a Google search first and use other software because they know it’s more feature complete, more known, trusted and better future supported are the best to work with. We are all using the products of not invented here syndrome right now, don’t forget that.
ess3 3y ago
Actually reimplementing something that’s already exists is a great way to get an understanding of it precisely so you can deal with the questions you mention. My rule of thumb is that you should always understand at least one level of abstraction down.
goto11 3y ago
> Identifying high-quality libraries and frameworks that meet your project needs This is a really important skill that I find difficult and frustrating. Does anyone have good advice or resources?
- deterministic 3y ago
  It’s simple really: write libraries and frameworks yourself and use them for real projects. It will give you a great understanding of what makes a quality library/framework and make you a better programmer.
  - goto11 3y ago
    Are you perchance working in academia? Your suggestion seems to assume unlimited time and resources to write everything yourself, while the purpose of using third-party components usually is to save time and resources.
    - deterministic 3y ago
      Nope. 30+ years software developer in industry. Think about it as a hobby. Most people have time for hobbies. The rest spend way too much time on social media or watching YouTube. Manage your time better.
      - goto11 3y ago
        > Manage your time better. Thank you for the advice which is no doubt well-intentioned. But don't kid yourself. If you use your spare time for research and development to solve problems at work, then it is not really a hobby, it is just you working overtime for free. If that makes you happy, good for you, but it doesn't change the fact that time and resources are limited. The time is just cheaper in your case.
        deterministic 3y ago
        I am spending time educating myself. That is super valuable for me and opens up new career opportunities. And yes it might also happen to benefit the job I do currently. But that is not the #1 priority.
hombre_fatal 3y ago
Honestly this just sounds like me when I first encountered coding puzzles like project euler / leetcode and interview challenge practice problems. I made up excuses like this because I simply didn't want to spend time on lower level problems and instead wanted to feel good sticking to my high level application comfort zone. I eventually learned to like solving lower level problems and I'm much better off for it. And these days I can tell people who haven't paid their dues because the only solutions they ever come up with are slow quadratic ones no matter the occasion. No, building your own text editor, compiler, OS, ray tracer is not going to make you a worse engineer nor keep you from learning how to google and evaluate libraries or think critically, nor are those things reserved for an engineer who isn't building those things.
culi 3y ago
> In fact, it might make you worse at software engineering, because it embodies the disastrous "Not Invented Here" doctrine I get your point against NIH, but I don't see how doing such a project on your own in your freetime could possibly make you a worse programmer. If anything, you would likely learn exactly what it takes to go down the NIH path so strictly and be able to make better choices
heresie-dabord 3y ago
> might make you a better programmer, it won't make you a better software engineer This is the distinction between solving "local" problems and solving problems at scale. But I think there is a transitional phase where requirements (gathering, documenting), unit testing, UI design (human or programmatical) are essential. Learning these as nuts-and-bolts in any language is essential to becoming skilled in the craft.
pmelendez 3y ago
There are a few things on this comments that is causing noise in my mind. >might make you a better programmer, it won't make you a better software engineer I think it is a short-sighted definition of SE but in any case, there are not any mention anywhere in the article claiming that this was aimed to software engineers. Also, although I appreciate the concerns of "Not Invented Here" doctrine, we also have the other side of the spectrum, where we don't know how to do a left-pad anymore [1], and introduce dependencies everywhere. The third point is that I don't think that having more knowledge on how things works contributes with the "Not Invented Here" mindset. I would argue is the opposite. When engineers are hungry on learning, they find excuses to make address that appetite at work, but once they know the tradeoffs and amount of effort to make it work, they will think twice before starting anything from scratch. [1] https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code https://qz.com/646467/how-one-programmer-broke-the-internet-...
ryloric 3y ago
Idk why everyone is piling on this comment, they clearly make a distinction between programming and software engineer.
mebassett 3y ago
There is a lot of disagreement towards this idea in the thread, but I really like this project suggestion for someone to make a multi-faceted project that'll help someone "grow up" and it resonates with something I've long felt missing from comp sci education: that it doesn't focus enough on studying and replicated big successful projects. You mostly learn by doing, true, but you learn by copying what the masters did. Software engineering education feels more like taking classes and writing code for exercises, whereas it probably should have more projects of the form "study this complicated-implementation-of-X and use those techniques in your implementation of Y", where x = google search, postgresql, whatevers and y is a complicated project [for one person or a small team] like your own single concurrent user search engine I've held this idea for a really long time, in fact, I've not actually looked at comp sci education recently, so maybe this isn't true anymore, or maybe it never has been
yawboakye 3y ago
one wonders whether you’ve created having completed two of the items on the list, i can confirm that it makes you better, whichever way you’d like to measure it. as a google engineer told me once: operating systems and compilers are where all the good ideas are. study them! only thing that beats that is probably writing them yourself, even if toy versions.
frankie_t 3y ago
It really depends on your goals. Why would someone want to become a better software engineer? To me, the things you listed aren't fun, so I wouldn't do them in my spare time unless I see the benefit in it. If we are talking making more money at a job, I'd say that for most people I worked with and me included there are more important things to concentrate on. Like, soft skills, communicating clearly, understanding politics at work. Myself, I'm okay with learning things you listed from more senior colleagues, on the job, and just having fun in my spare time.
4m1rk 3y ago
You can't be a better software engineer before being a better programmer.

by p-e-w 3y ago

whateveracct 3y ago

[flagged]

deleted 3y ago
[deleted]
p-e-w 3y ago
I strongly disagree. Software engineering is simply the next higher abstraction level above programming. Just like programming is the level of abstraction above writing machine code, and writing machine code is the level above hardware manipulation. Almost all "programming" problems have been solved. Today's software is laughably inefficient not because we need better implementations of data structures, but because putting all those existing pieces together properly is a neglected discipline. The best place to see this is in Machine Learning. The vast majority of ML/AI software that is published (which often implements revolutionary techniques!) is of incredibly poor quality from a software engineering standpoint. As a result, practical AI applications are orders of magnitude slower than they could be.
- nsguy 3y ago
  Not to split hairs but writing machine code is programming. I'm not sure what you're thinking about as "hardware manipulation". Programming is just one (and a very important one) part of what it takes to be a "software engineer" (or software developer, or whatever you want to call having skills to deliver, or be part of a team that delivers, major software projects).
- ericjmorey 3y ago
  "The vast majority of ML/AI software that is published (which often implements revolutionary techniques!) is of incredibly poor quality from a software engineering standpoint. As a result, practical AI applications are orders of magnitude slower than they could be." That's quite the claim. Why should anyone take your word for it?
  - p-e-w 3y ago
    There are literally performance breakthroughs being made every few weeks (most recently, PowerInfer, which can speed up LLM inference by 10x or more) where the main improvement amounts to the application of caching/preloading techniques. If the "engineering" part of ML had kept up with the "science" part, I have no doubt that performance for typical use cases would be 100x-1000x higher than we are seeing today.
    - wiseowise 3y ago
      > If the "engineering" part of ML had kept up with the "science" part, I have no doubt that performance for typical use cases would be 100x-1000x higher than we are seeing today. Even more of a claim than previous one.
- Xeamek 3y ago
  >Almost all "programming" problems have been solved. Today's software is laughably inefficient not because we need better implementations of data structures, but because putting all those existing pieces together properly is a neglected discipline. YES! But You learn that writing low level code, not high level. Writing Text editor, or even a compiler, you don't spend much time on algorithm optimization. You spend it on designing good abstractions. You are given basic tools, and have to come up with good architecture on your own. And because you are forced to use more levels then with pre-made libraries (that create the structure for you), you will see your mistakes much faster.
slumberdisrupt 3y ago
I genuinely love your sentiment. We have way too much faux asceticism and way too little self-mockery in this godforsaken industry that pays us all way too much money. Programming is wizardry. But being a professional one (a software engineer) is a ridiculous occupation we're coerced into. We all know this, otherwise Silicon Valley wouldn't have been a universally cathartic show. But GP is completely correct that all of the fun, magical projects in the OP article are antithetical to the profession.
- whateveracct 3y ago
  > But GP is completely correct that all of the fun, magical projects in the OP article are antithetical to the profession. I guess I was emphasizing GP's point. The profession is the problem. Not the subject matter. I love computers.

Turing_Machine 3y ago

What if we don't want to be "software engineers"? I'm still not totally convinced that "software engineer" is even a thing, frankly.

p-e-w 3y ago
Who do you think puts systems like a web search engine together? It's certainly not people who are lost in details like string matching or how ACID is implemented in distributed databases. Of course you need all these things in order for the search engine to work – but if you write them yourself, even if you think about them excessively, you will never finish. Software engineering is knowing that you don't need to know everything, you only need to know where to find everything. "Programming" is an important part of it, but far from the only one.
- deleted 3y ago
  [deleted]
- steve_taylor 3y ago
  > Who do you think puts systems like a web search engine together? A really good programmer who is free from the tyranny of software “engineering”.
  - carom 3y ago
    I believe this more and more as I go through my career. There is usually one person carrying progress hard. If you can get three or four of those people and another one to coordinate them you can do truly amazing things. Usually though you just need one unencumbered by bureaucracy.
    - sfn42 3y ago
      Not sure what you guys think software engineering means, but it's definitely not the same as what I think it means.

samstave 3y ago

>> Writing code that will still be readable to you (and others) a few years from now - This will be a wonderful feature for copilots, to allow for the human to provide a verbal narrative of the logic/process or even the environment and of others in the room commenting on why a such-and-such is being done, and the copilot elegantly documenting the narrative. Wait until we have "Thick Code" (where an app comes with an AI generated Mini-Documentary-Series on the creation of the piece (as an optional function of your enterprise Copilot Agent (With the [Security Assistant] [Compliance Assistant] [Legal Assistant] [Media Assistant] and whatever Alignment Agents are required and inform the narrative of the project. EDIT: @Tao3300 - I do that all the time. You just need to be a better MeatPT and infer "like, my, opinion, man"

Tao3300 3y ago
I suffered a fatal error because I expected a few end parentheses there.

toast0 3y ago

I'm working on a hobby OS. It's mostly just a fun project, but there's certainly a lot of deciding what to write and what to use, and identifying quality libraries when you want to use something. You can start from nothing and build your own board and bring it up with no software you didn't write or you can use a commercially available board, bios or uefi boot, use an existing boot loader, use an existing libc, etc etc, and explore the specific thing you're interested in, while still earning a lot of systems knowledge. Deciding when to optimize isn't too hard when nobody is ever going to use it (only optimize run time if something really takes too long, otherwise don't do anything tremendously stupid, but optimize for less development time because this is taking forever) Readable code would be nice, because sometimes it takes a long time to get back to it, but I've also designed in a real reason to come back and retouch everything (x86 -> amd64; aarch64 as a stretch) and maybe I'll make things reasonable then... Although it's not my strongest skill. Even most simple OSes become complex. Maybe the dependency chain isn't too long though. Seems to tick all your boxes, IMHO.

short_throw 3y ago

If you want learn how things scale across a team and last years, read or contribute to open source code. It takes years for a single person to get a project to the point where it's a good learning ground for scaling and maintenance. Gluing a few libraries together is real software engineering but unless you're really invested in the outcome it's not that engaging and it's not that educational.

p-e-w 3y ago
> Gluing a few libraries together is real software engineering but unless you're really invested in the outcome it's not that engaging and it's not that educational. That's only true if complex systems don't interest you. Personally, I have always found the experience of "putting the pieces together" and orchestrating highly diverse systems into a coherent whole to be much more educational than learning about algorithmic details. I also generally find making things work well more interesting than making things work.
- tonyg 3y ago
  > That's only true if complex systems don't interest you. Gluing a few libraries together will certainly produce a complex system.
- exe34 3y ago
  So if I'm understanding your argument correctly, if you enjoy it, it's what everybody else should do. The other points are just post-hoc justification.
- Xeamek 3y ago
  I think your view of low-level projects from scratch is simply wrong, if you describe them as 'algorithmic details'.

sarafiq 3y ago

There is a need for both "solving complex problems alone" and "learning how to work as a team developing a complex project within a timeline". The Software Engineering part is best learnt in industry under a mentor.

DeathArrow 3y ago
The tooth extraction skill is fundamental if you are a dentist. You learn it in school. Knowing how to market and sell that skill is valuable but you can learn it while working. CS I learned in school, engineering I learned while working.

Sparkyte 3y ago

I agree with this, only working on a project with certain expectations of delivery can make you a better software engineer.

bufio 3y ago

Gentlemen, you can't hack in here; this is Hacker News.

arcanemachiner 3y ago
I'm shocked, SHOCKED, to find that hacking is going on in here!
deleted 3y ago
[deleted]

anta40 3y ago

I think those who deal with OS kernel, compiler etc on daily basis also understand those points. They use various tools to make code review, time management etc easier. Not all hackers are overly obsessed with the most efficient Big O and various low level (sorry pun intended) details.

DeathArrow 3y ago
>Not all hackers are overly obsessed with the most efficient Big O and various low level (sorry pun intended) details. A NAND gate is low level. Big O is just theory that anyone should know if he's worth something as a programmer.
- Arcanum-XIII 3y ago
  Most of the time, big O is not relevant if you're not writing new algorithm. For the usual CRUD app? Not so much. Heck even for more involved work, you will not touch them because your solution will be called or will call some external tools. The good news is that, while still valuable, you won't have to lose time creating a new broken wheel. The bad news is that this part of CompSci is now mostly fundamental research.
  - atq2119 3y ago
    And then you get needlessly inefficient CRUD apps that take minutes or even hours to generate a simple report because they're doing accidentally quadratic or even cubic work somewhere.
  - sfn42 3y ago
    This is just wrong. Understanding the difference between O(1), O(n) etc is essential for literally everyone who writes code. Every single programmer is better with this understanding than without it. You should know the complexity of the code you write - and most of the time that doesn't even require actively thinking about it. If you know the basics of complexity analysis it's just intuitive.

da4id 3y ago

Do you have any good resources for what you mentioned?

yoyohello13 3y ago
Have good critical thinking skills and you’re pretty much there.
p-e-w 3y ago
I wish. As you can plainly see in this thread, most "real programmers" consider such things beneath them, and the scarcity of relevant resources is a natural consequence.
- wiseowise 3y ago
  Nobody considers them beneath them, you just found a straw man and keep doubling down on it.

johnnyanmac 3y ago

I figured that was the point here. It's nice to follow tutorials for a new language or framework or whatnot, but to really stress and test what you absorbed, you want some non-trivial project with minimal dependencies. the game section exemplies this the best: >The idea here is to implement a well-defined game from start to finish without getting bogged down on the other fun stuff (e.g., game design and art). Also, it is best if you use a barebones 2D graphics library (e.g., SDL, SFML, PyGame), not a big game engine that'll hide all of the interesting bits from you. This kind of game won't go on sale nor even be a portfolio piece for a non-junior engineer, but it's a good enough idea for someone like me who say, wanted to really wanted to solidify their Rust skills and show what my and the language's weaknesses are. After I can get that done, I may feel confident enough to move on to work on a toy renderer based on what I figured out in the game. And then later contributing to a proper engine that already has a lot of groundwork figured out. I'd just be getting in the way if I jumped straight into trying to throw anything but the simplest PRs into Bevy.

bee_rider 3y ago

I don’t really see how one could have any hope of performing engineering with any sort of rigor while throwing away big-O. Then again, big-O is useless in many cases because real computers have too many arbitrary performance thresholds. I suspect software engineering is impossible, or at least, nobody has made the model required to do it.

sfn42 3y ago
Going from O(n) to O(1) on an operation where n is 10 could be a performance downgrade. That doesn't mean it's useless, it just means there's more to it than the big O. Asymptotic complexity is about how things scale up. You should care about it when you're working with things that scale. Doesn't mean it's useless if your data is static, just means you need to understand when to go for the high scaling solution and when not to.
- kragen 3y ago
  if your input data doesn't scale then either ① you're writing a real-time control program such as a motor controller and you care about not just the asymptotic complexity of your algorithms but their worst-case execution time, in microseconds; or ② you should just do the computation by hand with pencil and paper instead of writing and debugging a program to do it for you at the point that the input data becomes too big for ② to be an appealing option, n is at least 100, which means you probably care a lot about whether your program is O(n) or O(n⁴), at least if you wrote it in something slow like python maybe the time to start worrying about it is after you start the program debugged and running for the first time
- bee_rider 3y ago
  It can be useful in general, but it is hard to provide the type of tolerances expressed in physical units you’d expect in an engineered solution when all the constants are ignored.
  - sfn42 3y ago
    I'm not sure what you mean. Sounds a bit like you might be misunderstanding what were talking about.

steve_taylor 3y ago

Notice it said programmer, not software engineer. Speaking for myself, software engineering is something I do because I have bills to pay. Programming is something I started doing because it’s fun. The projects in this article are the types of projects I would do for fun if I had the time.

DeathArrow 3y ago
>Speaking for myself, software engineering is something I do because I have bills to pay. Programming is something I started doing because it’s fun. Same for me. While I strive to do stuff fast, in a way that is easy to maintain, modify and extend for work, see the big picture, take the right time of shortcuts, understand business and customers, there is not were the fun is. I enjoy the time when I am alone with the PC, far from user stories and scrums. I love to program, to convince the PC to do the trick I tell it to do. What I enjoy also is CS, algorithms, data structures, abstract thinking.
ixaxaar 3y ago
Also, I'd add that doing HoTT or other such "high theory" stuff actually enhances my understanding of the fundamentals and gives me the machinery to handle complexities systematically that building a mere web search engine would not. I already have a job for building such commodity stuff everyday.
tugberkk 3y ago
I like watching Tsoding in my spare time and I love what he calls "recreational programming". That is what we need imho, more recreational programming.
serial_dev 3y ago
> Software engineering is something I do because I have bills to pay. Programming is something I started doing because it’s fun. Sudden clarity. I feel like printing it and putting it next to my workstation, so that I remember this both when doing something fun and when having to work so that my bills get paid.
- 3y ago

DeathArrow 3y ago

>While writing a text editor, a compiler, an operating system, or a raytracer might make you a better programmer, it won't make you a better software engineer. To me learning the engineering part is easiest. And also not terribly valuable if you don't know CS fundamentals.

tehnub 3y ago

This is like telling a student learning an instrument never to practice scales or études because there are other skills necessary for playing in an orchestra. Those other skills are important, but you have to develop your chops at some point, and being a better a programmer will certainly help with basically all those skills you listed anyway.

p-e-w 3y ago
> This is like telling a student learning an instrument never to practice scales or études No, it's like recognizing that an orchestra conductor doesn't need to play every instrument (or even any instrument!) in order to make music. Or alternatively, that a violin player doesn't need to understand the physics of acoustic dispersion in order to be the best in the world at playing the violin.
- tock 3y ago
  Yes a software engineer understanding data structures and runtime complexity is akin to a violin player understanding the physics of acoustic dispersion.
  - p-e-w 3y ago
    Is it not? I can't tell if you're being sarcastic or not.
    - Arch-TK 3y ago
      I would say, given how absurd the statement is, sarcastic.
    - khazhoux 3y ago
      The analogy would be a software engineer having to understand resistors and capacitors and transistors. Also, I don’t think you’ll find a single conductor on this planet who is not skilled in at least one instrument. Terrible analogy.
  - 3y ago

bmitc 3y ago

> In that spirit, I offer the following alternative challenge: Create a web search engine. Yea, no thanks. If I'm going to work on something in my spare time, I am going to work on something that I am interested in and find fun. I generally loathe the dirty work often involved in software engineering and debugging all of these dependencies all the time. It's exhausting. Doing things from scratch puts me in charge. And I mean, creating a search engine is the ultimate "not invented here" project. I'd just pay for and use Google, Bing, or whatever else API to do things for me. So it doesn't even make sense in the context of your objections.

p-e-w 3y ago
> And I mean, creating a search engine is the ultimate "not invented here" project. I'd just pay for and use Google, Bing, or whatever else API to do things for me. So it doesn't even make sense in the context of your objections. Except that none of the established search engines actually does full-text search (anymore). There is no way to prevent those engines from breaking apart and rewriting your query as they see fit. No matter how many quotation marks you put around it, and even if you enable "Verbatim" mode or whatever. So if you just implement the option to do that (which is actually the default if you leverage a standard FTS library), you already have something that AFAIK you can't get anywhere else. Then you can work on kicking the blogspam out of your index, and before long you will have a search engine of real value for yourself (and possibly others). Or, you can build a crappy clone of a text editor from the 70s.
- exe34 3y ago
  This sounds like an excellent project! I look forward to following your work on GitHub. Everybody needs a crappy search engine from the 90s!
- mkii 3y ago
  > and before long you will have a search engine of real value for yourself (and possibly others) How much do you think hosting would cost?

bobsomers 3y ago

I don't understand why you think a search engine requires the use of "real engineering skills" like picking and choosing libraries and identifying which opportunities will yield fruitful optimization. Literally all of the listed projects, text editors, compilers, operating systems, and ray tracers, can exercise the exact same activities. I'm more inclined to think that your comment is really more revealing about what kind of knowledge is in your own personal wheelhouse. Having worked on a ray tracer that was subsequently used on many feature films, I can assure you that all of your bullet points apply to that project, too. It's much more a matter of whether you want to do something small scale and fun, or whether you want to suck all the joy out of it by applying the same soul crushing constraints we already get paid to do in our day jobs. Bleh.

PH95VuimJjqBqy 3y ago
> I'm more inclined to think that your comment is really more revealing about what kind of knowledge is in your own personal wheelhouse. That's my read as well.
p-e-w 3y ago
> Literally all of the listed projects, text editors, compilers, operating systems, and ray tracers, can exercise the exact same activities. In the linked article, these projects are all explicitly described as opportunities to learn about low-level stuff like how to efficiently store editable text. The difference with a web search engine is that nobody today can build such a thing completely from scratch, therefore it forces you to give up the toxic NIH mentality, which in my experience is usually driven by elitism and ego (on full display in this comment thread), not by some lofty desire to learn something new.
- 59nadir 3y ago
  > The difference with a web search engine is that nobody today can build such a thing completely from scratch I'm sorry, but can you substantiate this claim? I've seen no indication that a search engine is not buildable from scratch at all.

BlackjackCF 3y ago

I agree that these are all valuable things for a software engineer to learn, but in my experience, it’s valuable to at least take a crack at the projects listed in the post. I wasn’t under any illusion that I was going to build anything amazing, and gave me deeper appreciation of why you don’t try and build everything in-house.

auggierose 3y ago

This sounds like sensible advice. But there are a few problems with making software purely from ready-made building blocks. Here are two of them: 1) Much more often than not, the ready-made building blocks are crap. Your software will reflect that crappiness, and your life will consist not of writing software, but of maintaining and massaging crap. 2) Even if your ready-made building blocks are high-quality, they limit what kind of software you can write. Ideally, you imagine what you want your software to do, and then you write software to do it. You create the building blocks in that process. If you are only using ready-made building blocks, there is a lot of great software you just cannot write. And I think we all see that, because there isn't much great software around. These problems might not be a concern for you. If you write software that is pretty much just more of the same, 2) does not apply to you. And while 1) is still problematic, it will be just manageable because you use these building blocks mostly on their happy path. But I have bad news for you. This kind of software will soon be written not by you, but by an AI.

p-e-w 3y ago
> If you are only using ready-made building blocks, there is a lot of great software you just cannot write. And I think we all see that, because there isn't much great software around. I claim that the opposite is true: The reason there are so few software projects that are really great is because high-level skills, not low-level ones, are in short supply. Every CS graduate can implement Paxos. The number of people with the experience needed to take an implementation of Paxos, and many other components, and put them together into something that works really well is much, much smaller. > But I have bad news for you. This kind of software will soon be written not by you, but by an AI. AI is actually much better at writing low-level code than high-level code. If you tell it to implement a piece table or a Fibonacci heap, it will do so without problems, in any language. If you tell it to create a simple CRUD application using existing libraries, it will output something that doesn't actually work.
- auggierose 3y ago
  I guess it all depends on your definition of low-level. High-level can also be low-level, just with different building blocks, if the building blocks are based on well-specified abstractions. I am actually currently writing a novel text editor, and I can guarantee you, my requirements mean that an AI cannot help me with that directly, because the underlying mechanisms I need don't exist yet. The AI is still helpful though when researching existing mechanisms, and to pinpoint why they don't work for me. > AI is actually much better at writing low-level code than high-level code. > If you tell it to create a simple CRUD application using existing libraries, it will output something that doesn't actually work. If your low-level code is just a copy or adaptation of something existing, then yes, AI is really good at that. Something novel though, AI cannot help you much here. On the other hand, simple CRUD applications will all be done by AI very soon. Probably not using existing libraries though, because these are crap. But there is certainly a role for humans currently in high-level design, I agree with you here. But this high-level design crucially relies on your ability for low-level design, with the ability to imagine and then implement (possibly with the help of AI) the building blocks you need, not just reuse the ones that are there.

hurril 3y ago

You're making a strawman here, based on a very much false dichotomy. The things you list here are all very important. But something you learn from writing a larger piece of software yourself is that you get to appreciate complexity and "bulk" (for lack of a better term) that you yourself created. You get to ask yourself questions like: Is all this code needed? (Why is it 400 lines to do X?!) Why did this get so involved/ complicated?! How do I progress (eat the elefant)? Can I (teach myself to) make small iterative units of progress every day, for instance? These kinds of katas are very very useful for a 5-7 year beginner (which I use as a label of respect - the longer we can remain beginners, the longer we learn with an open attitude!)

deleted 3y ago

[deleted]

cryogenicfire 3y ago

I don't understand why you believe low-level "intrusive" programming and general software engineering are mutually exclusive. There is a big, open field for high quality, low-level software to be written well bearing good practices and practical decisions. 1. To be faif, these kinds of "hacking" projects aren't technically meant for learning software engineering in the context of a job, but they are really interesting and a treasure trove of new knowledge for anyone willing to put in the effort. I don't think it's a bad thing. A person can learn to be a good software developer and simultaneously have deep knowledge of the systems they are working with. 2. The goal of this project is not to research and implement some novel algorithm to replace existing works. The goal is to learn how things work. You are still going to use libraries, procedures and algorithms designed by other people and compile them into a cohesive system, it is merely the depth and complexity of the problem that makes it an interesting learning challenge. It's not about some absurd NIH philosophy, it's about curiosity. 3. Software development is not as straightforward as you make it seem. It's not just about best practices, management and finding an optimal way to combine existing libraries into your project. Sometimes you run into unforeseen issues, strange niches that your libraries may not account for. Sometimes you need to dive into the specifics of these libraries, understand at a lower level why they do not function for your use case. You might need to "hack" these libraries. Fork and customise them to truly fit in your project and produce the desired results. And the only thing that will help you then is deep knowledge and the ability to work with low level software.

tester756 3y ago
>I don't understand why you believe low-level "intrusive" programming and general software engineering are mutually exclusive. Hmm, while I don't fully agree with his comment, then your comment reminded me how I hate to discuss programming languages and programming ecosystems with C people. It feels like C/low lvl people often refer to some "standard" that they treat as a bible of programming languages even when nobody mentions C and they treat GCC/CLANG as a perfect reference for compilers. So, while high level devs dont understand low lvl stuff, then I believe low lvl programmers aren't creative at very abstract level, very abstract system modeling cuz they're too used to modeling everything with integers in C. And they're too used to being hurt by C ecosystem that they believe it is "normal". High level people (web developers) spend a lot of time arguing about their buzzwords like microservices, ddd, oop vs fp, etc.

melagonster 3y ago

Is not this the main point of the editor war? programmers built tool for enhancing programming abilities, but they argued with which tool is best one...

PedroBatista 3y ago

I could not disagree more. If it is not not invented here, on the limit is not invented anywhere because nobody knows how to build these things, which we have been on that path more or less. Perhaps you were visualizing that guy on your job who creates his own libraries and obscure systems within the system before telling anyone and that is generally a bad peer. But the Universe is much bigger and diverse than that, people who programmed their own things but do a Google search first and use other software because they know it’s more feature complete, more known, trusted and better future supported are the best to work with. We are all using the products of not invented here syndrome right now, don’t forget that.

ess3 3y ago

Actually reimplementing something that’s already exists is a great way to get an understanding of it precisely so you can deal with the questions you mention. My rule of thumb is that you should always understand at least one level of abstraction down.

goto11 3y ago

> Identifying high-quality libraries and frameworks that meet your project needs This is a really important skill that I find difficult and frustrating. Does anyone have good advice or resources?

deterministic 3y ago
It’s simple really: write libraries and frameworks yourself and use them for real projects. It will give you a great understanding of what makes a quality library/framework and make you a better programmer.
- goto11 3y ago
  Are you perchance working in academia? Your suggestion seems to assume unlimited time and resources to write everything yourself, while the purpose of using third-party components usually is to save time and resources.
  - deterministic 3y ago
    Nope. 30+ years software developer in industry. Think about it as a hobby. Most people have time for hobbies. The rest spend way too much time on social media or watching YouTube. Manage your time better.
    - goto11 3y ago
      > Manage your time better. Thank you for the advice which is no doubt well-intentioned. But don't kid yourself. If you use your spare time for research and development to solve problems at work, then it is not really a hobby, it is just you working overtime for free. If that makes you happy, good for you, but it doesn't change the fact that time and resources are limited. The time is just cheaper in your case.
      - deterministic 3y ago
        I am spending time educating myself. That is super valuable for me and opens up new career opportunities. And yes it might also happen to benefit the job I do currently. But that is not the #1 priority.

hombre_fatal 3y ago

Honestly this just sounds like me when I first encountered coding puzzles like project euler / leetcode and interview challenge practice problems. I made up excuses like this because I simply didn't want to spend time on lower level problems and instead wanted to feel good sticking to my high level application comfort zone. I eventually learned to like solving lower level problems and I'm much better off for it. And these days I can tell people who haven't paid their dues because the only solutions they ever come up with are slow quadratic ones no matter the occasion. No, building your own text editor, compiler, OS, ray tracer is not going to make you a worse engineer nor keep you from learning how to google and evaluate libraries or think critically, nor are those things reserved for an engineer who isn't building those things.

culi 3y ago

> In fact, it might make you worse at software engineering, because it embodies the disastrous "Not Invented Here" doctrine I get your point against NIH, but I don't see how doing such a project on your own in your freetime could possibly make you a worse programmer. If anything, you would likely learn exactly what it takes to go down the NIH path so strictly and be able to make better choices

heresie-dabord 3y ago

> might make you a better programmer, it won't make you a better software engineer This is the distinction between solving "local" problems and solving problems at scale. But I think there is a transitional phase where requirements (gathering, documenting), unit testing, UI design (human or programmatical) are essential. Learning these as nuts-and-bolts in any language is essential to becoming skilled in the craft.

pmelendez 3y ago

There are a few things on this comments that is causing noise in my mind. >might make you a better programmer, it won't make you a better software engineer I think it is a short-sighted definition of SE but in any case, there are not any mention anywhere in the article claiming that this was aimed to software engineers. Also, although I appreciate the concerns of "Not Invented Here" doctrine, we also have the other side of the spectrum, where we don't know how to do a left-pad anymore [1], and introduce dependencies everywhere. The third point is that I don't think that having more knowledge on how things works contributes with the "Not Invented Here" mindset. I would argue is the opposite. When engineers are hungry on learning, they find excuses to make address that appetite at work, but once they know the tradeoffs and amount of effort to make it work, they will think twice before starting anything from scratch. [1] https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code https://qz.com/646467/how-one-programmer-broke-the-internet-...

ryloric 3y ago

Idk why everyone is piling on this comment, they clearly make a distinction between programming and software engineer.

mebassett 3y ago

There is a lot of disagreement towards this idea in the thread, but I really like this project suggestion for someone to make a multi-faceted project that'll help someone "grow up" and it resonates with something I've long felt missing from comp sci education: that it doesn't focus enough on studying and replicated big successful projects. You mostly learn by doing, true, but you learn by copying what the masters did. Software engineering education feels more like taking classes and writing code for exercises, whereas it probably should have more projects of the form "study this complicated-implementation-of-X and use those techniques in your implementation of Y", where x = google search, postgresql, whatevers and y is a complicated project [for one person or a small team] like your own single concurrent user search engine I've held this idea for a really long time, in fact, I've not actually looked at comp sci education recently, so maybe this isn't true anymore, or maybe it never has been

yawboakye 3y ago

one wonders whether you’ve created having completed two of the items on the list, i can confirm that it makes you better, whichever way you’d like to measure it. as a google engineer told me once: operating systems and compilers are where all the good ideas are. study them! only thing that beats that is probably writing them yourself, even if toy versions.

frankie_t 3y ago

It really depends on your goals. Why would someone want to become a better software engineer? To me, the things you listed aren't fun, so I wouldn't do them in my spare time unless I see the benefit in it. If we are talking making more money at a job, I'd say that for most people I worked with and me included there are more important things to concentrate on. Like, soft skills, communicating clearly, understanding politics at work. Myself, I'm okay with learning things you listed from more senior colleagues, on the job, and just having fun in my spare time.

4m1rk 3y ago

You can't be a better software engineer before being a better programmer.