12 ms·
Ask HN: How to get good at designing big and complex applications?
Hi HN! I am a grad student with a focus on CS, specifically Software Engineering. I like to explore different technologies that are used to build a product and am fascinated by the way people design such big complex systems. For instance, I am taking a class on Operating Systems right now which also requires me to develop my own kernel. I am finding it very exciting, challenging and fun. I would really like to know what all I can do to get better at designing such big complex systems. I try to read open source but it always feels like a challenge. I don't know if I have the aptitude for this but studying CS and the thought processes of people who have spent so much effort in this field gives me a lot of joy. So, if you have any advice, resources, books or tutorials that you think might help me, please do share. I would be really obliged. Thanks!
- kaazhan 7y agoHi, There's no single way. In fact, beeing a good software architect is all about having many ways in solving a problem, knowing wich solution is the most adapted depending on the context. All is about culture and cleverness To improve, you can : - Follow/read the well know architects. For example, I read this blog https://martinfowler.com/ https://martinfowler.com/ and watch most of the conferencies he gives. - Learn the design patterns and the architectural ones. - Read a lot of code, like github repos. It does not care if you're reading repos on a subject close to the one you're interested to. You need to see a lot of architectures to see some clever solutions some peoples have used to solve some specific problems. I don't think there's any shortcut. - get interested in meta development. Like "why does the languages work the way they work ?", read about how code is produced in real conditions (what are the processes involved), etc.
- quickthrower2 7y agoI’d also add go to excellent meetups with high quality talks. They attract good people, get talking and you’ll have 100 things to google when you get home!
- kaazhan 7y agoYes, getting involved not only behind your screen is really important. They attract good peoples that give you tons of subjects to study, but most important, it makes you feel like you're part of a community and you're not doing all of this for nothing, or in a wrong way. For that part, I think it's true whatever is the field in which you want to improve
- rochak 7y agoThis is something I can do! Thanks, I will try to go to some meetups.
- rochak 7y agoWhat to do when those talks are too complex for my understanding?
- quickthrower2 7y agoThey are for me too. Ask questions after the talk. Get the gist if you can even if you don’t grok all the details. Speakers often go too fast through stuff so I don’t take it personal if I don’t understand it all or any of it.
- rochak 7y agoThanks! For some reason I find it difficult to just approach people and talk about stuff with them.
- kaazhan 7y agoYou're still a studient. It is not possible for you to have a good knowledge of software architecture because it is a really complex domain. If you show interest in it, read about it, go to conferences and ask questions, don't worry. Peoples will act cool with you. Anyway, the only way to progress in dev/software archi/devops/etc is to put your ego aside and accept the fact a lot of peoples knows whay more than you. If you go speak with them and they treat you like garbage, this is their fault, not yours. If you're really shy/impressed, maybe you can go with some friends and speak about it together. It will help you feel like you understood some things. Architecture is not filled only with youg peoples, and if you're a group of 20+ y/o speaking together, eventually the older ones will come and speak to you. then you can ask them what to read/watch to have deeper understanding
- matijash 7y agoCongrats on your will and enthusiasm, I believe this is already a great first step! Besides Martin Fowler which was already recommended, I would also recommend the books by Robert C. Martin (Uncle Bob), and his "Clean" series - there are books "Clean Code", "Clean Architecture" - all of them are really good and useful and will give you a solid foundation to build on. Other than that, the best thing you can do is to get practice & ideally surround yourself with more experienced developers. Either get an internship in a company building something more complex, or participate in e.g. Google Summer of Code. Doing fun side projects is also a great way - you learn a lot and also have something to show & write about - win-win!
- rochak 7y agoI have got an intern at AWS so I hope I get to see some good design and meet intelligent people there.
- gentleman11 7y agoI have been looking at clean architecture and trying to apply it to a Unity game. My impression is that industry wisdom favours a lot less OOP and a lot less abstraction in their architectures (eg, Jonathan blows talks, hundreds of posts on Reddit), mostly for performance and velocity reasons. Do these techniques apply to games, or are they the wrong tool for the jobs?
- rochak 7y agoHave you checked [0]? [0] https://gameprogrammingpatterns.com/ https://gameprogrammingpatterns.com/
- rasikjain 7y agoThis comes with experience. There is no single answer to get good at the designing complex systems. The more you work in different applications, the more you discuss with your peers in your organization and other meetup groups, explore different design patterns (Martin Fowler), Follow well known Architects or Company engineering blog [0] posts, Try doing proof of concepts etc. Eventually as you progress with experience, I am sure you will get that knowledge. I used to follow the below link [1] to understand how the systems work at different companies globally (e.g Facebook, Google, Netflix, PlentyofFish etc). This has very interesting content. [0] https://github.com/kilimchoi/engineering-blogs https://github.com/kilimchoi/engineering-blogs [1] http://highscalability.com/all-time-favorites/ http://highscalability.com/all-time-favorites/
- rochak 7y agoThanks for the advice and the links! Also, I came across a book called Designing Data Intensive Applications. It was very highly rated on Amazon and on HN as well. Do you think it would help me? It explained a lot of concepts but after reading for a while, I found myself not following it. So, I decided to tackle each topic from scratch.
- aynyc 7y agoDo people find Martin Fowler and his peers' work to be good in real world? I'm a system/application architect who spent years coding. I used to read books by Martin Fowler and others religiously. But now, I realize most of the books and talks are really just fluff pieces because the examples worked in perfect condition, where as my experience is from the battle field. As far as getting good at designing big and complex application, I'll give what I know. 1. Everything complex came from a small system. So make sure your small application works first. Twitter/Google/Facebook didn't get to what they have now on day 1. They failed a lot to get here. 2. Designing monolithic system is a lot easier than micro services. 3. Read a lot of codes and how different component interact with each other. For Python, projects such as Scrapy, Flask and Django are good example to start.
- auspex 7y agoI find the concepts he makes you think about are helpful. Think through the big problems and make sure you keep them in mind as your are coding the subsystems. Make sure any code you write could be used by another system in case you need to break it apart later. Make things pluggable though interfaces. No global references.
- BlargMcLarg 7y agoHonestly, I found most books are written more as a basis for people who need something to hold onto and can't do it on their own (yet), and for people to understand the ridiculous enterprise systems. Most of the time, I see good developers take longer to come up with various structures and deciding which one to go for than the time they'd spend fixing these things afterwards. On the other hand, inexperienced devs will trip up no matter what. That's why they are inexperienced, give them a chance and some time. Realistically, good systems are built by good devs who understand you need to continuously maintain and evolve the system according to your needs, and keep an eye on relatively simple things like method structures while doing so. Its easier to refactor when all your methods name themselves, are bite-sized in what they do and you provide a sandbox for other people to work in, or a blackbox for them to work with. And clear errors. Don't be that guy who hides the stack trace and doesn't keep track of the state of the app. Please don't.
- 7y ago
- bg4 7y ago"A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system."
- gavinray 7y agoI just made the connection between your statement "A complex system that works is invariably found to have evolved from a simple system that worked." and the common understanding among experienced developers that proficiency in software and the primary goal when designing and building a system is perpetually fighting complexity. You design and write the cleanest, most simplistic & adequate solutions for problems as they arise and ruthlessly battle introducing complexity as features and scope grow. Inevitably, after many years you will wind up with a system which by sheer size is complex. But only reason it works is because of the never-ending battle in the fight against complexity.
- bg4 7y agoYes, have to relentlessly drive down complexity. Starting with simple, pure functions that work and composing them together to build up more complex functionality goes a long way.
- YayamiOmate 7y agoDepending on the context this is either simply false or a truism. And thus imho offers no real informational value. You can't design complex system without some buiding blocks which basically definition of 'complex', but you can design a complex system before it's deployed and it can succeed. Itreatively designed system can hit a scalabilty wall or plateu where adding features is slower that a redesign. An evolving system design is a subject to a local minima of scalability and it needs careful apporach to maintain it. This sounds kinda like a rephrased "there's no problem that can't be solved by adding another abstraction layer".
- delian66 7y ago
- segmondy 7y agoStudy & Experience, that's it. Also understand the difference between these programmer application developer software engineer computer scientists there are very different roles, it's okay to be one and really good at one. BTW, you mentioned "design" so I'm going to hinge on that, design not the actual implementation. In that case study existing systems. 90% reading can teach you design, on actual implementation, more like 90% doing.
- rochak 7y agoWhat if I’m not inclined to just one of these? I like learning things and figuring out why which is why I see myself fitting into every category.
- segmondy 7y agoNothing wrong with that, if you are a generalist you will find yourself intersecting those. Else you will find yourself focused in one. There are very few organizations in the world where you get to do all of those, except for a startup in the initial stage or by you pursing your own side projects. Something like game programming might intersect all (programming - using GPU, app development - the game engine, software engineering - building a scalable game server, computer science - utilizing appropriate algorithms and data structures that most people will tell you they never use in the real world) if you want to do the programming thing - learn low level programming, buy an arduino etc and learn to program it. app development - build websites, mobile apps, desktop apps. software engineering - contribute to a large scale app (kubernetes, postgres, etc) computer scientist - study latest CS papers, replicate result, write your own paper and come up with your own ideas.
- rochak 7y agoAfter you define what these terms mean, I think I am more inclined towards programming and software engineering. Thanks!
- Scarblac 7y agoThe first three of these are synonyms. I've had all three job titles, over six different companies for roughly the same work.
- avinassh 7y agoCheck the books of Architecture of Open Source Applications - http://aosabook.org/en/index.html http://aosabook.org/en/index.html
- rochak 7y agoThis is a good find! Thanks!
- tilolebo 7y agoThis book (Designing Data Intensive Applications) was an interesting read: https://dataintensive.net/ https://dataintensive.net/
- rochak 7y agoThanks! I knew about this book before but couldn’t go through it a lot because it started to get very tough. So, I decided to read each and every concept from the basics before I jumped into that book again.
- subhobroto 7y agoThe early decision you will have to make is whether you want to be a theoretical specialist who will mostly make money consulting working at a consulting firm or university or whether you will be solving actual business problems down to the metal. Each has a completely different trajectory at this point in your life. A great example is to read the great debate between Andrew Tanenbaum and Linus Torvalds over kernel design as its closely related to what you're currently studying. Linus would have clearly failed Tanenbaum's OS class. Yet I use Linux every day, as does millions of others, while Tanenbaum's MINIX is on a CDROM that came with the book that I bought for my OS class. This is not to say Linux is still a monolith, but that would be getting in way too much details.
- theandrewbailey 7y ago> Yet I use Linux every day, as does millions of others, while Tanenbaum's MINIX is on a CDROM that came with the book that I bought for my OS class. You sure about that? Do you use an Intel CPU? https://www.zdnet.com/article/minix-intels-hidden-in-chip-operating-system/ https://www.zdnet.com/article/minix-intels-hidden-in-chip-op...
- subhobroto 7y agoAndrew, I didnt know that. Thank you. Now, let me ask you this. If someone was to ask you either of: 1. "Between MINIX and Linux, which is a great example to study as an OS that's deployed on billions of machines worldwide and used by millions of engineers?" 2. "Between MINIX and Linux, learning administering which, offers me the best job prospects in the general tech industry?" What would be your answer in each case?
- delian66 7y agoMinix was not voluntarily chosen by Intel's customers...
- ooooak 7y ago> How to get good at designing big and complex applications? Write small and simple programs.