9 ms·
The Architecture of Open Source Applications
- eatonphil 6y agoI've had the idea for a while to start a club to read through open source projects, compile them, extend them, note architectures and design decisions, look at documentation and CI/testing organization, etc. Do something like one project per month with one topic among the above per week. Haven't found the right group of folks to experiment on with this yet. Edit: Send me an email if you'd like and I'll think about hosting this virtually.
- nandhinianand 6y agoMe too.. I've not found the trigger to keep doing it on my own.. So a group might work..
- badrchoubai 6y agoI’m interested in this
- faustocarva 6y agoSure i'm also interested
- Jarwain 6y agoSounds like fun! If you end up putting this together, I'd definitely be interested
- tobylane 6y agoAre you looking for a book club like group to hop around with or some projects eager to have you?
- serial_dev 6y agoThat would be really cool. Maybe live streaming the content?
- MoroCode 6y agoThis sounds really interesting !
- iandinwoodie 6y agoI’m interested; email sent!
- spirobel 6y agome too. I actually did this with the discourse.org codebase. I can make a youtube video about it if someone is interested. I spent over a year digging into it.
- farias0 6y agoI'm definitely interested!
- EGreg 6y agoOoh can you do ours? It’s a fair bit more complex mand general-purpose than Discourse, though. Would be great to see your impressions as you dig into it and maybe record yourself on youtube. Any way I can be of assistance, let me know. https://github.com/Qbix/Platform https://github.com/Qbix/Platform
- thor24 6y agoInterested!
- sali0 6y agoVery interested, would love to be a part of this.
- dzink 6y agoI’d love to join as well. Email sent.
- di1eep 6y agoVery much interested. Email sent.
- jcranmer 6y agoOn a similar note, I've thought about putting together a presentation on debugging and understanding large codebases by fixing a bug on a large project whose source code I've literally never seen before. I figure that by picking a codebase I have 0 experience with, there's less risk of skipping over steps. The biggest hurdle for me has been actually finding a project I'm interested in where I haven't yet gone poking around the source code for some reason or another.
- eatonphil 6y agoYeah, I've also been interested in writing more about orienting yourself in novel codebases. I don't think it's explored very much. It's a pain to write about because you have to find some contrived error (real or not) in a contrived app. I've got a draft of a post but without the examples filled in. Would like to read (and share) what you've got.
- monksy 6y agoIt's not explored very much because most new developers try to push for greenfield development and they're trying to split everything up even when it doesn't need it. Now we're having multiples of codebases to comb through.
- gnufx 6y agoSo true. Also with respect to documentation, which you have to put in a blog rather than contributing it to the project where it can be maintained. It's bizarre to be literally the last port of call for work you've actually done, when people would rather refer to obsolete blogs on things you addressed years ago.
- aahortwwy 6y agoAt a previous job of mine modifying large and unfamiliar codebases was a major aspect of our team's role. I always thought it would be a good way to interview candidates, but never got to try it out.
- 6y ago
- mathieubordere 6y agoemail sent!
- phkahler 6y agoDo it yourself on youtube. Have a patreon page. Go over the code yourself to get your impressions and then interview the main contributors about the project. Done right I think that may go somewhere.
- danbmil99 6y agoI would definitely subscribe
- Cognitron 6y agoSame
- dwoot 6y agosamesies
- spicyramen 6y ago+1
- holografix 6y agoI would pay for this. Specially if it’s done for small to medium sized Go projects.
- deleted 6y ago[deleted]
- jooz 6y agoTo understand large codebases is my biggest weakness. I would be definitively interested.
- bartvk 6y agoI don't find them interesting just because they're large. I've worked on some large projects and sometimes, it was just a lot of code because there were loads of business exceptions to this and that.
- vnxli 6y agoEmail sent. Thanks! Sounds interesting
- hinkley 6y agoAt this point, the field of programming is broad enough that we really should have a moral equivalent of Comparative Literature and we don't.
- ryfia 6y agoEmail sent as well
- rgrau 6y agoWith a friend we started having those, and we've been thinking about streaming them, but they always end up being quite chaotic. It'd be very interesting to figure out how to make them fluid enough and yet easy to follow for more watchers. We both do some pre reading of the topic, and "pair program" through it, changing some bits and running the code to see if the current mental model still holds. The person who drives has to be very verbose explaining what he's doing ("Let's go to this function definition, ok, we see this struct here, let's leave a mark for later and visit that struct"), because otherwise, the one watching is like "wtf, are we looking at now". We're currently going over emacs' native compiler branch and it's been lots of fun so far. The conversational interaction of Lisps makes those explorations very lively. I'd be interested in joining, of course :)
- specialist 6y agoPlease submit a "Show HN" once this launches. I'd joyfully watch any recordings. FWIW: I started a study group in the late 90s, which continues today. Our original topic (theme) was Design Patterns. Took three attempts to become self-sustaining. Mostly for an in-group to congeal. We had to experiment with the format, work out the details. Stuff like meeting frequency, best way to do the food, delegating tasks. After each book, the group gets together to choose the next book. Over the last 20+ years, we tackled most vogue programming languages, Prolog, SICP, concurrency, ML & big data, k8s, methodologies, algos. (One notable omission are the consensus algos.) So here's my unsolicited advice: #1 As soon as you can, delegate the discussion leader task, then step back and focus on coordination (hosting). Once you figure out a mostly working format. After showing everyone how it's done, hand it off. Ideally, do round robin with the participants. After a while it becomes clear who's good at discussions. First reason is coordinator is a full time job itself. Preparing to lead a discussion is a lot of work and a different headspace. Reading the source material, doing background research (reading other reviews of same), distilling 3-5 essential open-ended questions about the chapter, etc. Second reason is emotional engagement, investment. Everyone has to pull their weight. Just like with a church, there's no commitment without sacrifice. You're hosting a discussion group, not a seminar. #2 Experiment, have a plan, make decisions democratically, do post mortems after each series. Ask each participant to write what worked, what didn't, what they'd change (aka The Good, The Bad, and The Ugly). Send all write-ups to group pre meeting. Then discuss. As coordinator, my job was to facilitate, carry out the group's will, keep everything on track. (Well, mostly the job was to show up.) There have now been a handful of coordinators; so apparently our culture and format were successful. So if I was going to tackle AOSP today with a new crew, I'd draft a plan, invite everyone to also propose plans, present plans to the group, have a list of open questions to be decided, keep a list of new questions raised, keep minutes, send out a recap. I'd probably host every thing on git, to ease collaboration. My opening position would be to suggest a first "season" 4-6 months long, meet alternate weeks, use a survey to divine which chapters to cover (based on group interest), another survey to divine meeting schedule. Ok. Enough of my brain dump. Again, whatever you end up doing, please do a Show HN. What you're doing is very exciting. Happy hunting.
- eatonphil 6y ago
- AleZ88 6y agoGreat idea. Sent email
- astatine 6y agoI was inspired by the AOSA many years ago and I had studied (and held out) as an example the Apache architecture (about 18-20 years ago). It is indeed a pity that so many younger software engineers really have no idea of how heavily used real-world systems get built. Nor do they understand all the various tradeoffs and corner cases that need to get handled to make products work well in real life. Building Apache from source (including a few key addon modules) and configuring it to work for a small set of tests was step 1 at my organization then. Those successful at this would then get into the source itself. Fun times!
- fizwhiz 6y ago> It is indeed a pity that so many younger software engineers really have no idea of how heavily used real-world systems get built. Nor do they understand all the various tradeoffs and corner cases that need to get handled to make products work well in real life. Isn't that a hallmark of a junior engineer :) ? If you're trying to achieve deeper technical depth (which takes time), it helps to be a part of an organization that rewards said technical depth.
- mesaframe 6y ago> it helps to be a part of an organization that rewards said technical depth. I agree. My previous job wanted to employees to finish work ASAP. There were no incentives for quality. Performance was judged on the basis of how fast one can push things out irrespective of quality. As a result there was lack of engineering.
- pault 6y agoNot only that, but those types of organizations are also the ones that are likely to tell experienced engineers that they are "overqualified" for the job because they want more juniors that won't push back against harmful practices.
- Aeolun 6y agoOnly for a while. After a number of failed ‘initiatives’ someone at the top is bound to start asking questions.
- chubot 6y agoFWIW the Oil project was partly (negatively) inspired by the very helpful description of bash's architecture: https://aosabook.org/en/bash.html https://aosabook.org/en/bash.html The maintainer expresses some regret about the parser architecture. I referenced this in several 2016-2017 blog posts: OSH Parses Shell Scripts Up Front in a Single Pass. Other Shells Don't - http://www.oilshell.org/blog/2016/10/13.html http://www.oilshell.org/blog/2016/10/13.html OSH Can Be Parsed With Two Tokens of Lookahead - http://www.oilshell.org/blog/2016/11/17.html http://www.oilshell.org/blog/2016/11/17.html The Thinner Waist of the Interpreter - http://www.oilshell.org/blog/2017/01/26.html http://www.oilshell.org/blog/2017/01/26.html Word Evaluation - http://www.oilshell.org/blog/2017/03/09.html http://www.oilshell.org/blog/2017/03/09.html ---- This principled architecture ended up paying off in a couple unexpected ways, which I wrote about earlier this year: Oil Uses Its Parser For History And Completion - http://www.oilshell.org/blog/2020/01/history-and-completion.html http://www.oilshell.org/blog/2020/01/history-and-completion.... Oil's Parser Doesn't Worry About Aliases and Prompts - http://www.oilshell.org/blog/2020/01/alias-and-prompt.html http://www.oilshell.org/blog/2020/01/alias-and-prompt.html Summary: in addition to the regrets expressed by the maintainer in the article, bash actually has multiple ad hoc, incorrect parsers for its own language! They are incorrect to the point that a separate project (bash-completion) has code to paper over the errors. Oil uses the same parser for all use cases. Moreover, its parser isn't littered with orthogonal concerns like the interactive prompt and expanding aliases. ---- However I still need help finishing the project -- see what I've cut here: http://www.oilshell.org/blog/2020/08/risks.html http://www.oilshell.org/blog/2020/08/risks.html And feel free to contact me (address on home page and in profile)
- dllthomas 6y agoI've been working through the first volume in a reading group with a few friends. Some chapters are definitely better than others, and it often requires awareness that it captures a snapshot of thinking from ~10 years ago, but it's been a good experience.
- p1nkpineapple 6y agoCould you please elaborate on the chapters that you feel are the best to study? Cheers!
- dllthomas 6y agoSo far we've read the chapters on Asterisk, Audacity, Bash, BerkeleyDB, CMake, Continuous Integration, Eclipse, and Graphite. Of these, I liked the chapters on Bash, BerkeleyDB, and Graphite better than those on Asterisk, Audacity, CMake, and CI. I had trouble getting through the chapter on Eclipse, although there were some interesting bits. The chapter on Bash suffered a little, in my personal experience, because about 70% of it was a rehash (with a slightly, but not sufficiently, different perspective) of information present in the Bash man page, which I had previously read in a reading group with some of these same people. But really, the chapter that's best to study is probably the one you're currently most interested in, even if it's "not as good" as another considered in the abstract. Just remember not to assume that the entire book is of the same quality, whatever your experience.
- dang 6y agoIf curious see also 2019 https://news.ycombinator.com/item?id=20731206 https://news.ycombinator.com/item?id=20731206 2015 https://news.ycombinator.com/item?id=9418788 https://news.ycombinator.com/item?id=9418788 2013 https://news.ycombinator.com/item?id=6404929 https://news.ycombinator.com/item?id=6404929 2013 https://news.ycombinator.com/item?id=6613748 https://news.ycombinator.com/item?id=6613748 2011 https://news.ycombinator.com/item?id=2904425 https://news.ycombinator.com/item?id=2904425 2011 https://news.ycombinator.com/item?id=2598643 https://news.ycombinator.com/item?id=2598643 2011 (a bit) https://news.ycombinator.com/item?id=2578071 https://news.ycombinator.com/item?id=2578071 There have been many other submissions (https://news.ycombinator.com/from?site=aosabook.org https://news.ycombinator.com/from?site=aosabook.org), but those seem to be the nontrivial threads about the book itself.
- amelius 6y agoI'd like to see Kicad, Freecad and Inkscape covered.
- tepitoperrito 6y agoCan't speak about the other projects listed but FreeCAD has a great landing page on their wiki. https://wiki.freecadweb.org/Developer_hub https://wiki.freecadweb.org/Developer_hub (I especially liked the forum post from the contributor who detailed the geometric constraint solver - very neat)
- unixhero 6y agoGreat book! I think I will read the whole thing.
- rektide 6y agoTelepathy is one of the coolest most useful most modular pieces of software i've ever used, a modular pluggable framework for chat, & I am always so glad to have a chapter of this book talking about that all.
- site-packages1 6y agoVery good book, but I would love a version working with modern tooling and software products. I remember going through these maybe 5-7 years ago and thinking at the time that they were using some really old paradigms and software. Would love to see some covering software with modern languages as well, like Rust.
- spicyramen 6y agoThis should be part of any serious computer science program subject.
- pattato 6y agoIs all the content on that website representative of what is in the book when you buy it?
- aabbcc1241 6y agoWant to double star this resource. It covers many interesting (and advanced?) topics like distributed system, building your own database, meta programming, and template engine.
- pkphilip 6y agoThis is a brilliant idea! It would be great if they start with something everyone uses such as editors - example: Vi, Emacs, Code, browsers such as Mozilla etc.
- legulere 6y agoI think the most important you can take away from it, is that a good architecture takes the problem into account that the project tries to solve. Too often I hear calls for some kind of architectural holy grail, that solves all problems.
- gnufx 6y agoIt's a pity it's "open source" and not free software including Emacs, in particular, which has been rather successful itself and as a model long-term. I wonder how some of these architectures have really helped with maintenance. For instance, I'm familiar with Open MPI and its rather sorry maintenance history with continual breakage and at least partially unnecessary lack of compatibility. Then it still seems the best bet for general purpose MPI...