4 ms·
Large codebases are inevitable. Infrastructure stands on infrastructure. We stand on the shoulders of giants (or at least geeks with good posture). One of the
by ChrisMarshallNY 3y ago
Large codebases are inevitable. Infrastructure stands on infrastructure.
We stand on the shoulders of giants (or at least geeks with good posture).
One of the things that folks don't like to think about, is that people, these days, aren't really any smarter than they were, five thousand years ago (in fact, Jared Diamond posited that we're actually a bit stupider).
We just have a baseline social infrastructure that allows our "starting point" to be that much more advanced. We are the equivalent of privileged offspring, inheriting family wealth. If we are trained to manage and utilize that wealth, it grows. If we aren't, it will likely get squandered.
I started off, writing my own operating systems in assembly. I even have the manual, for my very first engineering project, in 1987 (including the OS code)[0]. It worked, but it was very primitive. I can't imagine writing an OS in assembler, these days. I'm glad someone else did, in C (or whatever).
So my "starting point," these days, is having a high-quality, parallel operating system, with a developer SDK (again, someone else that knows what they are doing, did the hard work for me), a compiler and linker (thanks, Chris Lattner, et al), etc.
For my own projects, I use a lot of dependencies. The difference is, is that I write most of my own dependencies. These are standalone modules, with each one having its own project lifecycle, documentation, testing, etc. Even if no one else will use them, I still publish them on GitHub, as supported open-source projects, because that means that I don't need to worry about them, when I use them in my own aggregate projects.
Basically, "divide and conquer." Big things are composed of small things. It's important for me to do a good job on the small things, so the big things turn out better.
But that's just me, and the way I work. YMMV.
[0] https://littlegreenviper.com/wp-content/uploads/2022/07/TF30194-Manual-1987.pdf https://littlegreenviper.com/wp-content/uploads/2022/07/TF30... (Downloads a PDF)
- CyberDildonics 3y agoBasically, "divide and conquer." Big things are composed of small things. I don't think this is the controversial part or the difficult conclusion, the challenge is how exactly to do that.
- ChrisMarshallNY 3y agoWhich is what I mentioned. I think the “controversial” part, is probably the care taken, in developing the small things. The dependencies on which I rely (operating systems, compilers, SDKs, etc.) are all very well-designed, solid, and supported. I’m not a huge fan of surfing the bleeding edge. I like to use mature tools (usually. I’ve been writing Swift, since the day it was announced). Like I said, I spend a great deal of time, on each of my modules. Often, the testing code far outweighs the product code. Documentation is also a big deal. I write code and documentation for an audience of one (Yours Truly), but I publish it for everyone. It’s not my concern, whether or not the projects get any GitHub stars. In fact, I prefer it, if they don’t. Here’s an example. I’m currently writing an SDK to provide cleaned data for ML, so I can play around with ML, but it will be quite some time, before I get to that point. I’m still working on making the SDK rock-solid. I have spent the last couple of days, just working on the parser. I still need to write the query/importer. Yesterday, was all about unit tests for the parser. The next part, will require a test harness[0] (a SwiftUI app). Unit tests won’t be useful for that. It will also be my platform for the ML application. Also, code coverage for the parser tests is only about 85%. I plan to use the test harness to test the remaining 15%. Many folks would simply rush through this part, but they would end up shipping (or throwing out) the sloppy code they wrote as a lash-up. Even if I decide that I don’t have a current application for ML, I’ll still have a top-Quality SDK, to use as a component for future work. The app I just released, has a backend server, that I originally designed as a general-purpose application server. It took me seven months to write, as it was written in layers, and each layer was rigorously tested, before moving to the next. The purpose of that project, was as a teaching exercise. I didn’t actually have an application in mind, when I wrote it. The server is a lot more complex than it needs to be, because it was designed as a general-purpose server, and the current implementation uses a fraction of its capacity. But the server was available, when I started work on the app, and I already knew it was extremely high-Quality, so using it wasn’t a big deal. It probably saved me a couple of months, and I really have a difficult time, trusting backend dependencies. [0] https://littlegreenviper.com/miscellany/testing-harness-vs-unit/ https://littlegreenviper.com/miscellany/testing-harness-vs-u...
- CyberDildonics 3y ago
- breather 3y ago> fact, Jared Diamond posited that we're actually a bit stupider Please don't enable this man when there are literally millions of trained historians you can talk to.
- lproven 3y agoHe isn't a historian, though. It does seem plausible. We are descended from Cro-magnon man; Neanderthals had bigger brains than us. https://www.archaeology.org/issues/103-1309/trenches/1138-neanderthals-brains-devoted-to-visual-processing https://www.archaeology.org/issues/103-1309/trenches/1138-ne... And we have smaller brains than our Cro-magnon ancestors. https://phys.org/news/2011-06-farming-blame-size-brains.html https://phys.org/news/2011-06-farming-blame-size-brains.html
- breather 3y ago> He isn't a historian, though. Doesn't stop him from pushing his own grand narratives of history, apparently. The claim isn't particularly interesting given the nebulous and culture-bound definition of intelligence, but he's not a person I would tap into for a sense of expertise backed by consensus.
- fm2606 3y agoIt is my understanding this is how Apple does their development. Everything is home grown and not a hodge podge of dependencies from outside the company. I do like Mac products but just can't get passed their walled garden. However, to have the quality they do I guess it kind of makes sense to have the garden. I personally like the idea of few-to-none outside dependencies, solid testing and solid, up-to-date documentation. I need to work on being more disciplined and motivated to do it.
- ChrisMarshallNY 3y agoI think that Facebook is similar, but more of their stuff comes from acquisitions, so the conglomerate is quite a bit more heterogenous, than a company like Apple. I think that most corporations don't like to cede control. I have been told that one of the big reasons for Boeing's troubles, is that they have pushed a lot of important functionality to external contractors.
- jiggawatts 3y ago> Large codebases are inevitable. I vehemently disagree. New codebases might be built on top of large stacks of framework code and infrastructure code, but source repositories ought to actually be tighter and more business-focused than they used to be. But they're often bloated monstrosities, filled with resume-padding architecture astronaut bullshit, or contorted to fit the latest hip framework's requirements. Apologies ahead for this long-winded rant, but this exact issue has been boiling my blood for weeks now because I've been wading through the neck-deep mess that a customer has made, such as writing an astonishing three megabytes of source (typed by hand, not generated!) for a single web form. One! Especially in the era of modern web frameworks, we can avoid most of the "ceremony" of older systems like C or C++ where just making a call to a database was pages of fiddly boilerplate. Instead of embracing this, I see developers reintroducing unnecessary code because they must have gotten so used to the verbosity that they felt exposed without it. Let me give a concrete example from a codebase I've just spent a week reviewing. I've read through hundreds of pages of code, found dozens of critical resource leaks and security issues, and I'm yet to encounter any actual business logic. The first mistake that happened was that someone decided to use Angular. Why? It's hip! It's cool! It's what all the kids are doing! Except now, you have to have an API instead of just web pages. That API will need to serialize things to JSON, so you need to data transfer objects and Newtonsoft code. Suddenly, you need to deal with JWT, and make sure the untrusted client doesn't receive internal (secret) fields from the database. The client is no longer strongly typed or closely tied to the C# code, so you need a full megabyte of test code that you never would have needed before to validate all the Typescript. This code does nothing of business value. It's a form. You fill it out, and hit submit. There's no "interactivity", there's no single-page-app features used, it's just... waste. Then, then, on top of that, someone that really likes "grand cathedral" style code architecture decided that this whole thing would be more secure if there was a "middle tier". You know, an API. No, not the Angular API facing the Internet! A back-end API. For no reason. For an app written by two developers that has its own, dedicated database, and will never interact with any other app, tool, or system. So now you have to write server-to-server API call code including the DTOs and related definitions, the server code, the client code, the service authentication code, the error handling, the distributed tracing, the HTTP header forwarding (securely!), and on, and on. All typed out by hand, because: "REST is easy!" No wait, there's more: The original architect also wanted to make the system "database agnostic", so the mid-tier repeats every abstraction. It has interfaces, types, etc... so that you can "swap in" another database. This never happened, and will never happen. This is purely wasted effort. Then someone put API Management in front of this. Why? Because the word "API" was used. Remember: Angular uses JSON APIs, not HTML, so it's a "thing" that "needs" management. Then that API management -- a cloud SaaS service -- has a reverse proxy in front of it operating in API management mode for "security". Meanwhile all these layers are riddled with URL injection, SQL injection, and similar basic security errors precisely because it's too hard to add a new field to the form, so developers get lazy and simply add a generic "ExecuteQuery(string sql)" mid-tier method and move on with their lives. This is why we have "too much code". Because architects can't justify their positions if there's only two icons with a single arrow between them in their designs. Because "hip new thing" is a shiny we all love to chase. Because of magical thinking that assumes that more abstractions means the code is more good. We have nobody to blame for this but ourselves.