8 ms·

Why not have Lego like Software?

23 points by Mad_Fury 8y ago

Hi Folks, Really enjoy the HN great community with so much knowledge and enthusiasm, been lurking for a long time just reading the stuff you folks share and discuss, i could go on and on :) but need to keep it short... Consider I am simply asking a question and no implying anything, rather just wondering... Would it be possible or rather better if we built software in a Lego like way, Particularly i was wondering about Linux and some other tools (audio players, etc, etc) I understand that all the distros do things in their own way, but at the same time share a common Core (correct me if i am wrong) Kernel, then different groups build things around it. Why not unify those groups and make them work on the Shared Components and Individual - ill try to explain... Imagine Having a Single distro which once you install the Core on your Computer asks you Which group you belong? -Musician -Scientist -Education -Artist - Developer etc. Based on that it installs the UI + Tools for that domain, Where also if needed be Switching from a Domain to Another Domain just removes all the packages and then asks again what you wish to add Just like Plugins for IDEs Wouldnt it make the Software more Robust and also everyone gets to work and build his own vision + make the shared core much more reliable, I have been thinking about this and wondering about it, also been reading and searching but havent found the answer i was looking for really, I might have also skipped a few details on what i was going for, but i guess you who are versed in this topic would Elaborate more on it and give your comments. Looking forward to the answers :) (if the title is wrong suggest a change, be glad to update to attract more thoughts and discussion, Reading suggestions welcome, Books, podcasts anything...) Thanks

exikyut 8y ago
Quote-replying is probably a good idea (For once regex find and replace in my editor worked!) > I like the part in Photoshop, Where you could say for what you wish to use it and it would rearrange the menus or add and remove certain options which are needed for that Use case, this and some other cases influenced my question, That's nice UX design, I agree. It certainly increases the Q&A workload though. > Do you think it is a waste of resources having so many tools which do so many things the same way + a few unique options, Most definitely, but we're stuck with no alternatives at this point. > I like the ShareX tool (screen recording) which asks me to install the extra feature if i wish to use Screen recording, (goes back to the previous comenter who mentioned it is basically packages) This is a wise approach. The problem is that it requires additional testing load than if everything were installed by default. > I hope i am making myself clear in the questions and comments :D You're fine. > not a native eng speaker But easily understandable :) > and also didnt really prepare the question well You've held together very well. > it just came to me in a rush as it was stuck in my head for a long time, sorry about that, It's fine. Really. > Sometimes i look at old software and some new one (sublime text) where devs worked hard for our resources where as today, (luckily i have 16 GB) there seems to be a lack for performance or the will to design efficient systems (due to abundance of resources in PCs?) I do this every day running Chrome on a 12-year old single-core 32-bit laptop with 2GB RAM :D everything's constantly swapping like crazy. Sometimes a single tab will take 10 seconds to open. New URLs routinely take half a minute or more for the webpage to display. It's fun, and a lesson in... something, I'm not sure what, but I'm definitely learning it inside and out. I think it's all about the vague threshold that management can't get away with shipping a product because it's so horrible that users will simply outright reject it. Software bloats out to the point where it sits just above that unusability threshold, then it hovers there. I think this is the rule that governs everything. Enterprise systems are at the worst end of this; they're frequently so horribly slow they're unbearable. This is because they are used by a captive audience that cannot switch away even if they want to. Pretty much everything else "settles" after a while so it sits just above this threshold. What with the progression of tech, this means that if you install Word 97 in a Win95/98 VM (or maybe it'll even work natively :D), you'll notice that it only uses 8MB RAM, and then when you minimize the window it suddenly collapses down to 4MB! Or, for another example, the cross-platform old KDX filesharing client uses all of 8MB RAM on Linux. Never seen another app like it. I think the reason this is depressing is because when we run out of RAM we get sad, and then we think of how amazing things could be... but in real life it doesn't work that way, when we build applications we take advantage of the resources that are available. > What did you mean isolating users into groups? I wouldnt want that :D, but i mentioned it as a layer on top of Core things, example > All users want video and web tech - but then some are more specific - these specifics would be the packages they get bundled on top of core features, and the Collective groups of devs work on those OK, I meant this bit in your post: > Imagine Having a Single distro which once you install the Core on your Computer asks you Which group you belong? > -Musician -Scientist -Education -Artist - Developer etc. These are very vague titles, and their meaning is not very unambiguously defined. I could interpret them in many different ways. This is where the leaky abstractions start. Using your groupings: - If I'm a game developer, I might be a musician, artist and developer. - If I'm a teacher, I might pick education and scientist. You said "which group [do] you belong [to]?", with group being singular not plural. This is where I was getting isolation from. Okay, sure, so you make it so people can be in multiple groups. But then you loose the optimal efficiency you're chasing; if I'm a game developer I might want specific art tools installed, and installing _all_ the art tools might be a waste. So, okay, say you break the categories down, and add "game developer"... but then the 500 different kinds of game developers will all have something to say, and this is where things get really messy. :) To rephrase my earlier comment, providing the complexity and not isolating it has worked out to be the best practical solution. In order for grouping to be successful, it would need to be better, and provide more flexibility than simply throwing everything in everyone's face. So the grouping mechanism needs to hold up to the kinds of edge cases I've described. > Linux drivers and installing certain software is sometimes a real pain and can vary from distro to distro , if the linux is to succeed shouldnt we collectively fight to avoid this barrier Consider that 100 and probably 1000 people before you have tried to figure this out. As I said, I gave the same considerations 16 years :) so surely others have thought long about this as well. Ultimately the solution there is to simply learn the various systems. > Forgive me folks its my first time doing this i think i jump here and there ;D perhaps even confusing you sometimes... Nope, it's fine, I can follow along. I have a slightly tangential thought to add at this point. The reason I personally went down the specific paths I did with trying to understand the way things were and the rationale behind everything, was as a coping/reaction mechanism for my ADHD. I had a hard time trying to grapple with the seemingly infinite complexity I was being presented with (with Linux, open source, software development, etc), and I tried various techniques to try and make sense of everything in spite of not being able to grasp the bigger picture. Naturally, this noble endeavor was itself also compromised due to the ADHD. I couldn't even workaround my ADHD effectively, because of my ADHD. Sigh. Turtles all the way down. I've found that alternative therapies can be unbelievably effective for mental health, so I've made some good strides with comprehension in recent years. You mention jumping around a bit, that you decided to talk about this in a rush, etc. I used to be like that :) (let me see if I'm right: you posted this in a rush before you overthought about it or got bored?) so I'll mention something I realized that was specifically helpful from an ADHD standpoint. First of all, when I started poking Linux I wanted to find "the one true Linux, the most authentic experience". It took me years to understand why I even wanted this in the first place and why it seemed so necessary. It had to do with how I learned: I couldn't just throw a bunch of information in my head and have it arrange itself; the attention span necessary for that to function wasn't available. I needed to handhold the organizational process a bit more, and to do that I needed to explicitly find and tag foundational aspects of what I was learning about so I could keep those in mind as I stumbled on new information and figured out what to associate it with. Problem was, I couldn't figure out what to classify as foundational. Where was "the root"? Here's an answer that took me 10 years to realize: Linux is a relativist system, not an absolute system. Or, in more contemporary parlance (as noted by another comment in this thread), it's bottom-up, not top-down. So everything is relative. There are no roots. The best way to properly understand "Linux" is to simply just dive in somewhere but focus on absorbing as much history as you can; each historical bit of info will help you make more sense of things. > I will update the email Thanks Cool! FWIW, I must say upfront that I'm TERRIBLE at frequent back and forth communication. I tend to use email as a once-every-6-months ping mechanism. If I say hi any more frequently than that, miracles are happening :) So, don't be surprised if I take ages to reply. I burn out easily (still working on fixing this).

BjoernKW 8y ago
From my point of view, the main benefit of microservices is risk reduction. Microservices allow you to iterate rapidly and roll out new features without potentially affecting your application as a whole. "Do one thing and do it well." can also be an argument for using microservices. However, your application has to be of a certain size in order for that to be a valid point. If your application merely has a few endpoints, most of which are rarely - if ever - used as each other's inputs, then using microservices only adds development overhead without yielding considerable benefit. This is also why I'm critical of the surprisingly common one-size-fits-all "Use microservices for everything" approach. Microservices are just another design pattern. They're not a cure-all for every software development problem. I, for one, am doubtful about the front-end / back-end developer dichotomy because it tends to create new silos. I don't particularly like "full-stack developer" either because I think engineers should aim to be problem solvers rather than people who merely turn someone else's solution into code. Solving a particular problem in some cases might just involve the back-end or the front-end but in most cases it not only involves both but other areas such as requirements engineering or design as well.
- mlthoughts2018 8y ago
  > “From my point of view, the main benefit of microservices is risk reduction. Microservices allow you to iterate rapidly and roll out new features without potentially affecting your application as a whole.” I think this is the same reason why “do one thing and do it well” is useful in software components or shell utilities too. Basically, what you said about microservices is just a restatement of Single Responsibility Principle / decoupling, in general, and it’s not unique to microservices. In this sense, I’d say microservices are not a design pattern. Rather, it is an organizational strategy for minimizing the risks of coupling in other design patterns, which is also why microservices are explicitly a good idea in cases when a service might only have a few independent endpoints. I don’t buy the argument that it adds overhead. In fact I’d say it explicitly reduces overhead, because the boilerplate of wrapping it in a web service is just too tiny to care about compared with the overhead required to manage the complexity when a system tries to be responsible for too many things. In other words, I think good hygiene would say defaulting to microservices is the right idea, and consolidating should require evidence that it provides the expected benefits. Personally I am reminded of the funny Andrew Gelman quote, “Just because something is counter-intuitive doesn’t make it true.” I think it’s fashionable to criticize microservice designs and people jump on the bandwagon of pretending like criticizing microservice designs is somehow more well-rounded or more nuanced, but really it’s not. It’s just an attempt to make something fashionable via contrarianism. > “I think engineers should aim to be problem solvers” I agree with this very much and it is why I think cross-silo / cross-functional engineering is junk. The best way to organize a team of problem solvers is to clearly delineate the boundaries between what the problem solving responsibilities are. If people use the term “problem solver” to just be an ambiguous bucket of unlimited complexity by virtue of expecting people to “just solve the problem” no matter what the impedance mismatch between their skill area and the problem area, then it renders the term “problem solver” meaningless, and will just lead to a big mess of dysfunction. As an engineering manager, one of my most important jobs is to empower engineers to say “no.” To reject inappropriately allocated work or inappropriate cross-functional pivoting driven by ineffective short term business thinking. The engineers have to be empowered to say, “Hey, that work should not be assigned to us” because it signals whether the business overall has structured itself and staffed itself appropriately to actually respond to the business problems sent by Mother Nature at us. If we are structured wrongly, yet try to paper over it by pressuring engineers to be HR-approved “problem solvers” in the sense of more just doing what they are told instead of doing what would actually work, it will be medium-term detrimental and counter-productive. In this sense, I think “silo” is a good word, and indicates a good engineering culture where different silos have the power to say “no” and where specializations are respected and planned for by the management. Yes, of course that can be subverted into an anti-pattern in poorly run businesses, but that’s a property of bad management, not of “silos” in and of themselves.
  - BjoernKW 8y ago
    > I don’t buy the argument that it adds overhead. In fact I’d say it explicitly reduces overhead At the very least microservices add infrastructure overhead and network latency. In terms of everyday development complexity, instead of 1 project for a monolithic application you now have n projects, one for each microservice, which is fine if your organization is large enough and sufficiently well-organised to adhere to a "One team is responsible for each microservice." organisational pattern. If, however, your developers find themselves juggling multiple projects at a time and having to run several microservices on their local machine in order to make a change this might indicate that either there's an impedance mismatch between the communication structure of your organization and the design of your product (https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law) or that your application simply isn't sufficiently complex yet to warrant the use of microservice patterns. > In other words, I think good hygiene would say defaulting to microservices is the right idea Good hygiene is possible without microservices, too, just as microservices can be used to build the worst kind of monolith: The distributed monolith. > If people use the term “problem solver” to just be an ambiguous bucket of unlimited complexity by virtue of expecting people to “just solve the problem” no matter what the impedance mismatch between their skill area and the problem area, then it renders the term “problem solver” meaningless, and will just lead to a big mess of dysfunction. I suppose it depends on how an organisation manages and breaks down tasks. If for example you work on a user story level a single developer can indeed be responsible for implementing a complete story including all aspects that might be involved. "Responsible for" doesn't mean "You have to do it all by yourself.". If you have a particular problem ask a specialist and ideally everyone involved will even learn something new in the process. When viewed from an impedance mismatch perspective the front-end / back-end distinction in some ways is quite arbitrary. The prototypical example of impedance mismatch is the one between object-oriented programming and relational data. However, most would expect back-end developers to be both skilled in - say - Java and a relevant SQL dialect. The impedance mismatch between a front-end technology such as Angular and a back-end technology such as Spring (both of which make use of similar patterns) is much less pronounced than that between SQL and an object-oriented language.
    - mlthoughts2018 8y ago

> "which is fine if your organization is large enough and sufficiently well-organised to adhere to a "One team is responsible for each microservice." organisational pattern." I actually think the opposite is true. Having the developers on a single team manage N different service-specific projects is far simpler and lower overhead, because the infrastructure overhead of N different projects is the trivial part, compared with the complexity of managing the internal state of a single project responsible for too many things. It's like separating a software implementation into multiple different functional units, either different classes or modules or functions, to decouple them and keep them cleanly separated for testing. The "overhead" of N different specific classes is not actually a legitimate concern compared with the overhead of a single class trying to do N overlapping things (or functional units if using a different programming paradigm). It's no different for microservices. Plus it's fairly simple to run your own container orchestration layer that allows efficiently mapping different microservices onto heterogeneous hardware resources, whether in your own data center or with a cloud provider. For example, my team operates different web services for different machine learning applications. One service might do nothing but predict the age of a person in a photo. Another web service might do nothing by predict the gender. From experience, I can tell you that muddying these two things together into a single web service, just because the two endpoints are separate concepts, is a bad idea. It complicates testing, the organization of acceptance test data sets, the ability to scale out the resource needs of the different models separately, to control how they are deployed. You superficially feel like you are "saving effort" on the tiny boilerplate code that is repeated between the two to expose endpoints, sanitize inputs, etc. (most of which can be factored into an internal software package anyway, to have good code reuse), but really you are not at all, because the growth of the testing code, single-project deployment and build code, etc., dwarfs whatever tiny amount of repeat web framework stuff there might have been. They are just different logical units of concern, even if they are components of the same broader API of machine learning services we provide. I think it would represent a classic example of what you are describing as a case when you'd prefer not to organize it as microservices, and yet experience has conclusively shown my team at least that there is much greater boilerplate and mangled overhead involved with putting them into the same web service than with keeping them separate and duplicating an utterly tiny amount of work on the web service wrapper part. And this is all with the same team members maintaining and extended all of the services in a small company. It just streamlines and reduces our work to do it with microservices. > "If, however, your developers find themselves juggling multiple projects at a time and having to run several microservices on their local machine in order to make a change this might indicate that either there's an impedance mismatch between the communication structure of your organization and the design of your product" I actually disagree with this, and also it is an incorrect use of "Conway's Law" (which is about software organizational structure taking on the same organizational structure of the social and communication patterns of the company, and being limited to repeat those structures as the communication process limits the design). None of that affects sets of services differently whether they are bundles or separated as microservices. Conway's Law would affect the design limitation the same way in either case, and the organization as separate or coupled services would be a totally different thing, happening at a metaorganizational level to the level where Conway's law can be applicable to software. It's also actually quite nice and easy to write extremely simple tooling with e.g. docker-compose and a trivial Makefile to control local development that has effects spanning multiple microservices. And in fact this creates really nice, version controlled artifacts that indicate how that local testing and organization works, which is often severely lacking inside the internals of a more monolithic multi-service single project, where you might be able to run it locally with superficially simpler looking commands or resources, but then the internal operation is overly coupled between services and you cannot easily see which things you can modify or change without impacting other things in unintended ways. Lastly, I think you missed the point of my comment on impedance mismatch. The problem would be taking someone who is a specialized engineer in X but then expecting them to "be a problem solver" by spending time on Y, where the person's value add is huge if you let them work on X, but is tiny if you ask them to work on Y, even after accounting for whether the business needs X or Y at the moment. For example, taking one of your best front-end engineers and requiring them to fix some broken database normalization problem, or write a stored procedure, instead of having pre-arranged for the team to contain the necessary expertise such that the front-end person can delegate that task to someone whose comparative value add is all on the data engineering side, meanwhile the front-end person goes back to work on more front-end tasks, where their comparative value add is high. The impedance mismatch comes from recognizing that if you keep a specialist in X busy with work in X, it is more optimal for the company (and more morale building for the worker), than if you ask them to be preemptible to stop doing X and instead do Y or Z which ostensibly should be things farmed out to specialists in Y or Z. It can be this way with a backend engineer, a front-end engineer, a scientific computing expert, a database expert, whatever. You are taking the classic impedance mismatch example of an ORM too literally, because the general idea is that you are restricting the valuable flow of something (the specialized skill of the worker) instead of fundamentally reorganizing resources so as to not need to restrict it (by cleanly separating responsibilities into different silos of specialization).