8 ms·
I suspect that most people would be better off favoring inlined code over modules and microservices. It's okay to not organize your code. It's okay to have fil
by surprisetalk 4y ago
I suspect that most people would be better off favoring inlined code over modules and microservices.
It's okay to not organize your code. It's okay to have files with 10,000 lines. It's okay not to put "business logic" in a special place. It's okay to make merge conflicts.
The overhead devs spend worrying about code organization may vastly exceed the amount of time floundering with messy programs.
Microservices aren't free, and neither are modules.
[1] Jonathan Blow rant: https://www.youtube.com/watch?v=5Nc68IdNKdg&t=364s https://www.youtube.com/watch?v=5Nc68IdNKdg&t=364s
[2] Jon Carmack rant: http://number-none.com/blow/john_carmack_on_inlined_code.html http://number-none.com/blow/john_carmack_on_inlined_code.htm...
- jraph 4y agoHow can I sell your idea? I easily find my way in messy codes with grep. With modules, I need to know where to search to begin with, and in which version. Fortunately, I have never had the occasion to deal with microservices.
- nailer 4y ago> With modules, I need to know where to search to begin with You can just grep / search all the files. > in which version. grep / search doesn't search through time whether you're using one file or many modules. You probably want git bisect if you can't find where something is anymore (or 'git log' if you have good commit messages),
- hgsgm 4y agogrep can search multiple files at once.
- jraph 4y agoI actually use ripgrep. But modules can be in different repositories I haven't cloned yet.
- mypalmike 4y agoRepo search.
- dmitriid 4y agoJust use a proper IDE. It doesn't care about how your code is structured and can easily show you what you look for in context. (And other tools like symbol search https://www.jetbrains.com/help/idea/searching-everywhere.html https://www.jetbrains.com/help/idea/searching-everywhere.htm...)
- surprisetalk 4y agoI'm unsure how to "sell" this idea. I don't want to force my view on others until I truly understand the problem that they're trying to solve with modules/microservices. For searching multiple files, ripgrep works really well in neovim :) [1] https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep
- hgsgm 4y agoBlow and Carmack are game programmers. They are brilliant, but their local programs and data are tiny compared to distributed systems over social graphs, where N^2 user-user edges interact.
- AnIdiotOnTheNet 4y agoYou say that like real-time multiplayer gaming doesn't exist or something. Both of them have worked on those.... I think Carmack invented a lot of the techniques we use for those. Yeah sure, the scale is smaller, but you can't get away with making the user wait 10 seconds to render text and images to a screen either. I think the software world might be a lot better place if more developers thought like game developers.
- ivanhoe 4y agoLet's not turn it into penis measuring contents, please. Code organization and requirements differ significantly between different programming niches, and the today's accepted practices are not some randomly invented caprices, but the result of the slow (and painful) evolution we've been fighting through past decades. Each niche has optimized over time for its own needs and requirements. My web apis have hundreds of controllers and keeping them in separate files makes it way easier to manage. I know that because we used to keep it all in a single file and it sucked, so over time I learned not to do it anymore. Does it mean that embedded systems devs should organize their code in the same way? I have no idea, that's up to them to decide, based on their specific environment, code and experience.
- AnIdiotOnTheNet 4y ago> Each niche has optimized over time for its own needs and requirements. Sure, but those "needs and requirements" aren't necessarily aligned with things that produce good software, and I think a lot of software development these days is not aligned. Further, I think the evolutionary path of the web in particular has produced a monstrosity that we'd be better off scrapping and starting over with at this point, but that's a tangential discussion.
- pshirshov 4y agoModules are lot cheaper if you have a solver for them. Microservices are the same modules. Though they force-add distributiveness, even where it can be avoided, which is fundamentally worse. And they make integration and many other things lot harder.
- surprisetalk 4y agoWhat is a "solver"? Do you have any resources where I can learn about them?
- pshirshov 4y agoWell, anything what allows you to express the wiring problem in formal terms and solve it. A dependency injection library. Implicit resolution mechanism in Scala. They may solve basic wiring problem. distage can do more, like wire your tests, manage component lifecycles, consider configurations while solving dependency graph to alter it in a sound manner.
- alserio 4y agoI do not agree. I believe that professionally "most people" don't work on the same code every day and don't work alone. Modules are a mean of abstraction and are a classic application of "divide et impera" and you'll need them pretty soon to avoid keeping the whole thing in your head. But different cultures of programming have a different meaning of what a module is, so, maybe, I'm misunderstanding your point
- mbork_pl 4y ago> It's okay to have files with 10,000 lines. Ever since my time as a mathematician (I worked at a university) and using LaTeX extensively, I never understood the "divide your documents/code into many small files" mantra. With tools like grep (and its editor equivalents), jumping to definition, ripgrep et al., I have little problem working with files spanning thousands of lines. And yet I keep hearing that I should divide my big files into many smaller ones. Why, really?
- bombolo 4y agoIf you change 1 .c file, your compiler needs to recompile the .o file for that file and link it with the unchanged .o files. If you have everything into one .c file, you need to recompile the whole thing every change.
- beefield 4y agoBecause one day someone else may need to read and understand your code?
- hgsgm 4y agoMathematicians rarely collaborate by forking and improving other papers. They rewrite from scratch, because communicating ideas isn't considered the important part, getting credit for a new paper is.
- mbork_pl 4y agoFair enough, though I've been working as a programmer for over 6 years now (and I've been programming on and off, as a hobby, for over 3 decades).
- em500 4y agoBut is it generally easier to read and understand, say 10 files of 1000 lines or 100 files of 100 lines or 1000 files of 10 lines, compared to one 10,000 lines file? (I don't know the answer, and don't have any strong opinion on this.)
- SomeoneOnTheWeb 4y agoI strongly disagree on this one. 10+K lines files are absolutely unreadable most of the time. Separating business logic than other parts of the application helps maintaining it and making everything evolve in parallel, without mixing things up. It also helps to clearly see where business logic happens.
- pshirshov 4y agoAnd of course it's lot easier to read 200k+ LoC shattered around twenty repos.
- koliber 4y agoThe problem arises when you need to read the code of other modules or services. If you can rely on them working as they should, and interact with them using their well-defined and correctly-behaving interfaces, you won't need to read the code. I'm a proponent of keeping things in a monolith as long as possible. Break code into files. Organize files into modules. A time may come when you need to separate out services. Often, people break things out too early, and don't spend enough effort on thinking how to break things up or on the interfaces.
- rightbyte 4y ago> The problem arises when you need to read the code of other modules or services. If you can rely on them working as they should, and interact with them using their well-defined and correctly-behaving interfaces, you won't need to read the code. You can say the exact same thing about C-headers though.
- pshirshov 4y ago> and interact with them using their well-defined and correctly-behaving interfaces, you won't need to read the code. Don't you want determinism and deterministic simultations? If you do, you'll also need stub implementations (mocks, dummies) for your interfaces. Some notes on that: https://blog.7mind.io/constructive-test-taxonomy.html https://blog.7mind.io/constructive-test-taxonomy.html > A time may come when you need to separate out services. Most likely it won't if you organise properly. For example, if each your component is an OSGi module.
- pron 4y agoIt's a valid hypothesis, but without empirical data the question is not easy to settle (and neither Carmack nor Blow are notable authorities on systems that have to be maintained by changing teams of hundreds of people that come and go, maintaining a large codebase over a couple of decades; if anything, most of their experience is on a very different kind of shorter-lived codebases, as game engines are often largely rewritten every few years). Also, the question of inlined code mostly applies to programming in the small, while modules are about programming in the large, so I don't think there's much relationship between the two.
- hbrn 4y ago> neither Carmack nor Blow are notable authorities on systems that have to be maintained by changing teams of hundreds of people that come and go Most systems don't have to be maintained by hundreds of people. And yet they are: maybe because people don't listen to folks like Carmack? We like stories about huge teams managing huge codebases. But what we should really be interested in is practices that small teams employ to make big impact.
- pron 4y agoI didn't mean a team of hundreds maintaining the product concurrently, but over its long lifetime. A rather average codebase lifetime for server software is 15-20 years. The kind of codebase that Carmack has experience with has a lifetime of about five years, after which it is often abandoned or drastically overhauled, and it's not like games have an exceptional quality or that their developers report an exceptionally good experience that other domains would do well to replicate what games do. So if I were a game developer I would definitely be interested in Carmack's experience -- he's a leading expert on computer graphics (and some low-level optimisation) and has significant domain expertise in games, but he hasn't demonstrated some unique know-how in maintaining a very large and constantly evolving codebase over many years. Others have more experience than him in domains that are more relevant to the ones discussed here.
- louthy 4y ago... then before you know it your code-base is 10 million lines long and you have no idea where anything is, what the side-effects of calling X are, what's been deprecated, onboarding is a nightmare, retention of staff is difficult, etc. You may be right for the smallest of applications, or whilst you're building an MVP, but if your application does anything substantial and has to live forever (a web app, for example), then you will have to get organised.
- worldsayshi 4y agoI think we are conflating (at least) two different issues here because we don't have a good way to deal with them separately. Closure on one hand and implicit logic on the other. Most of the time when I split out a method what I want is a closure that is clearly separated from the rest of the code. If I can have such closures, where input and output is clearly scoped, without defining a new method, that might be preferable. Lambdas could be one way to do it but they still inherit the surrounding closure so it's typically not as isolated as a method.
- louthy 4y ago> I think we are conflating (at least) two different issues For sure, I have absolutely no idea how your comment relates to what I wrote.
- worldsayshi 4y agoOops, it looks like I answered the wrong comment here. :/
- louthy 4y agoIt happens to the best of us! :)
- CraigJPerry 4y agoCounter point - https://github.com/microsoft/TypeScript/blob/main/src/compiler/checker.ts https://github.com/microsoft/TypeScript/blob/main/src/compil... - your task is to just make it a little bit faster. Where do you begin with a 2.65mb source file? It’s easy to miss the point of what the OP is saying here and get distracted by the fact this file is ridiculously huge. This file used to be a paltry 1k file, a 10k file, a 20k SLOC file… but it is where it is today because of the OP suggested approach.
- c0balt 4y agoYeah, I think this a very much worst case scenario though. <rant> You will (for most statements) both be able to find a best and worst case. That's the catch with most generalized statements/ principles, e.g., DRY. The challenge is to find a good "enough" solution because perfection is usually either unfeasibly expensive or impossible (different viewpoints, ...) </rant> Though it's kinda hilarious that the source code of a MS project is not natively viewable on a MS platform.
- jiggawatts 4y ago> your task is to just make it a little bit faster. Where do you begin With a trace from a profiler tool, which will tell you which line number is the hot spot. If run from any modern IDE, you can jump to the line with a mouse-click. In essence, the file boundaries ought not to make any difference to this process.
- CraigJPerry 4y ago>> In essence, the file boundaries ought not to make any difference to this process. I find this the strongest counter argument. File boundaries shouldn’t matter - but in practice they do. The ide will give up doing some of its analysis because this is not an optimised use case, the IDE vendors don’t optimise for it. Some IDEs will even give up just simple syntax highlighting when faced with a large file, never mind the fancy line by line annotations.
- dahfizz 4y ago
- ranting-moth 4y agoI can't say I agree with all your "okays", although If you prefix them with "In some cases it's okay", then I understand where you're coming from. The problem is when it's OK and for how long. If you have a team of people working with a codebase with all those "okays", then they have to be really good developers and know the code inside out. They have to agree when to refactor a business login out instead of adding a hacky "if" condition nested in another hacky "if" condition that depends on another argument and/or state. I guess what I'm trying to say that if those "okays" are in place, then there's a whole bunch of unwritten rules that come in place. But I agree that microservices certainly aren't free (I'd say they are crazy expensive) and modules aren't free either. But all those "okays" can end up costing you your codebase also.
- kubb 4y agoRespectfully, rants by niche celebrities are not something we should base our opinions on. If you're a single dev making a game, by all means, do what you want. If you work with me in a team, I expect a certain level of quality in the code you write that will get shipped as a part of the project I'm responsible for. It should be structured, tested, malleable, navigable and understandable.
- worldsayshi 4y agoI feel like this is a knee jerk reaction to the hyperbole of the parent comment rather than the contents of the actual linked talks. I'm watching Jonathan Blow's talk linked above and your comment does not seem relevant to that. Jonathan's points so far seem very reasonable. Rather than arguing for 10000 lines of code it's arguing that there is such a thing as premature code split. Moving code into a separate method has potential drawbacks as well. One suggested alternative is to split reusable code into a local lambda first and lift it into a separate code piece only once we need that code elsewhere. It seems to me that such approach would limit the complexity of the code graph that you need to keep in your head. (Then again, when I think about it maybe the idea isn't really that novel.)
- fruit2020 4y agoSo you think it’s easier to keep in your head lambdas in a 10k line file vs methods split by functionality across a number of smaller files?
- surprisetalk 4y ago> It should be structured, tested, malleable, navigable and understandable. Great comment! I personally find that most codebases are overstructured and undertested :) In my experience, module boundaries tend to make code less malleable, less navigable, and less understandable.
- deleted 4y ago[deleted]
- vsareto 4y ago
- jameshart 4y agoI’ve said this before about applying Carmack’s architectural input on this topic: Games are highly stateful, with a game loop that iterates over the same global in-memory data structure as fast as it can. You have (especially in Carmack era games) a single thread performing all the game state updates in sequence. So shared global state makes a ton of sense and simplifies things. Most web applications are highly stateless with request-oriented operations that access random pieces of permanently-stored data. You have multiple (usually distributed) threads updating data simultaneously, so shared global state complicates things. That game devs gravitate towards different patterns for their code than web service devs should not be a surprise.
- mr90210 4y agoI felt suspicious as soon as I saw Jon Carmack’s website being mentioned in a conversation about Microservices.
- surprisetalk 4y agoI hope this quote from the Carmack essay shows that his argument isn't necessarily restricted to game development: ---------- style C: void MajorFunction( void ) { // MinorFunction1 // MinorFunction2 // MinorFunction3 } > I have historically used "style A" to allow for not prototyping in all cases, although some people prefer "style B". The difference between the two isn't of any consequence. Michael Abrash used to write code in "style C", and I remember actually taking his code and converting it to "style A" in the interest of perceived readability improvements. > At this point, I think there are some definite advantages to "style C", but they are development process oriented, rather than discrete, quantifiable things, and they run counter to a fair amount of accepted conventional wisdom, so I am going to try and make a clear case for it. There isn't any dogma here, but considering exactly where it is and isn't appropriate is worthwhile. > In no way, shape, or form am I making a case that avoiding function calls alone directly helps performance.
- IshKebab 4y ago
- yodsanklai 4y ago> It's okay to have files with 10,000 lines. It's okay not to put "business logic" in a special place. Couldn't disagree more. As usual, it's a tradeoff. You could spend an infinite amount of time refactoring already fine programs. But complex code decreases developers productivity by orders of magnitude. Maybe it's not always worth refactoring legacy code, but you're always much better off if your code is modular with good separation of concerns.
- thiht 4y ago> It's okay not to put "business logic" in a special place It's not. This is the thing where you start thinking "YAGNI", yadayada, but you inevitably end up needing it. Layering with at least a service and a database/repositories is a no brainer for any non-toy app considering the benefits it brings. > It's okay to have files with 10,000 lines 10.000 lines is a LOT. I consider files to become hard to understand at 1.000 lines. I just wc'd the code base I work on, we have like 5 files with more than 1.000 lines and I know all of them (I cringed reading the names), because they're the ones we have the most problems with.
- PathOfEclipse 4y agoAs someone who has personally dealt with files as large as 60K lines, I disagree completely. I believe instead that structure and organization should be added as a business and codebase scales. The problem I think most orgs make is that, as they grow more successful, they don't take the time reorganize the system to support the growth, so, as the business scales 100x in employee account, employee efficiency is hampered by a code organization that was optimized for being small and nimble. It gets worse when the people who made the mess quit or move on, leaving the new hires to deal with it. I've seen this pattern enough times to wonder if it gets repeated with most companies or projects. I do agree that microservices and/or modules aren't magical solutions that should be universally applied. But they can be useful tools, depending on the situation, to organize or re-organize a system for particular purposes. Anecdotally, I've noticed that smart people who aren't good programmers tend to be able to write code quickly that can scale to a particular point, like 10k-100k lines of code. Past that point, productivity falls rapidly. I do believe that part of being a skilled developer is being able to both design a system that scales to millions of lines of code across an organization, and to operate well on one designed by someone else.
- fruit2020 4y agoWell said. You will see very fast if a dev is experienced or not by looking at code organization and naming. Although I deal with experienced ones that just like to live in clutter. You can be both smart and stupid at the same time.
- nonethewiser 4y agoNot sure I totally agree, but one strong point against factoring in-line code out into a function: You have to understand everything that calls that function before you change it. This is not always obvious. It takes time to figure out where it's called. IDEs make this easier but not bullet proof. Getting it wrong can cause major, unexpected problems.
- xnorswap 4y agoBroadly my heuristic for this is, "Would it make sense to run these functions in the other other?". If you split up MegaFunction(){} to Func1(){} Func2(){}, etc, but it never makes sense to call Func2 except after Func1, then you haven't actually created two functions, you've just created one function in two places. Refactoring should be about logical separation not about just dicing a steak because it's prettier that way.
- riknox 4y agoI think that's a reasonable heuristic, but I'd also say you have to take into account the human readability aspect of it. It sometimes does make sense IMO to split solely for that, if it allows you to "reduce" a complicated/confusing operation to a string name, leading to it being easier to understand at a glance.
- nottorp 4y agoShouldn't split MegaFunction into Func1 and Func2, you should: MegaFunction() { Func1(); Func2(); ... FuncN(); } People will call MegaFunction() but it will be logically split internally.
- xnorswap 4y agoThis is how it's taught in school but the argument is that you shouldn't do that, instead just inline Func1 and Func2 and comment it better. By chopping up MegaFunction like this, you've not actually separated Func1 and Func2 if they aren't really independent, they're just MegaFunction in disguise but now split across two places making it more difficult, not easier, to reason about. If you need the state (implied or explicitly passed in and out) from having executed Func1 to run Func2 then you're just creating spaghetti code. You're taking a big ball of mud and smearing it around instead of actually tackling the abstraction. This is how you end up with Func(a, b, c, d, &e, &f) which ends up changing your MegaFunction state. Or just as bad, Func1,2,N are all private functions, never called anywhere else outside the class MegaFunction is in, so logically (and from the point of view of testability) it's no different to having them all inline. If you're creating a function that's only ever called once, then instead of a function call you're probably better off with a comment to "name" that block and explain the process instead.
- snarf21 4y agoI think you are missing one point: mental load. I doubt people keep all their files in one directory or all their emails in one folder or just have one large drawer with every possible kitchen utensil in a pile. The same is true for code. Organizing your code around some logical divisions allows for thunking. I will agree that some people take it too far and focus too much on "perfect". But even some rudimentary directories to separate "areas" of code can save a lot of unnecessary mental gymnastics.
- hbrn 4y agoI agree, but the problem is that eventually it becomes not okay. So it requires a bit of nuance to understand when it is and when it isn't. Unfortunately most engineers don't like nuance, they want one-size-fits-all solutions.
- swsieber 4y ago> It's okay to have files with 10,000 lines. I find there are practical negative consequences to having a 10,000 line file (well, okay; we don't have those at work, we have one 30k line file). It slows both the IDE and git blame/history when doing stuff in that file (w.r.t. I'll look at the history of the code when making some decisions). These might not be a factor depending on your circumstances (e.g. a young company where git blame is less likely to be used or something not IDE driven). But they can actually hurt apart from pure code concerns.
- mypalmike 4y agoOrganizing code and services so that a rotating crew of thousands of engineers can be productive is critical to companies like Amazon and Google and Netflix. Inlining code is a micro-optimization on top of a micro-optimization (that is, choosing to [re]write a service in C/C++), not an architectural decision.
- semicolon_storm 4y agoAs someone who works at a place that previously lived by: >It's okay to not organize your code. It's okay to have files with 10,000 lines. It's okay not to put "business logic" in a special place. It's okay to make merge conflicts. I absolutely disagree. It's "okay" if you're struggling to survive as a business and worrying about the future 5+ years out is pointless since you don't even know if you'll make it the next 6 months. This mentality of there being no need for discipline or craftsmanship leads to an unmanageable codebase that nobody knows how to maintain, everybody is afraid to touch, and which can never be upgraded. You don't see the overhead of throwing discipline out the window because it's all being accrued as technical debt that you only encounter years down the road.
- bullen 4y agoI think we're all confused over the definition. Also one might understand what all the proponents are talking about better if they think about this more as a process and not some technological solution: https://github.com/tinspin/rupy/wiki/Process https://github.com/tinspin/rupy/wiki/Process All input I have is you want your code to run on many machines, in fact you want it to run the same on all machines you need to deliver and preferably more. Vertically and horizontally at the same time, so your services only call localhost but in many separate places. This in turn mandates a distributed database. And later you discover it has to be capable of async-to-async = no blocking ever anywhere in the whole solution. The way I do this is I hot-deploy my applications async. to all servers in the cluster, this is what a cluster node looks like in practice (the name next to Host: is the node): http://host.rupy.se http://host.rupy.se if you click "api & metrics" you'll see the services. With this not only do you get scalability, but also redundancy and development is maintained at live coding levels. This is the async. JSON over HTTP distributed database: http://root.rupy.se http://root.rupy.se (2000 lines hot-deployable and I can replace all databases I needed up until now)