14 ms·

Systems ideas that sound good but almost never work

• PaulHoule 2y ago
  (1) DSLs work great sometimes. See https://www.jooq.org/ https://www.jooq.org/ (2) Elastic Load Balancer is a control loop responsive to workloads, that kind of thing is a commodity (3) Under-provisioning is rampant in most industries; see https://erikbern.com/2018/03/27/waiting-time-load-factor-and-queueing-theory.html https://erikbern.com/2018/03/27/waiting-time-load-factor-and... and https://www.amazon.com/Goal-Process-Ongoing-Improvement/dp/0884271951 https://www.amazon.com/Goal-Process-Ongoing-Improvement/dp/0... (4) Anomaly detection is not inherently a problem of distributed systems like the others, but someone facing the problems they've been burned with might think they need it. Intellectually it's tough. The first algorithm I saw that felt halfway smart was https://scikit-learn.org/1.5/modules/outlier_detection.html#isolation-forest https://scikit-learn.org/1.5/modules/outlier_detection.html#... which is sometimes a miracle and I had good luck using it on text with the CNN-based embeddings we had in 2018 but none at all w/ SBERT.
  • Spivak 2y ago
    Also k8s is living proof that control loops can and do work. That's like the entire point of its existence.
    • kernelbandwidth 2y ago
      Counterpoint: k8s is a bad orchestration system with bad scaling properties compared to the state machine versions (Borg, Tupperware/Twine, and presumably others). I say this as someone who has both managed k8s at scale and was a core engineer of one of the proprietary schedulers.
    • xorcist 2y ago
      It is, and from experience it is also a good example how control loops are harder than you think. Few people understand it, and it is the source of much underlying trouble.
  • billyp-rva 2y ago
    > (1) DSLs work great sometimes. I'll take a stab at fleshing this out: DSLs work great when they have an IDE with autocomplete and a quick (or instant) feedback loop.
    • diggan 2y ago
      I was gonna say something like "DSLs work great when they're small, purposeful and easy to test", I guess yours kind of helps when they're not what I'd suggest :)
      • PaulHoule 2y ago
        Maybe his problem is with those yucky distributed systems as in Kube, Antilles, etc. I write plain ordinary Java programs that work with a cloud API and compile bash scripts that colonize machines.
        • verdverm 2y ago
          I think of Kube as more of abstractions whereas Wasp & Dark lang would be DSLs for the same concepts
      • tinthedev 2y ago
        This whole thread said exactly what I wanted to write. Feels bad to be so pre-empted. DSLs that solve a specific problem with a page or two of documentation overhead are great. Trying to reinvent paradigms or scope creep is where the pain comes in. Seems like the post author has been burned by that type of DSLs.
        • f1shy 2y ago
          I would say that is the line between DSL and just “L” another language…
        • earnestinger 2y ago
          > DSLs that solve a specific problem with a page or two of documentation overhead are great. Do you have any example? I’ve heard lots of good things of dsl, but never had the luck to witness it’s full glory. (except for regex, which I love, but it has more than two pages of docs)
  • pizlonator 2y ago
    I think you’re responding to the tweet from Martin that Steven includes at the top, not to Steven’s list.
    • kelnos 2y ago
      That's fine. The points made in the tweet are interesting to discuss too.
    • echelon 2y ago
      The tweet has some claims that don't fly. You don't get the luxury of offline migrations or single-master writes in the high-volume payments space. You simply don't have the option. The money must continually flow.
  • stickfigure 2y ago
    jOOQ is a disaster and I would not recommend it to anyone. You write some SQL queries, test them in datagrip or whatnot, then spend the next several hours figuring out how to convert them to the DSL. This problem is compounded when you use "exotic" SQL features like json expressions. Debugging is "print the generated sql, copy it into datagrip/whatnot, tune the query, then figure out how to retrofit that back into the DSL". It's a huge waste of time. The primary selling point of jOOQ is "type safe queries". That became irrelevant when IntelliJ started validating SQL in strings in your code against the real data. The workflow of editing SQL and testing it directly against the database is just better. jOOQ reinforces the OP's point about DSLs.
    • geophile 2y ago
      This is a very specific and popular subset of the DSL point: Let's just invent a language L that is better than horrible standard language X but translates to X. Imagine the vast cubicle farms of X programmers who will throw off their chains and flock to our better language L! In many scenarios (including JOOQ and all ORMs), X is SQL. I should know, I spent years working on a Java-based ORM. So believe me when I say: ORMs are terrible. To use SQL effectively, you have to understand how databases work at the physical level -- what's a B-tree lookup, what's a scan, how these combine, etc. etc. You can often rely on the optimizer to do a good job, but must also be able to figure out the physical picture when the optimizer (or DBA) got things wrong. You're using an ORM? To lift a phrase from another don't-do-this context: congratulations, you now have two problems. You now have to get the ORM to generate the SQL to do what really needs to be done. And then there are the generalizations of point made above: There are lots of tools that work with SQL. Lots of programmers who know SQL. Lots of careers that depend on SQL. Nobody gives a shit about your ORM just because it saves you the trouble of the easiest part of the data management problem.
      • sroussey 2y ago
        > To use SQL effectively, you have to understand how databases work at the physical level -- what's a B-tree lookup, what's a scan, how these combine, etc. This is a good reason to use an ORM. But also, as a ORM designer, don’t let the ORM be flexible to do any SQL. Only let it do performant data access.
  • corinroyal 2y ago
    I'm always baffled by hate for DSLs until I realize that what people are criticizing aren't DSLs, but DSLs you have to write from scratch. If you host your DSL on Lisp, then all you have to write is your domain logic, not the base language. Most of the work is already done, and your language is useful from day one. I don't understand why people insist on creating new languages from scratch just to watch them die on the vine, when these langs could have been hosted DSLs on Lisp and actually get used.
    • TimTheTinker 2y ago
      Not just Lisp, but any language that has strong support for either literal in-language data expressions like JSON or YAML, or meta-language support like Ruby, Elixir, JSX/TSX (or both!). Every time you write a React JSX expression, terraform file, config.yaml, etc., you're using a DSL. I once wrote a JSON DSL in Ruby that I used for a template-based C# code generator. This enabled a .NET reporting web app to create arbitrarily shaped reports from arbitrary rdmbs tables, saving our team thousands of hours. Another team would upload report data to a SQL Server instance, write a JSON file in the DSL, check it against a tiny schema validator website, submit it, and their reports would soon be live. One of the most productive decisions I ever made.
      • hot_gril 2y ago
        Technically yeah, but JSX isn't what people think of when you mention a DSL. I know JS, I know HTML, so I know JSX immediately since it's just templatized HTML inside JS.
    • CyberDildonics 2y ago
      This is generally a terrible way to work. Making a bunch of custom syntax even in the same language is just adding more stuff to memorize for no gain. Even in C using the "goes to operator" of while(i --> 0) or using special operator overloading like the C++ STL >> and << operators for concatenation is just making people memorize nonsense so someone writing can be clever. People don't give presentations with riddles and limericks either. It can be clever as a puzzle but when things need to get done, it is just indulging someone showing off their cleverness at the expense of everyone who has to deal with it.
  • travisjungroth 2y ago
    I've written two DSLs (one with a team) and I'd consider them both successful. They solved the problem and no one cursed them out. I think the most important factor is they were both small. They were very similar. I even reused the code. One was writing rules to validate giant forms, the other was writing rules to for decisions based on form responses. Ok, just ranting on DSLs. Good DSLs take someone from can't to can. A DSL that's meant to save time is way less likely to be useful because it's very likely to not save you time. In both of my DSLs, it's that we needed to get complex domain behavior into the program. So you either need to teach a programmer the domain, partner a programmer with a domain expert, or teach a domain expert how to program. Putting the power in the hands of the domain expert is attractive when there's a lot of work to be done. It frees up programmers to do other things and tightens the feedback loop. If it's a deep domain, it's not like you want to send your programmer to school to learn how to do this. If it's shallow, you can probably have someone cheaper do it. A DSL comes with a lot of cognitive overhead. If the other option is learning a full programming language, this becomes more reasonable. A time saving DSL is where someone already knows how to write code, they just want to write less of it. This is generally not as good because the savings are marginal. Then when some programmer wants to change something, they have to learn/remember this whole DSL instead of more straightforward code. Actually, this makes a simpler rule of thumb. A DSL for programmers is less likely to be a good idea than a DSL for non-programmers.
    • f1shy 2y ago
      DSL are just great, have never failed for me. I have done it several times, the last one just the past year, for programming complex ASICs. I‘ve seen uncountable times working like a charm. I cannot understand why seems it was bad for him…
      • travisjungroth 2y ago
        I think there are two big reasons DSLs often underdeliver. The first is not understanding the cognitive overhead budget. If it's something where it's being used infrequently or by a lot of new people, that's a lot of overhead to be spent each time. Sometimes people think of writing DSLs for tests because they have to write a bunch, but it's really easy for this to suck. I have a test turn red and now I have to learn a DSL to deal with it? Ew. The second is fuzzier. It's putting a DSL over something complex and hoping this will fix things. Writing SQL queries for this system takes a bunch of time and is error prone? Just put a DSL over it! Except all those details and errors are probably going to leak right through your DSL. You want to master the domain before you put a DSL over it.
  • hot_gril 2y ago
    I've yet to see a DSL work great. Every single time, I'm asking "why isn't this just Python (or some other lang)" especially the times when it's some jacked up variant of Python.
    • dude187 2y ago
      Groovy running on jython
    • hot_gril 2y ago
      Oh wait, regex is a DSL
  • einpoklum 2y ago
    (3.b) "Being clever rather than over-provisioning" is not generally thought of as a good idea. People would be rather apprehensive if you told them "I'm don't something really clever so we can under- or exactly-provision". I mean, sure, it may indeed work, but that's not the same thing. (5) Hybrid parallelism - also, many people think it's a bad idea because it makes your software system more complex. Again, it may be very useful sometimes, but it's not like many people would go "yes, that's just what I'm missing right now, let's do parallelism with different hardware and different parts of the workflow and everything will run something kind of different and it'll all work great like a symphony of different instruments".
  • crabbone 2y ago
    I've never seen a good DSL beside something like regular expressions, and even there, I hear, a lot of people are upset by the language. Examples of popular DSLs that I would characterize as bad if not outright failures: * HCL (Terraform configuration language). It was obvious from the very beginning that very common problems haven't been addressed in the language, like provisioning a variable number of similar appliances. The attempts to add the functionality later were clumsy and didn't solve the problem fully. * E4X (A JavaScript DSL for working with XML). In simple cases allowed for more concise expression of operations on XML, but very quickly could become an impenetrable wall of punctuation. This is very similar to Microsoft's Linq in that it gave no indication to the authors of how computationally complex the underlying code would be. Eventually, any code using this DSL would rewrite it in a less terse, but more easy to analyze way. * XUL (Firefox' UI language for extending the browser's chrome). It worked OK if what you wanted to do was Firefox extensions, but Firefox also wanted to sell this as a technology for enterprise to base their in-house applications on Firefox, and it was very lacking in that domain. It would require a lot of trickery and round-about ways of getting simple things done. * Common Lisp's string formatting language (as well as many others in this domain). Similar to above: works OK for small problems, but doesn't scale. Some formatting problems require some very weird solutions, or don't really have a solution at all (I absolutely hate it when I see code that calls format recursively). All in all. The most typical problem I see with this approach is that it's temporary and doesn't scale well. I.e. it will very soon run into the problems it doesn't have a good solution for. Large programs in DSL languages are often a nightmare to deal with.
• bee_rider 2y ago
  What do people want to use hybrid parallelism for in systems programming? It works really well in the HPC world I think (MPI+OpenMP on a node is the de-facto standard). But… I dunno, I guess when I think systems programming I think of the stuff doing all the bookkeeping. The bookkeeper better not steal all my cores! DSL is a sort of interesting one. What’s a DSL for systems programming? I’ll naively throw C and Rust as the systems programming DSLs. Of course, the domain of systems programming is, uh, controlling all the hardware. So it isn’t that surprising that the DSLs of systems programming quickly become the languages that everybody wants to use for everything, right? The problem isn’t the “specific language” part, it is that a good enough systems programming language quickly gets the domain of “everything,” haha.
  • mrkeen 2y ago
    Here's a DSL example https://copilot-language.github.io/ https://copilot-language.github.io/
• deleted 2y ago
  [deleted]
• shermantanktop 2y ago
  There are many successful examples of all of these. Using “almost” as an escape hatch doesn’t work here. This is just pessimism and weary cynicism. I get it, I’ve felt that way too, and sometimes it’s hard to talk an eager engineer out of a bad idea. But for me, this vibe is toxic.
  • yuliyp 2y ago
    A lot of those "successful" examples have teams of battle-scarred engineers dealing with all the failures of those ideas. Control loops running away to infinity or max/min bounds, a cache that can't recover from a distributed failure, corrupted live-migrated state, bursts causing overload at inconvenient times, spurious anomaly-detection alerts informing you of all the world's holidays, etc. Underneath all of those ideas is a tangle of complexity that almost everyone underestimates.
    • NBJack 2y ago
      I think that's fair: the success stories are very rarely "yay, we did it!" but much more often "this single change was the sole focus of a team/multiple teams for X months, and the launch/release/fix was considered a culmination of a significant investment of resources".
    • api 2y ago
      I know I'm not alone in this, but after doing this for more than 20 years I can't shake the idea that we are doing it wrong -- meaning programming. Is it really this nit-picky, brittle, and hard? The brittleness is what gets me. In physical mechanical and even analog electrical systems there are tolerances. Things can almost-work and still work for varying degrees of work. Software on the other hand is unbelievably brittle to the point that after 50 years of software engineering we still really can't re-use or properly modularize code. We are still stuck in "throw it away and do it over" and constantly reinventing wheels because there is no way to make wheels fit. The nature of digital means there are no tolerances. The concept isn't even valid. Things fit 100% or 0%. We keep inventing languages. They don't help. We keep inventing frameworks. They don't help. We keep trying "design patterns" and "methodologies." They don't help. If anything all the stuff we invent makes the problem worse. Now we have an ecosystem with 20 different languages, 50 different runtimes, and 30 variants of 5 OSes. Complexity goes up, costs go up, reusability never happens, etc. I remember for a while seeing things like the JVM and CLR (the VM for C# and friends) as a way out-- get away from the brittle C API and fully compiled static native code and into a runtime environment that allowed really solid error handling and introspection. But that paradigm never caught on for whatever reason, probably because it wasn't free enough. WASM is maybe promising.
• jmsdnns 2y ago
  p2p cache sharing is an interseting one. it seemed to work for spotify for a long time, but they eventually found it easier to use edge caching. afaik they still use it for places with less developed infrastructure here is a paper written by the people who designed it: https://www.csc.kth.se/~gkreitz/spotify-p2p10/ https://www.csc.kth.se/~gkreitz/spotify-p2p10/ this part of the abstract gets right to the point: 8.8% of music data played comes from Spotify's servers while the median playback latency is only 265 ms (including cached tracks)
  • dividuum 2y ago
    Steam added a P2P caching mechanism recently and allows downloading games for other local machines that already have the game installed. Greatly helps with avoiding duplicate downloads on a steam deck, for example. I also added a similar mechanism to my own product years ago. Works flawlessly. Using content based addressing already made it quite easy to implement.
• norir 2y ago
  DSLs make the impossible possible. Whether this is good or bad depends on the design and implementation.
  • heisenbit 2y ago
    Compared to general purpose programming when using real world DSLs one also can encounter the case where the unusually possible becomes virtually impossible.
    • fancyfredbot 2y ago
      Which often is exactly what the DSL authors would have intended. If a DSL can make the common case faster, easier to program, and less buggy then that DSL is a success. The uncommon case should be written in a general purpose language just as it would have been if there was no DSL. Writing a DSL to be general enough to handle every possible use case is probably going to end up being a general purpose language, and it's probably going to be worse than the one you started with.
• ajcp 2y ago
  It feels like problem with these aren't the ideas, rather the "let's just" approach/expectation in front of them. For instance "Let's just add an API." I think the approach to an API as "just" a feature to your product will be about as successful as saying "let's just add a UI". To implement a successful UI one needs to be thoughtful, thorough, and bring in people who specialize in it. Why should any other interface for your product be any different? It's not that it's a bad, or good, idea, rather one that shouldn't "just" be done.
  • vasco 2y ago
    Most professional advice is like relationship advice, people extrapolate something that went wrong for them into general cases but that rarely directly applies to someone who is not you in that very similar situation. And the advice that seems to work consistently gets so generic that is almost useless. Something like "be thoughtful, try to do the right thing and reflect back to see how it went" doesn't write blog posts or sells books!
  • gmuslera 2y ago
    Cargo cults around these system ideas are dangerous. The idea may be good, but without understanding what really makes them work or not is dangerous.
    • fullstackchris 2y ago
      Dangerous? In the sense a bad API is dangerous? If so then I've been in danger consuming catastrophic APIs for at least a decade now
  • justin_oaks 2y ago
    Exactly. There's no "just" when you're adding an API. APIs require a lot of work and complexity to do right: APIs need to be well-designed or the client may need to make multiple API calls when one should suffice. Or the API could be confusing and people will call it wrong or fail to use it at all. APIs require authentication and authorization. This means setting up OAuth2 or at least having a secure API token generation, storage, and validation. APIs need handle data securely or you'll leak data you shouldn't, or allow modification you shouldn't. APIs need to be performant or you database may be crushed under the load. This may involve caching which then adds the complexity of cache invalidation and other servers/processes to support caching. APIs need to be rate-limited or sloppy clients will hammer your API. APIs require thorough documentation or they're useless. You may even have to add SDKs (libraries for different programming languages) to make it easier for people to use the API. APIs need good error messages or users who are getting started will not know why their calls are failing.
• insane_dreamer 2y ago
  "seldom work as expected" would have been more accurate than "almost never work" I did like this: > More importantly, an offering an API doesn’t mean anyone wants to use it. Almost every new API comes up because the co/product wants features, but it doesn’t want to prioritize them enough and the theory is the API will be “evangelized” to some partner in the space. Turns out those people are not sitting around waiting to fill in holes in your product.
• stevebmark 2y ago
  I would add "Domain Driven Design" - locking your business design in place by trying to make your application match your business structure is a recipe for disaster. If you have a small or stagnant business you probably won't notice any issues. If your business is successful and/or grows, you're going to immediately regret trying to build domains with horrific descriptive names tied to your already obsolete business practices. Instead, design around functionality layers (how we've been doing it for decades, tried and true), and as much as possible keep business logic in config, rows in databases, and user workflows, which makes them extremely flexible.
  • jkaptur 2y ago
    Along with "make impossible states impossible to represent". If you're designing your types to make a state unrepresentable, you'd better be absolutely sure the state really is impossible for the lifetime of the design.
    • spencerflem 2y ago
      If its not impossible, you should handle it though. I think what that quote is against is the common middle ground where states are expected to be 'impossible' and thus not handled and cause bugs when they are found to be not actually. Either deciding that they are possible or must be impossible is usually better and which one to go with depends on the specifics
    • pvillano 2y ago
      The variant of this I've found useful is to have a separate types for raw/dirty and parsed/validated data
    • wesselbindt 2y ago
      I don't understand, you're not allowed to change the types? Suppose you disallow any number other than 1, 2, and 3, and you model this with some enum with three members or something. Then you find out that actually a prospective client would love to also work with with the number 4, then you just add 4 to the enum in your next release, no?
• tinthedev 2y ago
  The whole post, and both the Twitter messages sound like they refer to a very specific style of work. These things "sound good but almost never work" when they're taken as an afterthought or committed to without due research/process/design. Any architect worth their salt will avoid implementing hard solutions to these problems. These are mostly here to not solve crucial issues... but the article seems to be addressing a developer profile that adds APIs or asynchronous processing casually. Sounds like a strawman, or just exceedingly pessimistic look at the industry.
• riwsky 2y ago
  So many people here trying to thread the needle looking for subtle decision functions for exceptions. It's pretty simple, really: these ideas are awesome when I do them, and never work as intended when that idiot before me did them.
  • plagiarist 2y ago
    That sounds exactly right. And sometimes the idiot before me was myself from several months ago.
• NBJack 2y ago
  I'm wondering about the first example of device drivers. Whether we are talking about Linux or Windows, anything beyond generic device drivers (which push the problem to the hardware) will frequently be vendor specific. You often can't get the full potential of a device until you do so.
• ww520 2y ago
  Haha. This is a great list. I have heard variants of these ideas over the years.
• devjab 2y ago
  Around here people abstract because they are taught to abstract in multiple different CS education programs. For whatever reason they are taught Clean Architecture even though it’s horrible. Then they spend the next 5-10 years learning not to abstract or write 5 line functions which then lead down a chain of 900 function calls. As an external examiner or some or these CS students it always pains me to examine them in a curriculum that I know will lead to dead end job in stagnant small sized companies. It is what it is though. I guess it’s hard to compete with the entire industry of pseudo-science and shitty consultancy when nobody is pushing an “indirection manifesto”. Hilariously Uncle Bob will write off any criticism as “they misunderstood the principles”. He’s correct too, but maybe the principles are simply too vague when we’ve had them for 20 years and our industry has never been more of a mess.
  • bedobi 2y ago
    A lot of Robert C Martins pieces are just variations on his strong belief that ill-defined concepts like "craftsmanship" and "clean code" (which are basically just whatever his opinions are on any given day) is how to reduce defects and increase quality, not built-in safety and better tools, and if you think built-in safety and better tools are desirable, you're not a Real Programmer (tm). I'm not the only one who is skeptical of this toxic, holier-than-thou and dangerous attitude. Removing braces from if statements is a great example of another dangerous thing he advocates for no justifiable reason https://softwareengineering.stackexchange.com/questions/320247/has-can-anyone-challenge-uncle-bob-on-his-love-of-removing-useless-braces/320262 https://softwareengineering.stackexchange.com/questions/3202... Which caused the big OSX/iOS SSL bug in 2014, see https://www.imperialviolet.org/2014/02/22/applebug.html https://www.imperialviolet.org/2014/02/22/applebug.html This link and thread on hackernews is good too https://news.ycombinator.com/item?id=15440848 https://news.ycombinator.com/item?id=15440848 The current state of software safety discussion resembles the state of medical safety discussion 2, 3 decades ago (yeah, software is really really behind time). Back then, too, the thoughts on medical safety also were divided into 2 schools: the professionalism and the process oriented. The former school argues more or less what Uncle Bob argues: blame the damned and * who made the mistakes; be more careful, damn it. But of course, that stupidity fell out of favor. After all, when mistakes kill, people are serious about it. After a while, serious people realize that blaming and clamoring for care backfires big time. That's when they applied, you know, science and statistic to safety. So, tools are upgraded: better color coded medicine boxes, for example, or checklists in surgery. But it's more. They figured out what trainings and processes provide high impacts and do them rigorously. Nurses are taught (I am not kidding you) how to question doctors when weird things happen; identity verification (ever notice why nurses ask your birthday like a thousand times a day?) got extremely serious; etc. My take: give it a few more years, and software, too, probably will follow the same path. We needs more data, though.
• bhouston 2y ago
  I've executed a bunch of these ideas successfully. So it reads a little weird.
  • mdavid626 2y ago
    Either you really know what you are doing, or, you really don't.
• Xcelerate 2y ago
  IDK about anomaly detection being on that list. When I worked at a large tech company, the in-house anomaly detection and root cause analysis capabilities worked like magic compared to other places I've been. When done right, it can be extremely valuable.
  • kitd 2y ago
    Sounds like you hit the 1/10. Which is great and I agree, very rewarding.
  • jldugger 2y ago
    Half of the magic is knowing _when_ to apply it. If you just dump your prometheus timeseries data into an anomaly detection system and ask it to constantly scan for anomalies, you will always find them. As an SRE, I don't actually care about anomalies all that much. For initial alerting, I want phase shift detection. One customer sending a few bad API calls on one specific minute is uninteresting and pretty much inactionable. That same error rate over 10 minutes is more interesting and more likely to be a systems problem I can actually resolve. But the raw AD stream is just too damn noisy for all sorts of reasons. This is why our alerting tools have a `duration` field: any signal above threshold must remain so for multiple observation periods before summoning human inspection. And why health checks have grace periods and retries before killing services. Where anomaly detection works better, IMO, is post-alert analysis. At that point anomalies are welcomed as hypotheses, since the system features complex interactions between components. I've built a couple of dashboards using extremely simple math, like Laplace smoothing and time series correlation that help surface relevant information from the flood of metrics and logs collected. But critically, these tools generally don't use the time domain as their baseline. Usually, I'm comparing a cluster against another one in a different region, or one metric against another, rather than now versus twelve hours ago.
• bcrosby95 2y ago
  Unless you're a large corporation: fighting Conway's Law.
  • hot_gril 2y ago
    Why "unless" and not "especially when"? :)
• slooonz 2y ago
  "Let’s just cache this slow call"
• nottorp 2y ago
  > Let's make it cross-platform. So if making it cross platform "won't work", this is not about what won't work, but what is cheaper? Since the OP kinda mentions gaming, let the customers fiddle with their wine installs and steam decks and spend the time on adding more loot boxes instead?
  • IshKebab 2y ago
    Yeah I don't buy that at all. He says "9 times out of 10" and then gives MS Office as the proof? MS Office is the 10th time out of 10, not the 1-9. I think they absolutely could have made Office cross platform (it doesn't really do anything you can't do in Qt), and also the fact that they forked it and made two entirely separate sets of apps has pretty serious consequences. The feature sets are surprisingly different. E.g. you can add PDFs to documents on Mac; not on Windows. Most software can be made cross platform (as long as you don't explicitly prevent it by using a platform specific GUI toolkit or whatever).
• ghjfrdghibt 2y ago
  The "let's make it cross platform" is what lead me to dart and flutter. These aren't mentioned and for the developers of them is definitely a hard problem. But as far as I'm concerned they're doing a bang up job. I agree with sync though. Hard problem with no simple solution for an idiot like me to do.
• CyberDildonics 2y ago
  A lot of times what people want from a DSL is really just something that doesn't have to compile and/or something to run in a VM. Sometimes people just want a text format for data that could be done with json. Making a new language throws away everything and starts over including debugging, tools, syntax checking, auto completion, documentation etc. It is take way too lightly and just becomes a hassle. You can see this in multiple GUI libraries, where getting the parameters to set up a GUI is really not difficult and is just data through function calls, but it gets made into a separate XML like DSL markup language with it's own quirks and opacity, and that XML is for most people being given using a big string from within the language that they're using. This stuff persists because it sounds easier on paper and it just creates more problems in practice but it takes experience to realize all the you're losing. That's where designers need to be experienced and do what works instead of what will suck in people that don't know any better.
  • adzm 2y ago
    With how easy it is nowadays to embed Lua, and even JavaScript with just a little bit more effort, it's pretty inexcusable to develop your own language for a tiny space. A great example of this is Adobe embracing JavaScript for it's own extension and scripting and expression language, which has allowed it to benefit from an entire ecosystem of JavaScript libraries.
• throwaway63467 2y ago
  Hm I can think of several instances for each point where the ideas work just great. It’s just difficult getting complex stuff to work in general.
• quotemstr 2y ago
  The ideas in the first tweet screenshot: uh, they do sound good and they do work. Your system is full of control loops. (What do you think regulates how many dirty pages the kernel writes back to disk?)
• adzm 2y ago
  > Windows NT is riddled with excess abstractions that were never really used primarily because they were there from the start before there was a real plan to use them. It is? Like what? I know there are some abstractions for cpu architecture etc but they've come in handy for x64 and now arm and others in the past. I know the author certainly has some insight into this but I've never really thought of NT as being riddled with excess abstractions.
  • senderista 2y ago
    It's possible that he's referring to purely internal interfaces?
• liontwist 2y ago
  > Most of the first 25 years of computer science was figuring out how to make things work asynchronously This is not true. Computers focused on single threaded designs before getting thrown into parallelism (need parallelism to run into dining philosopher problems).
  • mrkeen 2y ago
    I'm not sure what counts as the "first 25 years", but Dining Philosophers is from 1965.
• bru3s 2y ago
  [dead]
• liontwist 2y ago
  Anyone have any experience with the control loop one? My uninformed guess is CS people just underestimate the skills and experience to analyze feedback systems and so write it off as a bad technology after a poor implementation. Maybe the real problem is you want to know when your system is maxing out the range you anticipated?
  • avidiax 2y ago
    Control theory is an entire discipline. One problem that constantly comes up is ratcheting or poisoning your own inputs. Let's say you want to block "noisy neighbors" from taking large amounts of resources, while allowing all loads to have bursty use of the full system power. Easy, right? Detect the noisy neighbors and throttle them to a small percentage. Unthrottle when they show some substantial idle time. But now, many of those noisy neighbors can't get out of jail because they will, of course, use nearly 100% of their restricted load, even if they would now be well-behaved and merely bursty. There is also a cascading effect, where you have N related bursty loads. One bursts for too long, gets throttled, and now the load is handled by the remaining N-1 loads. But that makes those loads more likely to get throttled, and so on. Only unthrottling all the loads simultaneously will allow them to return to normal bursty operation.
    • bobnamob 2y ago
      The fun thing about control theory in CS is that (in my experience) it's far easier to build accurate simulations or even run scaled down experiments than it would be in a physical setting. See articles like [1] or any of Marc Brooker's [2] blog for inspiration [1] https://medium.com/yandex/good-retry-bad-retry-an-incident-story-648072d3cee6 https://medium.com/yandex/good-retry-bad-retry-an-incident-s... [2] https://brooker.co.za/blog/2022/04/11/simulation.html https://brooker.co.za/blog/2022/04/11/simulation.html
    • liontwist 2y ago
      > Control theory is an entire discipline. That’s what I said.
• k__ 2y ago
  Most of these things work pretty well. It's just that they don't "just" work. Just because AWS or Google can pull it off doesn't mean it's something anyone can do.
  • sgarland 2y ago
    THIS. And adding to it, most people really do not like being told that they aren’t as clever as AWS or Google.
  • jcims 2y ago
    Yep, I think the title should be “things that are way harder than they seem”.
• asveikau 2y ago
  Just remember that this is the genius who brought Windows 8 and the Windows RT tablet that wouldn't run recompiled Win32 apps.
  • plagiarist 2y ago
    Maybe some experience making large mistakes is how they came up with the list?
    • deleted 2y ago
      [deleted]
    • asveikau 2y ago
      Judging by some of the list I don't think he's learned the right lessons.
• rogerthis 2y ago
  Event sourcing. PS: on mobile, can't expand (issues around versioning, data migration)
  • klysm 2y ago
    Incredibly hard to make work in practice because of the semantic muddiness that results from having events about events etc. You have to think about the domain and its first derivative
• mikewarot 2y ago
  >Let's just add access controls later. This (Computer Security) is a solved problem, kids. One of the lessons learned in the Viet Nam conflict was the need for a computer system that could safely handle multiple levels of classified data. The solution was Multilevel security, included in that was the Bell-LaPadula model. Several actually secure OSs emerged over the decades since, including KeyKOS, CapROS and Eros. I'm hoping that someday I can make Genode, the latest capability based operating system, my daily driver, so I never have to worry about virus scanners again.
  • nostrademons 2y ago
    Computer security is a solved problem for systems with a fully-specified set of capabilities, usage patterns, and legal operations. You follow the Principle of Least Privilege, build in defense in depth, use capabilities, and never expose anything that the user doesn't have a reason to access. The problem is that the constraints this imposes on the system usually do not line up with the constraints that the market will pay for. It's very common for customers to change their mind; decide they need to hack around access protections; add new users with new roles that are some hybrid of current access; ask for new features; not think through who should have access to new features; want to enable serendipity where untrusted users discover new use-cases and new markets for their product; and so on. It's also very common for them to ignore security as a differentiator when making their purchase decision, figuring that if there's a breach, somebody else will pay for it, or they'll be long gone from the company and unable to be blamed for it. So the market ends up bypassing the secure solutions that exist and choosing to buy insecure systems that can offer the features they want right now. When security is absolutely critical, like in military or certain financial applications, it's pretty easy to achieve. There are companies like Galois that specialize in "high assurance systems". But they are expensive for their feature set, and so the general public would rather buy from cheaper and more insecure options.
    • mikewarot 2y ago
      Eventually we're going to have to collectively decide that the operating system is the correct place to enforce capabilities, as is done in mainframe OSs. Memory holing any mention of this solution isn't productive in the long run.
• exabrial 2y ago
  I thought live migrating process state did work...? we have containers that can freeze/resume. Granted, you have to freeze the whole damn OS image, but it does seem to work. All of the others I agree, I've seen them tried so many times in my lifetime, from mainframes to serverless, and nobody gets them right.
  • thraxil 2y ago
    Yeah, and I was live migrating entire VMs between hosts on a Xen cluster 10-15 years ago. It was easy for me to get working, reliable, and I was far from an expert.
• xyzsparetimexyz 2y ago
  I've been writing glsl recently, and I think it's a good example of a DSL that _doesn't__ work. It should have just been C without the standard library, or ideally C++. Not being able to use pointers is especially painful. Hlsl is better and has some nice quality of life features. Rust-gpu is the best approach but basically nobody uses it
• nostrademons 2y ago
  There is a pattern to all of these problems, notably that they are all orthogonal concerns that add constraints to the sequential data-munging programming model that programmers are familiar with. Whenever you add constraints, you add things that future programmers need to think about, for all future development they do on the system. It is very easy to get into a situation where the system is overconstrained and it's impossible to make forward progress without relaxing some of the constraints. Even if it's not impossible, it's going to be slow, as developers need to consider how their new feature interacts with the API/security/synchronization/latency/other-platforms/native-code that the system has already committed to supporting. That's also why it's possible to support all of these attributes. If you make say transparent data synchronization a core value prop of the platform, then all future development supports that first, and you evolve your feature set based on what's possible with that constraint. That feature set might not be exactly what your users want, but it's what you support. Your product appeals to the customers for whom that is their #1 purchase decision.
• qianli_cs 2y ago
  Or let’s just reinvent a new database
  • Aloha 2y ago
    Or worse, lets reinvent the database, but not call it or think of it as a database. Or even worse yet, lets reinvent the filesystem then host it on top of a filesystem.
    • hot_gril 2y ago
      I've heard "it's not a database" multiple times, in reference to whatever custom DB our team inherited. Terrifying.
• ibejoeb 2y ago
  "Let's just sync the data" is the reason why rough days exist, as far as I'm concerned. I've run into so many systems that were designed for "internet scale" or whatever that add queues, event processing, etc., when the natural range is so far below that threshold. These teams are either naive or, at worst, taking advantage of non-engineering management and funneling money toward playing with these problems for the fun of it.
  • mrkeen 2y ago
    I read "just sync the data" as naively doing reads over here and writes over there, and hoping that the two sources of truth don't diverge. Queues and event processing are a necessity to do it right.
    • ibejoeb 2y ago
      That's sort of the point. The operative word is "just." In reality, it adds a huge subsystem that needs more resources than the overall system does. Rather than solve the underlying problem directly, they introduce (more interesting) problems to solve transitively via these complexities.
      • mrkeen 2y ago
        I don't know of any way to replicate data ("solve the underlying problem") aside from a replicated log. It's like git solving the problem without commits, or banking solving the problem without transactions.
        • ibejoeb 2y ago
          Ah, ok, I'm failing to communicate my point. The underlying problem in this scenario is something basic which doesn't require replication/sync at all. I'll use an actual example that fell on me a while back. There was an app that displayed a news feed. The theoretical upper limit of the volume, due to the size of audience, was hundreds of posts per day, while the practical number was dozens. The architecture was: * postgresql as the system of record * firestore as the upstream source for clients * ES for full-text search * client-side store for actual client-side reads * http api for mutations That required three sync systems: pg->firestore, pg->ES, and firestore->local store. Then it needed messages for the async mutations to propagate back to the clients. And then these things require more things to make them work, like data transformers to support the three different formats for each stage. This certainly did not require some giant CQRS system and could have been built entirely on postgres. It was a fractal of code that didn't go toward the actual objective.
• Brian_K_White 2y ago
  I don't understand the load-responsive control loop one. That's a basic and fundamental component in countless systems. The centrifugal governor on a 1800's steam engine or 1900's victrola record player is a load-responsive control loop. All of electronics is a mesh of load-responsive control loops. The automatic transmission in your car...
  • panic 2y ago
    It’s not totally clear, but it could be talking about CPU load in particular, which has some problems as described in https://arxiv.org/abs/2312.10172 https://arxiv.org/abs/2312.10172.
    • Spivak 2y ago
      I've always used connection backlog as the metric for load and it's worked pretty well. Most web servers have it as a number you can expose as a metric. It's not perfect but it's at least a true measure of when servers are behind.
  • lclarkmichalek 2y ago
    The usual issue is the addition of control loops without much understanding of the signals (CPU utilization is a fun one), and the addition of control loops without the consideration of other control loops. For example, you might find that your cross region load balancer gets into a fight with your in-process load shedding, because the load balancer's signals do not account for load shedding (or the way they account for the load shedding is inaccurate). Other issues might be the addition of control loops to optimize service local outcomes, to the detriment of global outcomes. My general take is that you want relatively few control loops, in positions of high leverage.
• LAC-Tech 2y ago
  There are equations/properties for data that syncs nicely and allows mutli-master writes. Ie, if your merges follow a few simple properties, this stuff will work. These are basically what CRDTs are. And I know what you're thinking; "but I'm not writing a mutli user text editor!" or "automerge won't scale!". But CRDTs aren't a library, they're a set of properties. if your whole data system obeys the properties - you have a CRDT.
  • klysm 2y ago
    I’m not really sure what you’re saying. Do you mean you achieve the gaurantees of CRDTs without using CRDTs? What technology are you implicitly referring to? Whenever I hear someone say “sync” data, I instantly get scared. Consensus is fraught with peril and very very very difficult to implement correctly.
    • LAC-Tech 2y ago
      It's more shifting the view point. CRDTs are not just something you 'use'. They are laws a system must be obey to achieve strong eventual consistency. Eventual Consistency: If Node A and Node B have received the same set of events (ie in any order), they will eventually have the same state. Strong eventual consistency: Exactly the same, but replace "eventually" with "immediately". Meaning as long as all nodes get the same events, they'll have the same state, straight away. That's sync that works. As long as your merge algorithm is commutative, associative, and atomic, your sync will work. That's what the CRDT people uncovered. A data structure can obey these laws (the aforementioned CRDT libraries) A database can obey these laws (the original Amazon Dynamo did this, it was a CRDT). Any arbitrary system can obey these laws (there's 1982 paper described this for a distributed file system)
      • klysm 2y ago
        Okay I’m with you, but if I’m not mistaken I believe you have slightly mis-characterized what strong eventual consistency is. > Strong eventual consistency: Exactly the same, but replace "eventually" with "immediately". I believe this isn’t quite correct. I was under the impression that the delivery doesn’t have to be immediate, but rather that any two nodes with the same set of events, regardless of received order, must be in the same state.
• poooka 2y ago
  Disagree with plugin talk. I use plugin system that loads from submodules so I can have same architecture in multiple airgaped networks. Can have implementations load (for that environment) outside of main code base. Nothing wrong w that given complexities of maintaining complex ass configs if everything was shoved into the same artifact.
• fancyfredbot 2y ago
  I've worked on several DSLs, a P2P cache, and a project employing hybrid parallelism. All of them worked. All great fun to create. With one exception these projects were good investments (The P2P cache wasn't necessary so never really paid off). My point is that it's definitely wrong to say these things almost never work. They are complicated but that complexity brings functionality which is hard to achieve in another way. The lesson from the P2P cache example was to be sure you actually need that functionality.
• mlhpdx 2y ago
  Everything on the list I’ve seen done (been hands-on with) very successfully, most of them repeatedly. I’ve also seen many, many more failed (and often pointless) attempts. I don’t have any great answer to why other than the importance (practical value to customers) of it being clear, well understood by leaders and doers, and high. If having an API is core to the value customers will realize then it’s likely a good API will emerge. Etc.
• mjaseem 2y ago
  I feel "Let's just" is doing much of the work here. You can add the phrase in front of any idea and make it sound awful: "Let's just make planes fly themselves" "Let's just put a giant battery in a car instead of an engine" "Let's just make electricity from wind" "Let's just be friends" I would love to see a list like this without such a big asterisk.
  • earnestinger 2y ago
    Let’s just make a list without asterisk.
• levzettelin 2y ago
  Lost me at "DSLs".
• retrocryptid 2y ago
  I think each of the ideas listed: dsls, control loops, being clever, etc. They all have their place, and I wouldn't say they never work. I've had each of these work well in specific situations. But the OP does have a point, they each can introduce more trouble then they're worth. Were I to write this post, I would have titled it something more like "Systems Ideas Yo Really Should Think About Long And Hard Before Doing." But yeah, that might not be enough warning.
• Sparkyte 2y ago
  All rely heavily on one aspect, consistency. If you system engineering can consistently operate without change, innovation or issue, then it is possible that these ideas could exist in a closed environment. However they just don't work when dealing with external factors.
• debarshri 2y ago
  I would like to add building a PaaS/ internal tools that abstract cloud for productivity sake. Bad idea.
• banq 2y ago
  Why do people keep talking about it? Is it because they've fallen into a dialectical trap? No, it's because they lack the awareness that "Context is King."
• dools 2y ago
  An amazing example of a DSL was Butler for Trello before it was acquired and dumbed down to a “no code” tool. It was magical to use. I had started consulting using it but they said I wasn’t allowed to use it anymore so I replicated the functionality as a library to run inside a Google sheet using Google apps script.
• deleted 2y ago
  [deleted]
• ChrisMarshallNY 2y ago
  Happy 2025, folks! My (recent) experience: > Let's make that asynchronous It can be done, but here, there be dragonnes. I just had to make a formerly synchronous load in a recently released app, into an async one, because users with large connection lists (think "friends," in Facebook, but not as "friendly"), were having extremely slow loads. This was a big change. First, I had to swap out an entire SDK that accessed the most important server in the app, because the old one didn't play well, with threads (my bad). That actually went fairly smoothly, because of the abstraction (boo hiss, I guess?) that I had used for the SDK. Took about a day, to have the operation running smoothly. Testing...Testing...Testing... Next, I had to test like crazy, for weeks, on the user-level code, because the new threading brought the beast out in that code. I found all kinds of places, where I had written thread-unstudly code. None of the issues were serious, and many folks would have said "Fuck it. Let's ship," but I'm a bit anal about certain things, and I'm not being paid, anyway... In the end, the conversion was a success (not one complaint —fingers crossed), and we got the results we needed, but it took a hell of a lot of testing (especially monkey testing), and Release Day was a nervous one. The UI basically didn't change at all (except for some loading throbbers on the profile avatars), but under the hood, a lot had changed. Sometimes, it needs to be done, but it's tempting to make it seem easy (which many folks will). I am a rather scarred veteran of "That should be easy," so I went in, with eyes open, and (as noted) had already prepared, with a certain level of abstraction. I figured that the SDK swap would be [relatively] easy, but didn’t bargain for all the little bugs, in the code I thought was already sorted.
• fatih-erikli-cg 2y ago
  [dead]
• cesaref 2y ago
  Let's consider the original list. It starts with DSLs. Are we saying that using SQL 'almost never works'? Shader languages? BPF? If I squint a bit i'd say that all the inference engines are either DSLs or APIs which wrap an underlying DSL, so TensorFlow etc. I could probably come up with similar examples from the rest of the list. If the message is 'system ideas which seem simple but are hard, so you should only use existing off the shelf examples of' then i'd certainly have more sympathy for the statement.
• starlite-5008 2y ago
  [dead]
• ChrisMarshallNY 2y ago
  The Programmer’s Credo: We do what we do. Not because it is easy; but because we thought it would be easy.
• Spiwux 2y ago
  I've implemented several of these ideas, and they worked just fine.
• Guthur 2y ago
  I don't necessarily think the ideas themselves are wrong but rather the devil is in the details. A good example is the DSL one. In reality every significantly complex software system is essentially a DSL, it will have it's own collection of nouns and verbs that constitute the application domain, this is a language and rarely is it understandable to anyone without the domain knowledge. The problems often arise when custom denotational semantics are added that hinder composition. This is overcome by 'not using a DSL' and instead just expose the semantics of your implementation language, but essentially an API is little different from a DSL just with all the baggage and coupling to the underlying implementation details.
• cratermoon 2y ago
  "Microsoft essentially existed because it made its business building cross-platform apps" If your idea of cross-platform is that making it work on all versions of Windows from 3.1 to Vista, sure. Microsoft has always been about capturing users within its ecosystem.