17 ms·

Designing very large JavaScript applications

434 points by kawera 9y ago

simonw 9y ago
I love Malte's take on what makes a senior engineer with regards to API design: > The way I would talk about myself as a senior engineer is that I’d say “I know how I would solve the problem” and because I know how I would solve it I could also teach someone else to do it. And my theory is that the next level is that I can say about myself “I know how others would solve the problem”. Let’s make that a bit more concrete. You make that sentence: “I can anticipate how the API choices that I’m making, or the abstractions that I’m introducing into a project, how they impact how other people would solve a problem.”
- aje403 9y ago
  I think "How confusing is this going to be to the one consuming this" (assuming you're not writing the front end as well) is the more important question to ask
  - abritinthebay 9y ago
    And the one most rarely asking in my experience too
    - silversmith 9y ago
      In my opinion, asking this question is the key benefit of test-driven development. You get to be the consumer, write down your expectations, and then work backwards from yours-as-consumer expectations.
      - sbov 9y ago
        I find the opposite tends to happen in Java. In the pursuit of testability, people make libraries much harder to use and understand.
        purerandomness 9y ago
        Writing good tests is a skill that has to be developed, just like anything else.
      - abritinthebay 9y ago
        It should be, but too often people write bad tests that test whst they wrote and not what it’s supposed to do. Writing good tests is a skill and most developers don’t seem to be as skilled in it as they think. The same concepts power much of Agile too though: XP & Scrum both complement this.
  - deleted 9y ago
    [deleted]
  - dahauns 9y ago
    Isn't that basically what he said? :)
  - joeyespo 9y ago
    This way of thinking was my main motivation behind Grip [0]. Use Readme-driven development and "play" with the API--whether it's in the form of a CLI, a library, or web API-- through usage examples. I've found this to be a wonderful way to get into the heads of dev users before you approach them directly. [0]: https://github.com/joeyespo/grip https://github.com/joeyespo/grip
- jordache 9y ago
  yep.. a very refreshing descriptor of a senior level dev. It is also a very intuitive notion of the behavior of a senior level dev, even if not frequently articulated as such. I'm working with a fellow dev right now, where this notion of empathy is lacking.
- sdrothrock 9y ago
  > I can anticipate how the API choices that I’m making, or the abstractions that I’m introducing into a project, how they impact how other people would solve a problem. I feel like if you take "API" out of this sentence, this is really the benchmark for any senior developer.
- jypepin 9y ago
  Yes I loved it too, and it is super relevant. I've had my fair share of team swaps, both from backend to frontend, and it is always very interesting to see how engineers who have been on a specific team/role for a while tend to lack empathy and forget that other engineers who end up using their APIs don't share the same context and knowledge (or worst, sometimes it just comes from laziness).
danielvinson 9y ago
This was posted multiple times in the last few days https://news.ycombinator.com/item?id=16880991 https://news.ycombinator.com/item?id=16880991 https://news.ycombinator.com/item?id=16844285 https://news.ycombinator.com/item?id=16844285
- tenaciousDaniel 9y ago
  There should be a feature that matches the url of your link against posts within the last 7 days, and alerts you to possible reposts prior to submitting.
  - fiatjaf 9y ago
    There is.
TAForObvReasons 9y ago
> This Javascript framework is not open source. The reason it is not open source is that it kind of came out at the same time as React and I was like “Does the world really need another JS framework to choose from?”. Google already has a few of those–Angular and Polymer–and felt like another one would confuse people, so I just thought we’d just keep it to ourselves. Given the current state of affairs, an open-sourcing of this framework would be welcome.
- leetrout 9y ago
  Do you have some specific reason for this (genuinely curious)? Think we've put ourselves in a corner, so to speak, with the Angular, react, preact, vue, ember, etc etc options and this would offer something compelling from the authors examples?
  - TAForObvReasons 9y ago
    The article discusses one of the major problems: "React component statically depend on their children." There are workarounds, sure, but there are all kinds of pitfalls with the dynamic workarounds, to the extent that I find myself breaking the react model regularly in applications (e.g. utilizing state outside of redux and forceUpdate)
antjanus 9y ago
I'm really appreciative of the mention of the inverted import. Inverting the flow really lets components "register" themselves and thus be fully contained rather than be "required" by another component. It can be powerful but it causes the problem that you never really know what all is registered to another component. Also, yeah, code-splitting isn't easy!
- JasonSage 9y ago
  Reminds me a lot of the League Client architecture that Riot wrote about: https://engineering.riotgames.com/news/under-hood-league-client%E2%80%99s-hextech-ui https://engineering.riotgames.com/news/under-hood-league-cli... Every component of the client is independently built, tested, versioned, and registers itself with the main process on startup. I think didn't grok the nature of "register" vs "require" and the subtle implications of containment vs requirement until now.
- taurath 9y ago
  A simple way to look at this is when you build the application it’ll be from the top down, but when people have to maintain it, they work from the bottom up. You shouldn’t have to know that there is a central registry for a file to work on a file, it should be clear by reading the file.
- camwest 9y ago
  I have a talk about these ideas in the Context of a React Application last year at React.js Conf - https://youtu.be/395ou6k6C6k https://youtu.be/395ou6k6C6k. It might be worth a watch if you’re interested in how to build applications like this on your own.
  - antjanus 9y ago
    thanks! I appreciate it! :)
  - rcanepa 9y ago
    Does anyone knows if this can be done in Vue.js?
randiantech 9y ago
Isnt React actually open source? https://github.com/facebook/react https://github.com/facebook/react
- trillic 9y ago
  Yup, Facebook switched it from a BSD + Patents license to MIT in September of 2017. https://github.com/facebook/react/commit/b765fb25ebc6e53bb8de2496d2828d9d01c2774b https://github.com/facebook/react/commit/b765fb25ebc6e53bb8d...
- pnevares 9y ago
  You were misled by the slide image interrupting the flow of the text: So, I build this JavaScript framework at Google. It is used by Photos, Sites, Plus, Drive, Play, the search engine, all these sites. Some of them are pretty large, you might have used a few of them. [Slide] This Javascript framework is not open source.
  - randiantech 9y ago
    Oh, thanks for the clarification.
    - anonytrary 9y ago
      I found it a tad bit confusing as well, thanks for asking.
komali2 9y ago
I've always disliked the "BIG TEXT CHOPPED-OUT BUT YOU'LL READ IT BELOW AGAIN IN JUST A SECOND" paradigm in magazines and newspapers. Like, I'm reading the article anyway, why suddenly spit some of the words in my face super big? I'm guessing it's for skimmers, catch their eye and bring them into an interesting paragraph that's further down from the top. But this article does it at the beginning... we read the sentence "Hello, I used to build very large JavaScript applications" three times. First as an image, then as a label for the image... and then as the first sentence of the article. Why lol
- nkantar 9y ago
  Those appear to be be slides from the talk that's the original form of this article. This doesn't necessarily justify the first slide's presence at the very beginning, but it might explain it.
- xbryanx 9y ago
  That text (here and in magazines, newspapers) isn't for you. It's for the casual browser who isn't reading the article yet. It's trying to suck you in and/or give you teensy bite of info even if you don't read the article. https://en.wikipedia.org/wiki/Pull_quote https://en.wikipedia.org/wiki/Pull_quote Of course this article format is kinda pushing the limits I agree.
  - saurik 9y ago
    Which is why it doesn't have any place being used on a web page :/.
    - zer00eyz 9y ago
      Why not? I admit to often reading things that are far outside my wheelhouse, or not always topics I assume I will enjoy. If I start one of these and find myself loosing interest, these quotes and call outs can draw me further in or push me right out.
      - deleted 9y ago
Silhouette 9y ago
I agree with much of what is written here, but the emphasis on code splitting seems like it risks hitting the wrong target. My first question is usually what caused even relatively simple web apps to need multi-megabytes of "optimised" JS code in the first place. All too often, the answer is not the application code itself but rather all the direct and transitive dependencies, where the developers yarn-added first and didn't ask questions later. And while obviously that offers some advantages in terms of reusing code, it does also have a very high price in terms of bloating our deliverables, particularly given the still relatively immature tools for optimising JS compared to traditional compiled languages. Maybe we should be looking at that problem first, before we start assuming that any large JS application will necessarily produce some huge monster outputs that need to rely on code splitting and all the overheads that brings just to get acceptable performance?
- BigJono 9y ago
  I think the author still has a point when talking about "very" large applications. His example of all the Google widgets is a good one. Even if you were to hand-craft all of them with optimised code (which I imagine is what they do at Google), it would still likely be too much code for a single package to load in a reasonable amount of time. But I agree completely with you as far as other projects go. If you drop the "very", it should be entirely possible to build a large project without the need for code splitting. Almost nobody is doing anything to the scale of the example talked about in this article, and almost everybody is using code splitting. In my experience, there's two root causes for this. Cause #1 is that barely any front-end developers understand or consider the cost of abstractions. Something like code-splitting is commonly seen as "free" because it takes a couple of lines to implement. The permanently increased complexity of the software and all that extra brain power required to grok it over the development lifetime is never taken into account. At least half of the devs I know are happy just banging in code-splitting at the start of a project with zero thought and I guarantee you they'd read this article and not understand the "sync -> async" example given to explain the downsides of code-splitting. Cause #2 is that devs are too eager to add new dependencies. By definition a popular library is going to be extremely bloated relative to your use case. When you 'yarn add react-redux-awesome-date-picker', what you're usually adding is an overly generic solution with a large amount of customisability and functionality that's not relevant to your use case. My go-to example from my own experience is with rc-collapse. rc-collapse is a container that animates to and from 0 height. It's like 200 lines and has 4 dependencies (god knows how many KB that adds). I've been using my own version that's 50 lines and 0 dependencies in production code for years and never run into a problem with it. I'm sure rc-collapse works around some fancy edge cases or supports older browsers or something, but I'm almost positive the extra weight isn't necessary in 90% of the projects that use it. This is kind of tangential, but another real problem with this mentality is that by implementing my own collapsible container, I learnt some important lessons about React, the DOM, browsers etc. Devs that play npm lego aren't generally going to get that extra knowledge, which will cost time in the long run.
abalone 9y ago
More like designing very long Medium posts. Is there any substance to this "we invented 'enhance', a reverse dependency pattern" thing or is it just old wine in new, javascripty bottles?
- pavlov 9y ago
  “Enhance” seems to be exactly equivalent to what would traditionally be called “plugins”.
- dnomad 9y ago
  This is just basic dependency injection. I agree it is mildly amusing to watch Javascript devs excitedly discover these principles but it does show that these principles are universal. Virtually everything in this talk applies to building very large Java and C++ applications. (And here I don't mean 10mb... rather global systems made up of dozens of services and hundreds of components.) All of this is about modularity -- of configuration, of state, of logic, and of concerns.
  - selestify 9y ago
    Is there a single place to learn about all of the common software engineering principles, as opposed to rediscovering universal principles over and over again?
    - Rapzid 9y ago
      In all those books from the many "What books would you recommend every programmer read?" questions on HN, Stack Overflow, and every other forum developers gather. EDIT: You said single place, sorry.. Safari books online, Codeproject, and MSDN are potential candidates.
    - Silhouette 9y ago
      Short answer: No, but your best bet might be to read old, substantial books, and find other work by the kind of people who read and wrote those books before they were old. Longer answer: Basically, you need to get away from the world of front-end web development, and from the echo chamber blog posts and conference talks by people who have only ever worked in that area. No-one is writing anything large, high-performance, high-reliability or long-lived there yet, so you need to look for experience from people who have had to do those things in other contexts instead. Classic books like Code Complete and The Pragmatic Programmer, early "serious OO" books like Design Patterns, and practical advice books of a similar generation in the Effective <programming language> style all contain a wealth of knowledge and insightful commentary. Of course some of the technical details are quite dated today and some of the specific techniques discussed might no longer be considered good practice a decade or two later, but many of the underlying principles and the discussions around them are as relevant as ever. A bit more recently, there were interesting discussions about a broad range of programming issues in Beautiful Code and the related titles, and there have been some interesting case studies based on open source software too. If you want to read some shorter pieces online, I recommend finding a few authors who work in fields like games or embedded software, where there are often significant performance and reliability constraints to deal with, not to mention hard deadlines that force difficult decisions and compromises. There are some healthy doses of reality in there that you won't find in less demanding environments. Enterprise applications, while often considered bland, can also be large and some of the longest-lived software we develop, and much has been written about organising and maintaining them, including real-world pressures like changing operating environments and large development teams whose members come and go. Perhaps a little ironically, this now includes a fair bit of back-end web development. Some of the writing around this is well worth a read as well, but beware that this part of the industry is plagued by consultants who talk a good talk but don't have much of a track record or other evidence to back up their advice. Approach with caution, particularly anyone who uses words like "agile", "lean", "craftsmanship" and anything else on the buzzword scale.
ngxoogler 9y ago
The framework he is talking about is trash compared to modern JavaScript UI frameworks. There are very few established patterns and each team seems to make up their own rules. Some of the worst code I've ever seen was a result of this framework. The code-splitting is probably the best feature and worked great when this was released, but smart lazy-loading of bundles is much easier in 2018. The most frustrating thing is that the framework developers consistently point to benchmarks created over 3 years ago to show how great it is.
- iovrthoughtthis 9y ago
  This article isn't about the framework though. It's incidental. This seems a very out of place comment.
  - ngxoogler 9y ago
    I get what you're saying and maybe I'm being harsh, but the whole backdrop of the article is this supposedly amazing framework that was built. It undermines a lot of his points that it is poorly used.
    - alexanderhouse 9y ago
      The apps listed in the talk (Photos, Sites, Plus, Drive, Play) all seem to have gotten much better lately? Are you actually sure you are talking about the same framework as the author? Google has so many it can be confusing. If you'd be talking about the closure library and dreaded goog.ui.Component, I'd agree but that isn't it.
- pier25 9y ago
  So what framework is it?
  - applecrazy 9y ago
    The article mentions that it's an in-house proprietary framework used in some live Google products, such as Drive, Photos, Search, and Play.
jordache 9y ago
which modern JS framework offers component level lazy loading? Angular offers only route based lazy loading, which as is limited as the article pointed out.
- djhalon 9y ago
  There are different ways to do it based on the transpiler you use. With Webpack and dynamic imports, you could create a custom loader around the promises returned by the Webpack generation. For React, you can use React Loadable (https://github.com/jamiebuilds/react-loadable https://github.com/jamiebuilds/react-loadable) that provides a Higher Order Component and Server Side rendering.
- llamafarmer 9y ago
  Vue.js does https://vuejs.org/v2/guide/components-dynamic-async.html https://vuejs.org/v2/guide/components-dynamic-async.html
- ngxoogler 9y ago
  It isn't well documented yet but this is possible in Angular (2+). Here's a good tutorial from someone who figured it out. https://blog.angularindepth.com/dynamically-loading-components-with-angular-cli-92a3c69bcd28 https://blog.angularindepth.com/dynamically-loading-componen...
- darkmarmot 9y ago
  My company's framework lazy loads every component automatically by default (you can override this behavior to batches as well).
- ggregoire 9y ago
  https://github.com/jamiebuilds/react-loadable https://github.com/jamiebuilds/react-loadable
- TheCoelacanth 9y ago
  Pretty much trivial to implement in react. class Loader extends React.Component { state = { Component: null }; componentWillMount() { import('./Component').then(Component => { this.setState({ Component }); }); } render() { const { Component } = this.state; if (!Component) { return <div>Loading...</div>; } else { return <Component/>; }; } } Might want to add a few lines of error handling for production use, but that is pretty much all you need.
freecodyx 9y ago
i hate javascript, it got too much hype...
darkmarmot 9y ago
My company also rolled our own framework to handle the case of 'very large web applications'. In our case, every component is a file (which is becoming more typical) -- but it differs in that every file dependency is automatically resolved and downloaded in a lazy, on-demand fashion. With HTTP 2.0, this means you can update a single file without a build process -- and everyone receives the update as a minimal change. Also, all paths are relative or built dynamically using something like C #define, so it's easy to refactor by moving entire directories about.
- tytytytytytytyt 9y ago
  Don't the popular web frameworks handle lazy loading things?
  - darkmarmot 9y ago
    Most of the popular frameworks generate a single JavaScript bundle that comprises an entire application. Some people break these bundles into multiple parts manually in a process called 'code-splitting' in order to do a really chunky kind of lazy load.
    - ngxoogler 9y ago
      Code splitting via webpack is not a manual process. Does your hand rolled framework handle minimization and obfuscation? Probably not because any changes would require a rebuild. Are your assets cached? If they are, I'm curious how you handle the cache busting when a single file gets updated.
      - ec109685 9y ago
        With HTTP/2, you can send files down independently, without creating a large bundle, so each file can be cached. You couldn’t obfuscate function names across files though unless you made the process deterministic based on module name.
G4BB3R 9y ago
I though the article would have just one line saying "Use Elm"
- napsterbr 9y ago
  After 1 year using Elm, we've had a lot of great and not-so-great experiences. For a large app I still think choosing Elm was overall a better choice than javascript, but it's not without downsides. Recently a new dev joined the front-end team. He had js background and no previous functional programming experience. In less than one month he was already shipping production code with confidence on our ~40kloc app. That goes to say that 1) I don't think lack of Elm programmers is a problem if the company is willing to train its employees and 2) I'm sure the front-end lead and the new dev would have a much harder time if we were using javascript. Surely this is our experience. YMMV.
  - always_good 9y ago
    The killer feature of Elm is something I discovered a year after taking a break from Elm: I revisited a production Elm application I hadn't touched in a year, approaching it almost regretfully with "why did I have to use Elm?", certain I'd have no idea what was going on after such a long absence. What actually happened was that I was immediately productive. Without having to credentialize in the code, I was able to stub out new features and push an update. An infrequent experience for me in other languages. I had only a figment of an idea of the code I had to write, but I got started and the compiler helped me the rest of the way. I still don't understand how to confidently organize an Elm app. But that describes my relationship with every front-end Javascript framework as well. I regularly choose poorly between component vs upstream state in React. But what I can do is refactor Elm code at a level that would be downright expensive in Javascript. As far as the sour graping going on in some nooks of the Elm ecosystem, I'm reminded of this post by Rich Hickey: https://www.reddit.com/r/Clojure/comments/73yznc/on_whose_authority/do1olag/ https://www.reddit.com/r/Clojure/comments/73yznc/on_whose_au...
- quikoa 9y ago
  Sadly Elm development is not going so well. There are open pull requests without a response for 2 years. Source: https://www.reddit.com/r/elm/comments/81bo14/do_we_need_to_move_away_from_elm/ https://www.reddit.com/r/elm/comments/81bo14/do_we_need_to_m...
arvinsim 9y ago
I really dislike these frivolous images that can be easily represented by blockquotes. It just wastes data.
auvi 9y ago
what is the metric to consider a JS app as "VERY large" as the author puts it? If SLOC is used as a proxy then how code from libraries (i.e. code that the team didn't write) are considered? Recently when I generated a Ruby-on-Rails app with webpack enabled, it downloaded 8148 JS files in the node_modules folder. 675K SLOC. I thought it was absolute madness.
- Silhouette 9y ago
  The size of node_modules probably isn't a great metric to use if it includes development tools and libraries (and all their transitive dependencies) as well as any libraries you're bundling into the files your user will download and run.
  - girvo 9y ago
    Well, I still have to have those on my computer and in my Docker container. And from a trust, auditing and security perspective it’s pretty horrible, even if multi-stage builds can help With Buble, Flow, and Roll-up, I can get a decent modern environment with as few dependencies as possible. Upsides? Much faster, too. Less disk space. Docker containers build faster, which is better for CI/CD. And it’s easier to understand the codebase and dependency tree, including development tools.
    - Silhouette 9y ago
      Just to clarify, I'm not disagreeing with you that the current dependency-heavy JS world is madness or that the bloat during development activities is unhelpful. I'm just saying that having a large/complicated development environment is probably a separate issue to having a large application for most other practical purposes, but looking at the size of node_modules will often conflate the two.
- cpburns2009 9y ago
  Wow, this must be pretty common. I compiled Bootstrap recently and it download 250 MiB worth of node modules which consisted of 969 dependencies, and 11673 JS files.
RoboHacker111 9y ago
This was a very interesting article! I'd currently consider myself a mid-level engineer, and I have recently been put in a position where I must lead a team of many devs and incorporate all their code. I did understand about half of this article, but it also indicates that there's a lot left for me to learn. I'm curious what kind of blogs or online courses or books I should search for if I wanted to better develop this skill within myself, and to design and architect these sorts of large applications with multiple contributors?
magicmouse 9y ago
It is tragic that the world standardized on Javascript. They are just now discovering some of the module issues that Prof. Wirth solved in his great Modula-2 langauge in 1974. His system allowed separate compilation, and a way to detect when the interface API changed so that it could be cross-checked. I used Modula-2 to great effect on very large projects, and am greatly relieved that the Modula-2 module system has more or less made it into JavaScript.
- tzahola 9y ago
  Don’t worry, they’ll discover types eventually.
  - deleted 9y ago
    [deleted]
  - Joeri 9y ago
    Aren't most large-scale javascript applications using typescript or flow these days? I definitely wouldn't try building a large scale javascript codebase without either of those anymore.
    - tzahola 9y ago
      Frontend developers at our company vetoed TypeScript for being too difficult to comprehend for JS devs.
      - z3t4 9y ago
        You can use JSdoc and get all the advantages from TypeScript/Flow and don't have to change your syntax or add complicated build steps and source maps.
        phillipburch 9y ago
        uhhh no, jsdoc doesn't fail at compile time for type errors, and doesn't have ide support.
polskibus 9y ago
Does Google really use React? I got that impression from skimming the article. That'd be interesting given they develop at least two competing tech like polymer and angular.
- pknopf 9y ago
  You can't really say "does Google use" because Google isn't monolith. Big companies usually keep teams cross-functional, leaving teams with the discretion to use the correct tool for the job/team.
  - k__ 9y ago
    One of those teams can use React and then Google would use React. Nobody cares how much of Google uses React...
- Joeri 9y ago
  The talk mentions they use an in-house framework, but react was probably substituted in to make it more approachable for the audience.
  - deleted 9y ago
    [deleted]
xab9 9y ago
It's interesting how in 2018 web development === react. React this, react that. Anyway, I think beginner empathy is when you anticipate the usage of your API, or modules. Pro empathy is when you talk about it and share your feelings and needs and then listen to those other users' feelings and needs. Fractured pluggable routing may be a good idea, but you know, at a certain point a centralized route definition may be a good idea too. There's no one solution fits all strategy. Major routes === modules (oh boy, enhance). We're all MVC (with islands of goodness) again. Frontend is the new backend, but it's fronted too. Doing frontend is hard. Dependency injection. Malte talks about the google search result page as an example for ultra complexity - Angular tries to sell itself for very high complexity ("enterprise") projects, yet I would eat my hat if Google plus or even the search result page would be better in Angular. The frontend landscape is fractured and everyone tries to sell you his or her snake oil. Base bundles: yes, they suck. That's why common chunk creation is delegated to the bundler (like webpack) and then you believe that if a datepicker used more than three times it's okay to load it all the time. Or you can be a smartass and do require-ensure like logic all yourself (maybe it's not your fault, maybe just a very clever manager made you do it). And then you will feel very smart and good and after a year or so you realize how it was stupid and then someone comes along and relegates it back to the bundler. Large applications in javascript are like Frankenstein's monster, but it's better to have a monster than a pile of dead flesh jolted by electricity with each and every release. My two cents.
- stratigos 9y ago
  > It's interesting how in 2018 web development === react. React this, react that. Its quite popular for sure, but after just doing a job search for about 6 months, I came across a surprising number of firms using ng, ember, and even vue in a couple places. modern ruby on rails with ujs is still hugely popular
  - xab9 9y ago
    You're lucky. I've been through a jub-hunt cycle, my experiences here are: lots of legacy stuff (mostly angular1), tons of react, some nu-angular and one vue (which is going to be my next job, not just because I love vue, but also because they understood the tech stuff I had been talking about, unlike many others).

434 points by kawera 9y ago

I love Malte's take on what makes a senior engineer with regards to API design: > The way I would talk about myself as a senior engineer is that I’d say “I know how I would solve the problem” and because I know how I would solve it I could also teach someone else to do it. And my theory is that the next level is that I can say about myself “I know how others would solve the problem”. Let’s make that a bit more concrete. You make that sentence: “I can anticipate how the API choices that I’m making, or the abstractions that I’m introducing into a project, how they impact how other people would solve a problem.”

aje403 9y ago
I think "How confusing is this going to be to the one consuming this" (assuming you're not writing the front end as well) is the more important question to ask
- abritinthebay 9y ago
  And the one most rarely asking in my experience too
  - silversmith 9y ago
    In my opinion, asking this question is the key benefit of test-driven development. You get to be the consumer, write down your expectations, and then work backwards from yours-as-consumer expectations.
    - sbov 9y ago
      I find the opposite tends to happen in Java. In the pursuit of testability, people make libraries much harder to use and understand.
      - purerandomness 9y ago
        Writing good tests is a skill that has to be developed, just like anything else.
    - abritinthebay 9y ago
      It should be, but too often people write bad tests that test whst they wrote and not what it’s supposed to do. Writing good tests is a skill and most developers don’t seem to be as skilled in it as they think. The same concepts power much of Agile too though: XP & Scrum both complement this.
- deleted 9y ago
  [deleted]
- dahauns 9y ago
  Isn't that basically what he said? :)
- joeyespo 9y ago
  This way of thinking was my main motivation behind Grip [0]. Use Readme-driven development and "play" with the API--whether it's in the form of a CLI, a library, or web API-- through usage examples. I've found this to be a wonderful way to get into the heads of dev users before you approach them directly. [0]: https://github.com/joeyespo/grip https://github.com/joeyespo/grip
jordache 9y ago
yep.. a very refreshing descriptor of a senior level dev. It is also a very intuitive notion of the behavior of a senior level dev, even if not frequently articulated as such. I'm working with a fellow dev right now, where this notion of empathy is lacking.
sdrothrock 9y ago
> I can anticipate how the API choices that I’m making, or the abstractions that I’m introducing into a project, how they impact how other people would solve a problem. I feel like if you take "API" out of this sentence, this is really the benchmark for any senior developer.
jypepin 9y ago
Yes I loved it too, and it is super relevant. I've had my fair share of team swaps, both from backend to frontend, and it is always very interesting to see how engineers who have been on a specific team/role for a while tend to lack empathy and forget that other engineers who end up using their APIs don't share the same context and knowledge (or worst, sometimes it just comes from laziness).

danielvinson 9y ago

This was posted multiple times in the last few days https://news.ycombinator.com/item?id=16880991 https://news.ycombinator.com/item?id=16880991 https://news.ycombinator.com/item?id=16844285 https://news.ycombinator.com/item?id=16844285

tenaciousDaniel 9y ago
There should be a feature that matches the url of your link against posts within the last 7 days, and alerts you to possible reposts prior to submitting.
- fiatjaf 9y ago
  There is.

TAForObvReasons 9y ago

> This Javascript framework is not open source. The reason it is not open source is that it kind of came out at the same time as React and I was like “Does the world really need another JS framework to choose from?”. Google already has a few of those–Angular and Polymer–and felt like another one would confuse people, so I just thought we’d just keep it to ourselves. Given the current state of affairs, an open-sourcing of this framework would be welcome.

leetrout 9y ago
Do you have some specific reason for this (genuinely curious)? Think we've put ourselves in a corner, so to speak, with the Angular, react, preact, vue, ember, etc etc options and this would offer something compelling from the authors examples?
- TAForObvReasons 9y ago
  The article discusses one of the major problems: "React component statically depend on their children." There are workarounds, sure, but there are all kinds of pitfalls with the dynamic workarounds, to the extent that I find myself breaking the react model regularly in applications (e.g. utilizing state outside of redux and forceUpdate)

antjanus 9y ago

I'm really appreciative of the mention of the inverted import. Inverting the flow really lets components "register" themselves and thus be fully contained rather than be "required" by another component. It can be powerful but it causes the problem that you never really know what all is registered to another component. Also, yeah, code-splitting isn't easy!

JasonSage 9y ago
Reminds me a lot of the League Client architecture that Riot wrote about: https://engineering.riotgames.com/news/under-hood-league-client%E2%80%99s-hextech-ui https://engineering.riotgames.com/news/under-hood-league-cli... Every component of the client is independently built, tested, versioned, and registers itself with the main process on startup. I think didn't grok the nature of "register" vs "require" and the subtle implications of containment vs requirement until now.
taurath 9y ago
A simple way to look at this is when you build the application it’ll be from the top down, but when people have to maintain it, they work from the bottom up. You shouldn’t have to know that there is a central registry for a file to work on a file, it should be clear by reading the file.
camwest 9y ago
I have a talk about these ideas in the Context of a React Application last year at React.js Conf - https://youtu.be/395ou6k6C6k https://youtu.be/395ou6k6C6k. It might be worth a watch if you’re interested in how to build applications like this on your own.
- antjanus 9y ago
  thanks! I appreciate it! :)
- rcanepa 9y ago
  Does anyone knows if this can be done in Vue.js?

randiantech 9y ago

Isnt React actually open source? https://github.com/facebook/react https://github.com/facebook/react

trillic 9y ago
Yup, Facebook switched it from a BSD + Patents license to MIT in September of 2017. https://github.com/facebook/react/commit/b765fb25ebc6e53bb8de2496d2828d9d01c2774b https://github.com/facebook/react/commit/b765fb25ebc6e53bb8d...
pnevares 9y ago
You were misled by the slide image interrupting the flow of the text: So, I build this JavaScript framework at Google. It is used by Photos, Sites, Plus, Drive, Play, the search engine, all these sites. Some of them are pretty large, you might have used a few of them. [Slide] This Javascript framework is not open source.
- randiantech 9y ago
  Oh, thanks for the clarification.
  - anonytrary 9y ago
    I found it a tad bit confusing as well, thanks for asking.

komali2 9y ago

I've always disliked the "BIG TEXT CHOPPED-OUT BUT YOU'LL READ IT BELOW AGAIN IN JUST A SECOND" paradigm in magazines and newspapers. Like, I'm reading the article anyway, why suddenly spit some of the words in my face super big? I'm guessing it's for skimmers, catch their eye and bring them into an interesting paragraph that's further down from the top. But this article does it at the beginning... we read the sentence "Hello, I used to build very large JavaScript applications" three times. First as an image, then as a label for the image... and then as the first sentence of the article. Why lol

nkantar 9y ago
Those appear to be be slides from the talk that's the original form of this article. This doesn't necessarily justify the first slide's presence at the very beginning, but it might explain it.
xbryanx 9y ago
That text (here and in magazines, newspapers) isn't for you. It's for the casual browser who isn't reading the article yet. It's trying to suck you in and/or give you teensy bite of info even if you don't read the article. https://en.wikipedia.org/wiki/Pull_quote https://en.wikipedia.org/wiki/Pull_quote Of course this article format is kinda pushing the limits I agree.
- saurik 9y ago
  Which is why it doesn't have any place being used on a web page :/.
  - zer00eyz 9y ago
    Why not? I admit to often reading things that are far outside my wheelhouse, or not always topics I assume I will enjoy. If I start one of these and find myself loosing interest, these quotes and call outs can draw me further in or push me right out.
    - deleted 9y ago

Silhouette 9y ago

I agree with much of what is written here, but the emphasis on code splitting seems like it risks hitting the wrong target. My first question is usually what caused even relatively simple web apps to need multi-megabytes of "optimised" JS code in the first place. All too often, the answer is not the application code itself but rather all the direct and transitive dependencies, where the developers yarn-added first and didn't ask questions later. And while obviously that offers some advantages in terms of reusing code, it does also have a very high price in terms of bloating our deliverables, particularly given the still relatively immature tools for optimising JS compared to traditional compiled languages. Maybe we should be looking at that problem first, before we start assuming that any large JS application will necessarily produce some huge monster outputs that need to rely on code splitting and all the overheads that brings just to get acceptable performance?

BigJono 9y ago
I think the author still has a point when talking about "very" large applications. His example of all the Google widgets is a good one. Even if you were to hand-craft all of them with optimised code (which I imagine is what they do at Google), it would still likely be too much code for a single package to load in a reasonable amount of time. But I agree completely with you as far as other projects go. If you drop the "very", it should be entirely possible to build a large project without the need for code splitting. Almost nobody is doing anything to the scale of the example talked about in this article, and almost everybody is using code splitting. In my experience, there's two root causes for this. Cause #1 is that barely any front-end developers understand or consider the cost of abstractions. Something like code-splitting is commonly seen as "free" because it takes a couple of lines to implement. The permanently increased complexity of the software and all that extra brain power required to grok it over the development lifetime is never taken into account. At least half of the devs I know are happy just banging in code-splitting at the start of a project with zero thought and I guarantee you they'd read this article and not understand the "sync -> async" example given to explain the downsides of code-splitting. Cause #2 is that devs are too eager to add new dependencies. By definition a popular library is going to be extremely bloated relative to your use case. When you 'yarn add react-redux-awesome-date-picker', what you're usually adding is an overly generic solution with a large amount of customisability and functionality that's not relevant to your use case. My go-to example from my own experience is with rc-collapse. rc-collapse is a container that animates to and from 0 height. It's like 200 lines and has 4 dependencies (god knows how many KB that adds). I've been using my own version that's 50 lines and 0 dependencies in production code for years and never run into a problem with it. I'm sure rc-collapse works around some fancy edge cases or supports older browsers or something, but I'm almost positive the extra weight isn't necessary in 90% of the projects that use it. This is kind of tangential, but another real problem with this mentality is that by implementing my own collapsible container, I learnt some important lessons about React, the DOM, browsers etc. Devs that play npm lego aren't generally going to get that extra knowledge, which will cost time in the long run.

abalone 9y ago

More like designing very long Medium posts. Is there any substance to this "we invented 'enhance', a reverse dependency pattern" thing or is it just old wine in new, javascripty bottles?

pavlov 9y ago
“Enhance” seems to be exactly equivalent to what would traditionally be called “plugins”.
dnomad 9y ago
This is just basic dependency injection. I agree it is mildly amusing to watch Javascript devs excitedly discover these principles but it does show that these principles are universal. Virtually everything in this talk applies to building very large Java and C++ applications. (And here I don't mean 10mb... rather global systems made up of dozens of services and hundreds of components.) All of this is about modularity -- of configuration, of state, of logic, and of concerns.
- selestify 9y ago
  Is there a single place to learn about all of the common software engineering principles, as opposed to rediscovering universal principles over and over again?
  - Rapzid 9y ago
    In all those books from the many "What books would you recommend every programmer read?" questions on HN, Stack Overflow, and every other forum developers gather. EDIT: You said single place, sorry.. Safari books online, Codeproject, and MSDN are potential candidates.
  - Silhouette 9y ago
    Short answer: No, but your best bet might be to read old, substantial books, and find other work by the kind of people who read and wrote those books before they were old. Longer answer: Basically, you need to get away from the world of front-end web development, and from the echo chamber blog posts and conference talks by people who have only ever worked in that area. No-one is writing anything large, high-performance, high-reliability or long-lived there yet, so you need to look for experience from people who have had to do those things in other contexts instead. Classic books like Code Complete and The Pragmatic Programmer, early "serious OO" books like Design Patterns, and practical advice books of a similar generation in the Effective <programming language> style all contain a wealth of knowledge and insightful commentary. Of course some of the technical details are quite dated today and some of the specific techniques discussed might no longer be considered good practice a decade or two later, but many of the underlying principles and the discussions around them are as relevant as ever. A bit more recently, there were interesting discussions about a broad range of programming issues in Beautiful Code and the related titles, and there have been some interesting case studies based on open source software too. If you want to read some shorter pieces online, I recommend finding a few authors who work in fields like games or embedded software, where there are often significant performance and reliability constraints to deal with, not to mention hard deadlines that force difficult decisions and compromises. There are some healthy doses of reality in there that you won't find in less demanding environments. Enterprise applications, while often considered bland, can also be large and some of the longest-lived software we develop, and much has been written about organising and maintaining them, including real-world pressures like changing operating environments and large development teams whose members come and go. Perhaps a little ironically, this now includes a fair bit of back-end web development. Some of the writing around this is well worth a read as well, but beware that this part of the industry is plagued by consultants who talk a good talk but don't have much of a track record or other evidence to back up their advice. Approach with caution, particularly anyone who uses words like "agile", "lean", "craftsmanship" and anything else on the buzzword scale.

ngxoogler 9y ago

The framework he is talking about is trash compared to modern JavaScript UI frameworks. There are very few established patterns and each team seems to make up their own rules. Some of the worst code I've ever seen was a result of this framework. The code-splitting is probably the best feature and worked great when this was released, but smart lazy-loading of bundles is much easier in 2018. The most frustrating thing is that the framework developers consistently point to benchmarks created over 3 years ago to show how great it is.

iovrthoughtthis 9y ago
This article isn't about the framework though. It's incidental. This seems a very out of place comment.
- ngxoogler 9y ago
  I get what you're saying and maybe I'm being harsh, but the whole backdrop of the article is this supposedly amazing framework that was built. It undermines a lot of his points that it is poorly used.
  - alexanderhouse 9y ago
    The apps listed in the talk (Photos, Sites, Plus, Drive, Play) all seem to have gotten much better lately? Are you actually sure you are talking about the same framework as the author? Google has so many it can be confusing. If you'd be talking about the closure library and dreaded goog.ui.Component, I'd agree but that isn't it.
pier25 9y ago
So what framework is it?
- applecrazy 9y ago
  The article mentions that it's an in-house proprietary framework used in some live Google products, such as Drive, Photos, Search, and Play.

jordache 9y ago

which modern JS framework offers component level lazy loading? Angular offers only route based lazy loading, which as is limited as the article pointed out.

djhalon 9y ago
There are different ways to do it based on the transpiler you use. With Webpack and dynamic imports, you could create a custom loader around the promises returned by the Webpack generation. For React, you can use React Loadable (https://github.com/jamiebuilds/react-loadable https://github.com/jamiebuilds/react-loadable) that provides a Higher Order Component and Server Side rendering.
llamafarmer 9y ago
Vue.js does https://vuejs.org/v2/guide/components-dynamic-async.html https://vuejs.org/v2/guide/components-dynamic-async.html
ngxoogler 9y ago
It isn't well documented yet but this is possible in Angular (2+). Here's a good tutorial from someone who figured it out. https://blog.angularindepth.com/dynamically-loading-components-with-angular-cli-92a3c69bcd28 https://blog.angularindepth.com/dynamically-loading-componen...
darkmarmot 9y ago
My company's framework lazy loads every component automatically by default (you can override this behavior to batches as well).
ggregoire 9y ago
https://github.com/jamiebuilds/react-loadable https://github.com/jamiebuilds/react-loadable
TheCoelacanth 9y ago
Pretty much trivial to implement in react. class Loader extends React.Component { state = { Component: null }; componentWillMount() { import('./Component').then(Component => { this.setState({ Component }); }); } render() { const { Component } = this.state; if (!Component) { return <div>Loading...</div>; } else { return <Component/>; }; } } Might want to add a few lines of error handling for production use, but that is pretty much all you need.

freecodyx 9y ago

i hate javascript, it got too much hype...

darkmarmot 9y ago

My company also rolled our own framework to handle the case of 'very large web applications'. In our case, every component is a file (which is becoming more typical) -- but it differs in that every file dependency is automatically resolved and downloaded in a lazy, on-demand fashion. With HTTP 2.0, this means you can update a single file without a build process -- and everyone receives the update as a minimal change. Also, all paths are relative or built dynamically using something like C #define, so it's easy to refactor by moving entire directories about.

tytytytytytytyt 9y ago
Don't the popular web frameworks handle lazy loading things?
- darkmarmot 9y ago
  Most of the popular frameworks generate a single JavaScript bundle that comprises an entire application. Some people break these bundles into multiple parts manually in a process called 'code-splitting' in order to do a really chunky kind of lazy load.
  - ngxoogler 9y ago
    Code splitting via webpack is not a manual process. Does your hand rolled framework handle minimization and obfuscation? Probably not because any changes would require a rebuild. Are your assets cached? If they are, I'm curious how you handle the cache busting when a single file gets updated.
    - ec109685 9y ago
      With HTTP/2, you can send files down independently, without creating a large bundle, so each file can be cached. You couldn’t obfuscate function names across files though unless you made the process deterministic based on module name.

G4BB3R 9y ago

I though the article would have just one line saying "Use Elm"

napsterbr 9y ago
After 1 year using Elm, we've had a lot of great and not-so-great experiences. For a large app I still think choosing Elm was overall a better choice than javascript, but it's not without downsides. Recently a new dev joined the front-end team. He had js background and no previous functional programming experience. In less than one month he was already shipping production code with confidence on our ~40kloc app. That goes to say that 1) I don't think lack of Elm programmers is a problem if the company is willing to train its employees and 2) I'm sure the front-end lead and the new dev would have a much harder time if we were using javascript. Surely this is our experience. YMMV.
- always_good 9y ago
  The killer feature of Elm is something I discovered a year after taking a break from Elm: I revisited a production Elm application I hadn't touched in a year, approaching it almost regretfully with "why did I have to use Elm?", certain I'd have no idea what was going on after such a long absence. What actually happened was that I was immediately productive. Without having to credentialize in the code, I was able to stub out new features and push an update. An infrequent experience for me in other languages. I had only a figment of an idea of the code I had to write, but I got started and the compiler helped me the rest of the way. I still don't understand how to confidently organize an Elm app. But that describes my relationship with every front-end Javascript framework as well. I regularly choose poorly between component vs upstream state in React. But what I can do is refactor Elm code at a level that would be downright expensive in Javascript. As far as the sour graping going on in some nooks of the Elm ecosystem, I'm reminded of this post by Rich Hickey: https://www.reddit.com/r/Clojure/comments/73yznc/on_whose_authority/do1olag/ https://www.reddit.com/r/Clojure/comments/73yznc/on_whose_au...
quikoa 9y ago
Sadly Elm development is not going so well. There are open pull requests without a response for 2 years. Source: https://www.reddit.com/r/elm/comments/81bo14/do_we_need_to_move_away_from_elm/ https://www.reddit.com/r/elm/comments/81bo14/do_we_need_to_m...

arvinsim 9y ago

I really dislike these frivolous images that can be easily represented by blockquotes. It just wastes data.

auvi 9y ago

what is the metric to consider a JS app as "VERY large" as the author puts it? If SLOC is used as a proxy then how code from libraries (i.e. code that the team didn't write) are considered? Recently when I generated a Ruby-on-Rails app with webpack enabled, it downloaded 8148 JS files in the node_modules folder. 675K SLOC. I thought it was absolute madness.

Silhouette 9y ago
The size of node_modules probably isn't a great metric to use if it includes development tools and libraries (and all their transitive dependencies) as well as any libraries you're bundling into the files your user will download and run.
- girvo 9y ago
  Well, I still have to have those on my computer and in my Docker container. And from a trust, auditing and security perspective it’s pretty horrible, even if multi-stage builds can help With Buble, Flow, and Roll-up, I can get a decent modern environment with as few dependencies as possible. Upsides? Much faster, too. Less disk space. Docker containers build faster, which is better for CI/CD. And it’s easier to understand the codebase and dependency tree, including development tools.
  - Silhouette 9y ago
    Just to clarify, I'm not disagreeing with you that the current dependency-heavy JS world is madness or that the bloat during development activities is unhelpful. I'm just saying that having a large/complicated development environment is probably a separate issue to having a large application for most other practical purposes, but looking at the size of node_modules will often conflate the two.
cpburns2009 9y ago
Wow, this must be pretty common. I compiled Bootstrap recently and it download 250 MiB worth of node modules which consisted of 969 dependencies, and 11673 JS files.

RoboHacker111 9y ago

This was a very interesting article! I'd currently consider myself a mid-level engineer, and I have recently been put in a position where I must lead a team of many devs and incorporate all their code. I did understand about half of this article, but it also indicates that there's a lot left for me to learn. I'm curious what kind of blogs or online courses or books I should search for if I wanted to better develop this skill within myself, and to design and architect these sorts of large applications with multiple contributors?

magicmouse 9y ago

It is tragic that the world standardized on Javascript. They are just now discovering some of the module issues that Prof. Wirth solved in his great Modula-2 langauge in 1974. His system allowed separate compilation, and a way to detect when the interface API changed so that it could be cross-checked. I used Modula-2 to great effect on very large projects, and am greatly relieved that the Modula-2 module system has more or less made it into JavaScript.

tzahola 9y ago
Don’t worry, they’ll discover types eventually.
- deleted 9y ago
  [deleted]
- Joeri 9y ago
  Aren't most large-scale javascript applications using typescript or flow these days? I definitely wouldn't try building a large scale javascript codebase without either of those anymore.
  - tzahola 9y ago
    Frontend developers at our company vetoed TypeScript for being too difficult to comprehend for JS devs.
    - z3t4 9y ago
      You can use JSdoc and get all the advantages from TypeScript/Flow and don't have to change your syntax or add complicated build steps and source maps.
      - phillipburch 9y ago
        uhhh no, jsdoc doesn't fail at compile time for type errors, and doesn't have ide support.

polskibus 9y ago

Does Google really use React? I got that impression from skimming the article. That'd be interesting given they develop at least two competing tech like polymer and angular.

pknopf 9y ago
You can't really say "does Google use" because Google isn't monolith. Big companies usually keep teams cross-functional, leaving teams with the discretion to use the correct tool for the job/team.
- k__ 9y ago
  One of those teams can use React and then Google would use React. Nobody cares how much of Google uses React...
Joeri 9y ago
The talk mentions they use an in-house framework, but react was probably substituted in to make it more approachable for the audience.
- deleted 9y ago
  [deleted]

xab9 9y ago

It's interesting how in 2018 web development === react. React this, react that. Anyway, I think beginner empathy is when you anticipate the usage of your API, or modules. Pro empathy is when you talk about it and share your feelings and needs and then listen to those other users' feelings and needs. Fractured pluggable routing may be a good idea, but you know, at a certain point a centralized route definition may be a good idea too. There's no one solution fits all strategy. Major routes === modules (oh boy, enhance). We're all MVC (with islands of goodness) again. Frontend is the new backend, but it's fronted too. Doing frontend is hard. Dependency injection. Malte talks about the google search result page as an example for ultra complexity - Angular tries to sell itself for very high complexity ("enterprise") projects, yet I would eat my hat if Google plus or even the search result page would be better in Angular. The frontend landscape is fractured and everyone tries to sell you his or her snake oil. Base bundles: yes, they suck. That's why common chunk creation is delegated to the bundler (like webpack) and then you believe that if a datepicker used more than three times it's okay to load it all the time. Or you can be a smartass and do require-ensure like logic all yourself (maybe it's not your fault, maybe just a very clever manager made you do it). And then you will feel very smart and good and after a year or so you realize how it was stupid and then someone comes along and relegates it back to the bundler. Large applications in javascript are like Frankenstein's monster, but it's better to have a monster than a pile of dead flesh jolted by electricity with each and every release. My two cents.

stratigos 9y ago
> It's interesting how in 2018 web development === react. React this, react that. Its quite popular for sure, but after just doing a job search for about 6 months, I came across a surprising number of firms using ng, ember, and even vue in a couple places. modern ruby on rails with ujs is still hugely popular
- xab9 9y ago
  You're lucky. I've been through a jub-hunt cycle, my experiences here are: lots of legacy stuff (mostly angular1), tons of react, some nu-angular and one vue (which is going to be my next job, not just because I love vue, but also because they understood the tech stuff I had been talking about, unlike many others).