24 ms·
Size of node_modules folder when installing the top 100 packages
- vnglst 7y agoI thought it might be (mildly) interesting to keep track of how the npm node_modules folder grows over time. So I made this website. Source code: https://github.com/vnglst/size-of-npm/ https://github.com/vnglst/size-of-npm/
- pixelbath 7y agoI find this post a bit ironic considering you built a 1.1 MB React app to display an image and a bit of text (and the image itself isn't even included in the repository).
- zamadatix 7y agoDear lord it even loads a service worker so it can cache the app.
- asdfaoeu 7y agoI mean that's smaller than the average of the top 100 modules.
- Frost1x 7y agoI don't find it ironic anymore, I find it the state of affairs in modern development environments. I'm always surprised how quickly people jump to using massive scaffolding, systems, or systems of systems to accomplish things that need about two orders of magnitude less complexity. Then again, it seems people that develop these monstrosities are never the ones to maintain them.
- jgalt212 7y agobuilt in job security (if not for the original dev), but for those that come after the dev0.
- sidlls 7y agoDon't underestimate managers who "encourage" unnecessary complexity to boost their own profile. Their direct reports have to "design" and then implement and maintain them. Then the manager gets a higher profile for managing teams that build and design "critical", complex systems. There is a lot of blame to go around in this industry--it isn't just individual engineers acting out of ignorance or resume building.
- ttty 7y agoYou're missing out the bug picture. All my projects started small. But day after day, you need a be feature and a new thing. It will grow. At that point you need to add these tools yourself, manually. I rather have a standard baseline to be up to speed and don't get bothered with this on each app.
- mark_and_sweep 7y ago> You're missing out the bug picture. Is this a Freudian slip? The amount of bugs in "modern web apps" seems rather huge indeed. > I rather have a standard baseline to be up to speed To me, the baseline for any kind of website or web app is, you know, HTML. When authoring static HTML or rendering it dynamically on the server, I'm up to speed instantly.
- rhizome 7y agoThe graybeards called that "creeping featuritis."
- Dockson 7y agoWhere are you getting 1.1 MB from? I'm getting 223 KB over the wire, which includes a 100 KB image.
- pixelbath 7y agoI cloned the repository for the site. Over-the-wire, though, I do get around 227 KB.
- city41 7y agoIt looks like the top 100 packages are stored in size/package.json? How do you calculate what the top 100 packages are and how do you update the package.json?
- capableweb 7y agoCool! You could even scrape the history of this. If you take a date that you want to scrape, then you resolve the top-level packages with their release date that you'll find next to the version in the package.json. Use the latest found version that is before the date you're trying to scrape for.
- viraptor 7y agoI can't find the code that does the actual package installation. Does it do deduplication / linking of modules, or does it install full tree?
- deleted 7y ago[deleted]
- leipert 7y agoIt runs on circle CI (look at the .circleci folder). Half the logic is in there, the other half in ./size. I am on mobile so I cannot check the methodology, but it seems like the CI is committing the stats back to the repo
- nkrisc 7y agoI used create-react-app recently to get up and running with React to learn more about it and try using it. It works as advertised. However I also noticed that I now have ~1500 dependencies in node_modules and it weighs in at about 214MB. I thought that was mildly interesting. Magic ain't cheap. EDIT: Silly me, I was looking at the audit number. It did seem ridiculously big.
- vemv 7y ago> 906,380 literally? (if not, what's the actual figure?)
- zamadatix 7y agohttps://npm.anvaka.com/#/view/2d/create-react-app https://npm.anvaka.com/#/view/2d/create-react-app Gives me 91
- silverwind 7y agocreate-react-app is just the CLI to bootstrap the project, you want to look at react-scripts instead: https://npm.anvaka.com/#/view/2d/react-scripts https://npm.anvaka.com/#/view/2d/react-scripts
- pier25 7y agoPersonally after having been bitten by some scaffolding tool I now configure everything from scratch. Webpack is not really as complicated as it seems and I prefer having a tailored suit instead of some generic pret a porter solution. Whenever I start working with a framework I create my own starter kit which I reuse, I don't start from scratch on every project.
- arvinsim 7y agoI have used CRA for so long that I have forgotten how to manually setup Webpack. Will try to relearn again. Hopefully, the new versions are not that bad.
- bransonf 7y agoDoes anyone know what that late November spike is all about?
- atomwaffel 7y agoSince this tracks the combined size of the top 100 npm packages, my guess is that those spikes are caused by packages entering and leaving the list.
- bransonf 7y agoThat would make sense. Probably the number 100ish spot got replaced by a much larger package. My initial thought was a widespread security patch or something.
- smabie 7y agoIt seems to me that this website could be a completely static site that is automatically regenerated once a day with a new graph. Instead you seem to be succumbing to the exact same problem you are highlighting. This a good example of how the culture of software engineering is moving in the wrong direction. We need to all make a stand and say no more to this culture. Of course, that’s made difficult by the fact it’s against our own collective interest. At this rate even the worst developers will be able to be employed managing some gargantuan and wholly unnecessary software stock. Not to knock on you to much, but this culture of bloat reminds about the classic “Evolution of a Programmer” joke: https://www.smart-jokes.org/programmer-evolution.html https://www.smart-jokes.org/programmer-evolution.html Let’s all try and be like the Guru Hacker, not the Seasoned Professional!
- lucasmullens 7y ago> We need to all make a stand and say no more to this culture. I don't think the path forward is to bash all websites that use a common npm library.
- mstade 7y agoUnfortunately the culture I’ve experienced in the past few years has been essentially that, but for anything that does not use a common npm library. “Why is this not done with React/Vue/Typescript/Angular/<insert favorite thing here>?” is a question that’s posed less as a question and more as a condescension in many cases. The rebuttal “because not using XYZ works just fine” is not often accepted, true as it may be, followed by bashing the non believer in question into submission. I’m with you though, I don’t think the bashing is necessary, in either direction. It just breeds contempt.
- ampersandy 7y agoIt took more time for me to read your complaint about "bloat" than it took to load the site, read it all, and come to the comments. This type of elitist posturing is more corrosive to my enjoyment of this profession than any poorly optimized website.
- marknadal 7y agoThis is why I don't. Vanilla JS! 0 Dependencies! Let the 1995 revolution begin!
- zamadatix 7y agoNothing wrong with modularizing code or pulling in complicated things as a dependencies. Rewriting the same tools a million times isn't a great world either. The only thing I avoid/take issue with is pulling huge dependencies to use only a tiny part. Like I wouldn't base anything designed like this page in my projects, it pulls in 1 MB of React to dynamically display an image that updates once a day.
- jaywalk 7y agoHave fun building anything remotely complex.
- johnhenry 7y agohttps://www.snowpack.dev/ https://www.snowpack.dev/
- fullstackchris 7y agoI think about this stuff from time to time... If we think out into the future (50-100 years?)... will there eventually be such a huge footprint for frameworks / packages that there is nobody who remembers how some of the old dependencies were written? Interesting to think of a future bug where nobody would know the solution because it lies so deep in a dependency somewhere :)
- fierarul 7y agoI bet in corporations this is already happening. But in public frameworks and such it will never happen since each generation wants to write their own framework.
- trufas 7y agoYou might find a talk[0] from Jon Blow on this very subject from last year interesting. [0] https://youtu.be/pW-SOdj4Kkk https://youtu.be/pW-SOdj4Kkk
- zemo 7y agoJonathan Blow publicly criticizes Twitter engineering for being unproductive because they haven't released many user-facing features. He regularly berates the entire web industry without knowing anything about how it works. His knowledge is limited to an extremely tiny domain of programming, and he regularly berates people for not following his philosophy. Meanwhile, it took him eight years to make The Witness. What did he build with his eight years time? A first person puzzle game with essentially no interactive physics, no character animations, and no NPC's to interact with. (I actually enjoy The Witness, fwiw.) The vast majority of developers do not have the budget to spend eight years on a single title, and wouldn't want to even if they did. The most notable thing about Jonathan Blow is how condescending he is.
- mrspeaker 7y agoIf that's what you take away from Jonathan Blow then you need to detach from your emotions a bit. He annoys me with how much his condescending attitude towards web development (because I'm a web developer) - but he justifies his positions and is open to argument about it. His talks (like that one linked above) are really inspirational, and he's released two highly successful product: two more than the majority of people you'll ever meet. He's passionate and smart and interesting - and writing him off like that, I think, is not justified.
- sneak 7y agoUnpopular opinion: 1GB is not a lot of data for most people in the first world.
- zamadatix 7y agoI think that's a relatively popular opinion and not what people mean when they point out the size of the node_modules folder is huge.
- Twirrim 7y agoMost people aren't living in the first world, and don't have high bandwidth / low latency connections with which to make pulling down 100s of MB of javascript a non-event. While, yes, they can choose to just not use such tools or libraries, it also presents a fun barrier for learning. It's also perfectly fine to not give a damn about either of the above.
- sneak 7y agoI wonder what percentage of React/Angular/et c contributors live in places where 1GB isn’t a lot of data.
- galvin 7y agoIt's worth noting that node_modules is for project dependencies and most of it won't be included in the final assets. Many projects make use of build tools, command line utilities, testing suites and other libraries in development which don't get deployed. Even for actual app dependencies, many packages include source code, type definitions and multiple choices of builds which inflate the size of the node_modules folder but don't get used in production. So hundreds of megabytes of dependencies can easily be used to make something only a few megabytes in size. Your point still stands of course!
- Macha 7y agoIt's also not a new problem. I remember downloading a 150mb JDK overnight on dial-up for Java 2.
- ryder9 7y ago
- pier25 7y agoI'm using Mithril on my current project and it includes everything you need in less than 10kB gzipped (vdom, components, router, http client). Another huge benefit is that it simplifies you application code immensely. Your store and models are simply vanilla JS since Mithril doesn't have reactivity, observables, etc. Mithril will simply redraw all the vdom whenever a user event happens or when its router/http client do something. You can also call m.redraw() if you need the UI to update if you for example receive something from a web socket. Obviously with this approach the performance cannot be as good as something like Svelte or Inferno, but it's significantly faster than React. https://krausest.github.io/js-framework-benchmark/current.html https://krausest.github.io/js-framework-benchmark/current.ht... Personally I use it with JSX since I can't stand the hyperscript notation. It's certainly not for everyone and all use cases but I think it's totally valid for a good portion of front end work if you can handle a double edged knife.
- chocolatkey 7y agoFor people like me who like the hyperscript syntax, or have to use it, I've found this site very useful for converting existing HTML templates quicker https://arthurclemens.github.io/mithril-template-converter/index.html https://arthurclemens.github.io/mithril-template-converter/i...
- baddox 7y agoUnless there's more to the story than the redraw methods you described, I would suspect that this approach would not be significantly faster than React if your React components correctly use shouldComponentUpdate and React.memo.
- Ididntdothis 7y agoI hope (and expect) that there will be larger libraries that implement the functionality of a lot of very small packages. JavaScript really needs a comprehensive standard library.
- dan-robertson 7y agoOn the other hand it tends to be harder to change things in a standard library (ie that is where good code goes to die) for want of not breaking dependencies. If all libraries are tiny then a set of “the best” libraries can evolve over time through the preferences of individual people. Obviously this doesn’t work so well for finding libraries or having them work great together.
- Ididntdothis 7y agoMost languages have standard libraries and things are fine that way. Most NPM modules have a scope that’s too small. Instead of 100 string modules they should get merged into one string library.
- lmm 7y agoLarge standard libraries are a major pain for evolving the language and the libraries in question. (E.g. Python's standard library is notorious as "where modules go to die"). Putting each piece into its own library is great as long as your tooling can manage them effectively. What could be useful is some concept of a "metalibrary" or "platform", a la the "rust platform" and "haskell platform" efforts - a sensible aggregate of libraries with compatible versions that work together and cover most of the common things you need - not as an inflexible monolith that you have to do big-bang upgrades of, but as a starting point that you can then customize as needed. Almost like a Linux distribution. But keeping the ability to upgrade one small piece without touching the others is really important.
- Ididntdothis 7y agoThings like string manipulation, array manipulation or similar are pretty well understood and could be put into a comprehensive library. On the few more projects I have seen there were some trivial packages like “isodd” or something like this which then recursively pulled in dozens of other trivial stuff. This just doesn’t seem to be a problem in other languages.
- keyle 7y agoNote to the author, less memes and better information... What's the chart in? Megabytes I presume?
- samirelanduk 7y agoIt says MB on the y-axis label.
- simlevesque 7y agoThe unit is on the y axis.
- keyle 7y agoEdit: my bad, the dark mode plugin didn't invert the legend text! Black on dark gray...
- z3t4 7y agoTired of all the bloat in JS, with the distributed app weighting a whopping 2 MB, I said to myself &#@£ this, I'm going to make a native app. What I ended up with was a 300MB statically linked binary - to show "hello world".
- ivanfon 7y agoWhat stack did you use?
- saagarjha 7y ago> statically linked Why?
- z3t4 7y agoI wanted to optimize startup time. And I was thinking it would be easier to distribute with just one "big" executable.
- gdxhyrd 7y agoI guess you are trolling, because a native app can print "hello world" in around 100 bytes. That's a difference of more than 6 orders of magnitude :')
- z3t4 7y agoIt is true that a web app needs a browser. But a native app needs an operating system (OS). Something that I find fascinating though are apps that you can boot into, eg. they do not require an OS. But if you want graphics, mouse support, play sound etc, it becomes quite complicated. Compare that to a web app where you get all that functionality for free, with no libraries needed. And in between we have the shared lib utopia with Debian et.al and dll-hell with Windows. Each stack have their advantages and disadvantages. I however don't see bundle sizes on the web as a huge problem, at least not yet. With service workers you can download the app once, then even use it offline. Everything is just one click away and can be reached via an URL. And if you stick with "vanilla" it can also be fast and lightweight.
- derefr 7y agoSo... is all this stuff actually Javascript? Or are these, say, .po localization files, CSS+PNG template files for client libs, etc.? Or, even worse, are these docs, tests, and other stuff that have no place in the checkout of a recursively-resolved dependency?
- thrower123 7y ago> Or, even worse, are these docs, tests, and other stuff that have no place in the checkout of a recursively-resolved dependency? Bingo. I've seen an embarassing number of what should be dev-dependencies pulled in transitively for some packages, as well. Nothing like seeing some four layer deep dependency pulling gulp-cli because somebody didn't know what they were doing...
- wruza 7y agoYarn folks try hard (it seems) to get rid of local node_modules and link directly to user-wide multi-version repos, but met the resistance from tool maintainers who just walk that dir directly. I can’t remember specific details now, but yarn pnp didn’t work for me, because of browserify and co apparently. As of the cache size, I simply do not care. E.g. installing gnome, kde or openoffice requires to download hundreds of megabytes of binaries and everyone is okay with that. If you’re not familiar with these packages, they draw large ui controls, copy files and render bad tables. Not much different from a regular node project.
- no_wizard 7y agoBig difference though, I'd say, is software from those respective foundations are generally better tested, better documented, and less likely to have security holes than the average npm installed module. I don't think they ever got "left-padded" as they say. Not that your overall point is wrong (most definitely, it is not in my opinion). I think the reality is that this is what actually drives the concern of the size of the node_modules folder, because deep down, all frontend developers know that so many of these packages have never been vetted, probably won't be vetted, and there is alot of duplicate functionality in many of these dependencies, or the libs themselves just aren't designed to be efficiently consumed. Other communities (like Python) have been having community level conversations around the size of their respective package folders & the validity of reproducing their builds, security etc for awhile now, and the governing bodies for those respective communities (and/or language) are working hard to reduce redundancy among packages & encourage cooperation (the PSF, for instance, has funded directly or helped facilitate funding for many keystone packages, like flask, IIRC. It is also my understanding that Microsoft has been doing (and is expanding its efforts to) do this via the https://dotnetfoundation.org/ https://dotnetfoundation.org/ among other efforts) Sadly, I have not seen the same effort within the NPM/frontend focused ecosystem. Though, maybe the open JS foundation will take off: https://openjsf.org/projects/ https://openjsf.org/projects/
- TeMPOraL 7y ago> E.g. installing gnome, kde or openoffice requires to download hundreds of megabytes of binaries and everyone is okay with that Come on. Gnome and KDE are not equivalent of a node project, they're an equivalent of a browser (or at least browser chrome). And OpenOffice does quite a lot more than pretty much any Node project out there.
- gjmacd 7y agoI think yarn workspaces and lerna can help reduce a lot of teh redundancy in your project(s).
- geuis 7y agoBeen spending the last few days building a new site for a long running project I own. For the server, I’m using hapijs. It fits my style a whole lot better than Express. For the client side, it’s entirely just raw css and js. Absolutely no frameworks. Eventually I’m going to have to use a few third party tools when I get around to adding square or braintree integration but that’s a way off. It’s an absolute joy to just sit down and get stuff done. Today I was able to move from getting the basic node server running in under an hour to building out a couple pages and writing some content. Added some css styling like back in the old days and without needing less or sass. Still only about as good as you’d expect from one day of work but it was so easy to do. I didn’t spend hours setting up tooling, researching which extra npm modules were needed, etc. There’s no React garbage, no Everest of overhead just to get to a point where I could work.
- chrisweekly 7y agoYour frustration with React and the npm ecosystem is common but easily remedied; try NextJs.
- 1_player 7y agoNextJS has been an absolute, undocumented nightmare at our company we've gone back to CRA and "hand rolled" SSR to retain our sanity. It's one of those modern magic frameworks that when you need to step outside the simple hello world example app you spend most of your days chasing buggy behaviours mentioned on unresolved month old github issues. When this happens, usually your only choice and suggestion is update to the beta version of some core library, which breaks tons of other packages that haven't yet been updated to the new rewritten API, rinse and repeat. Never again. To be honest, NextJS wasn't worse than the rest of the JS ecosystem, the problem is systemic.
- chrisweekly 7y agoYikes. Your experience in no way matches mine, nor that of many others I've worked with over the years. Using NextJs doesn't absolve you of the responsibility to understand the fundamentals of how your webapp works, but IMHO / IME it strikes a great balance when it comes to DX and convenience, with direct extensibility and support to drop a layer down to do things by hand; IOW, yeah there's some magic but it's the good kind. I'd be very interested to hear more about your "nightmare", and what circumstances led to the painted-corner of depending on beta versions of core libraries. Care to elaborate?
- pajeetLord666 7y agoMan this article is a piece of dog shit lmfao
- bazza451 7y agoDo hate this about javascript. one of the main reasons on the web side (apart from js not having a good STL) however is enterprises are still locked in to IE11 which requires all these tranpilers/bundlers to use modern syntax which usually are the heaviest of all of these dependencies.
- chispamed 7y agoVue-cli at least has a very neat feature; if you build with the —-modern flag it will create two versions of your scripts, one with modern syntax, smaller and faster for modern browsers and another legacy version for older browsers you have to support. It then uses the nomodule html attribute to automatically load the version your browser needs. This also has the nice effect of saving you bandwidth costs.
- kristianp 7y agoProbably lighter than Rails, whichs feels very heavy trying it again after a while. A fresh new rails app with devise and postgresql client nets me a vendor/bundle folder of 183MB. Of course it needs an install of node as well.
- norswap 7y agoIt would be better if it showed how much each of the top 100 packages consumed when installed alone (along with their dependencies), as well as how much weight they single-handedly contribute to the 100-package-install total size (i.e. dependencies pulled by only them and no other package in the top 100). Is the average weight high? Or are there a couple behemoths?
- golergka 7y agoQuote from the bottom of the website (easy to miss): > On my Macbook Pro the folder containing Xcode is larger than 13 GB. And to get the .NET application running at work I had to install over 20 GB of applications and dependencies. So, relativity speaking, JavaScript is still ok. Personally, I completely agree with that. A single gigabyte of hard drive space on a developer's machine or a modern server is not terribly wasteful.
- nizmow 7y agoBut also, the point about .NET is complete garbage. The full .NET 3.1 SDK download for Mac OS X is around 100MB. Based on that hyperbolic statement I'd be hesitant to trust anything else on the site. EDIT: I know he says "the application at work" and it's just an offhand comment, but you shouldn't read too much into it. Unless it means "I had to install Windows 10 on a VM" because it uses classic framework libraries that don't work with Mono or something.
- wafflesraccoon 7y agoOur cooperate Cisco security filter blocks the website `size-of-npm.netlify.com` for a "security threat". NET::ERR_CERT_AUTHORITY_INVALID is the listed error
- zelly 7y agoIf you think this is bad, it's going to get 1000x worse with WASM. A lot more rope there. You will see the Linux kernel being pulled in because someone used system threads.