12 ms·
Disabling npm's progress bar yields a 2x npm install speed improvement
- deleted 11y ago[deleted]
- gavinjoyce 11y agoSome further details on the issue: https://github.com/npm/npm/issues/11283 https://github.com/npm/npm/issues/11283
- VeilEm 11y agoPlease refrain from adding unhelpful comments to the issue.
- eric_h 11y agoIt is tremendously unfortunate that that needs to be said at all.
- voltagex_ 11y agoI guess it's GitHub's Eternal September. I'm all for more people having access to development tools, but GitHub really has to step up the tools available for managing issues/comments.
- bostonvaulter2 11y agoThere is this open letter to github from open source maintainers: https://github.com/dear-github/dear-github https://github.com/dear-github/dear-github
- skissane 11y agoI think bug tracker comment forms for open source projects need some kind of question first (at least from people who aren't active contributors), like "Please select your type of comment": * Nontechnical: I would like to express my eagerness that the bug be fixed or request its priority be increased, but don't have any additional technical information to add * Technical: I have useful technical information to add, beyond what is already in the bug Then "Nontechnical" type comments would be hidden from the developers by default (unless they specifically ask to see them), but their count could be used as a bug prioritisation mechanism (effectively leaving a "Nontechnical" type comment would be counted the same as a vote)
- striking 11y agoHow is this unhelpful exactly? It's a link to another discussion that readers might be interested in.
- X-Istence 11y agoOp was stating you shouldn't follow that link and go "Hey, me too" or "+1" on the Github Issue. Not that the link to the issue was unhelpful.
- baudehlo 11y agoGitHub badly needs an "approve" system that is external to comments, like bitbucket has.
- deleted 11y ago[deleted]
- striking 11y agoBecause I can't amend this comment: I misunderstood the commenter above me. My sincerest apologies. Have a nice day!
- l1ambda 11y ago+1
- deelowe 11y agoNow that's what I call a bug report: https://github.com/npm/npm/issues/11283#issuecomment-175246823 https://github.com/npm/npm/issues/11283#issuecomment-1752468...
- gkya 11y agoThat's what I call a slideshow.
- swang 11y agoI don't see this at all, what's more worrying is that npm 3.* is apparently way slower than npm 2.* I did this test on a npm library I wrote, installs some test tools and also has to compile some gyp stuff 2.14.2 progress=false real 0m26.589s user 0m12.086s sys 0m3.362s progress=true real 0m29.553s user 0m12.455s sys 0m3.486s 3.5.2 progress=false real 0m57.489s user 0m13.298s sys 0m3.421s progress=true real 1m1.084s user 0m15.690s sys 0m3.644s While I don't see any significant difference between progress bars or none for 3.x, what is 3.x doing that causes almost 30 seconds more churning?
- rhinoceraptor 11y agoIt uses a very different dependency resolution algorithm, the biggest difference as I understand it is it will install dependencies maximally flat whereas npm 2 was much more nested. I believe the reason it is slower is it can't do as much parallelism as npm 2. Also, npm 2 was susceptible to a lot of race conditions and non-determinism, especially with compiled dependencies.
- Too 11y agoThe funky progress bar is about the coolest thing with npm. They could let it take a minute if they wanted too and i would keep watching it with amazement.
- alanh 11y agonpm 2.x user here. Does anybody want to share a video of the 3.x progress bar? (Easy with QuickTime or CloudApp, btw)
- simi_ 11y agoSeems like a perfect task for asciinema https://asciinema.org/ https://asciinema.org/
- agumonkey 11y agohttps://www.youtube.com/watch?v=elWIVKWhWTk https://www.youtube.com/watch?v=elWIVKWhWTk
- DonHopkins 11y agoThis seems to make it go faster: http://bennyhillthis.com/?v=elWIVKWhWTk http://bennyhillthis.com/?v=elWIVKWhWTk
- freshyill 11y agoI see that with iTerm 2, but using Terminal gives me a very different experience. Here's a side-by-side: https://youtu.be/odoVfHHBYVM https://youtu.be/odoVfHHBYVM Until recently, I had no clue iTerm was doing something weird. I just figured npm install was just ugly and incomprehensible.
- bitwarrior 11y ago...has he controlled for network speed? Unless he's set up a local NPM caching proxy, he's going over the 'net for these packages. It'll take more than 1 datum before I believe this hype.
- gavinjoyce 11y agoMany people have confirmed a major speed improvement. I've repeated this with an empty cache, primed cache and in different orders many times and have always got the same results. Having said that, a good number of people have reported seeing no speed improvement too.
- rhinoceraptor 11y agoThe network utilisation is much different between npm 2 and 3: https://github.com/npm/npm/issues/10380 https://github.com/npm/npm/issues/10380
- draw_down 11y agoAlso the second run probably used stuff that was put in the npm cache during the first run. Not sure how good a test this is. A good read: "Programmers need to learn statistics or I will kill them all" http://zedshaw.com/archive/programmers-need-to-learn-statistics-or-i-will-kill-them-all/ http://zedshaw.com/archive/programmers-need-to-learn-statist...
- gavinjoyce 11y agoIt yields the same results in reverse order and also when the cache is primed for both and cleared for both
- untog 11y agoProgress bar or not, npm has gotten extremely slow with v3, to the extent that someone is working on a performance-centered alternative called ied: http://gugel.io/ied/ http://gugel.io/ied/ installs are literally the #1 thing npm needs to do well, and I really hope we see some improvement soon.
- smaili 11y agoAwesome tool, but wish the url was a bit easier to spell/remember!
- miketaylr 11y agojust gugel it
- striking 11y agoHeh, just "ied" will give you this: https://en.wikipedia.org/wiki/Improvised_explosive_device https://en.wikipedia.org/wiki/Improvised_explosive_device "ied package manager" works, though.
- nashashmi 11y agoNormally, I am not a fan of replacing words in the search engine with computer science stuff (think AJAX), but I can accept computer words replacing words that symbolize violence.
- uxp 11y agoYour results, even when logged out, are heavily biased towards your previous query genres. For instance, I'm not logged in, but am at home with an IP address that hasn't changed in over a year despite being leased. When I search Google for the term "Ruby", the entire first page is for the Ruby programing language. My father works with jewelry. Last week while visiting him, I did in fact search for the term "Ruby" on his computer in his normal browser, and even though he was also logged out, I wasn't able to find one result dealing with the programming language on the first two pages.
- morebetterer 11y agoUpon first seeing npm 3.x I immediately added `--progress false --color false` to my npm installs and never looked back. Color and terminal graphics are the work of the devil.
- coldtea 11y ago>Color and terminal graphics are the work of the devil. Or you know, it's 2016, and it's npm who does it badly.
- deleted 11y ago[deleted]
- morebetterer 11y agoI don't even want color and terminal graphics done goodly. I code with two monochrome terminals on the screen using vim and the command line.
- untog 11y agoNot everyone is you. While I agree that flashy graphics are silly in the command line, I find color-based highlighting to be quite useful.
- morebetterer 11y ago> Not everyone is you. Never claimed that was the case. God speed with your colors. May they bring you fame and good fortune.
- scrollaway 11y ago> Never claimed that was the case. Whether you claimed it is not in question. You're saying "[color in the terminal] is the work of the devil", which has a clear intent to dissuade people from coloring terminal output. Which, frankly, is insanity. Most apps do it well nowadays: --color=auto/always/never, usually defaulting to auto. So when GP says "not everyone is you", they mean be considerate. If you want color off, you're being catered for. But don't try and push that choice on everybody who isn't you.
- maaaats 11y agoCould it be linked with the terminal being used? Edit, reminds me of this, which is why I'm wondering http://stackoverflow.com/q/21947452/923847 http://stackoverflow.com/q/21947452/923847
- ghjnut 11y agoHere's a breakdown https://github.com/npm/npm/issues/11283#issuecomment-175246823 https://github.com/npm/npm/issues/11283#issuecomment-1752468...
- jessaustin 11y agoI guess we're supposed to assume that V8 is smart enough to inline that sort of function definition, but it seems deeply weird to me that drawBar() is defined inside the "hot" method like that. Also, I would have expected that streams would have been used for this. Logging is a very common "simple example" on node streams how-to pages.
- libeclipse 11y agoThis makes sense. It's like when I'm solving a project euler problem, I don't use verbose output because it runs much faster without having to print everything to the screen. (Python)
- jakub_g 11y agoThat's exactly my experience. Once I wrote some naive PE solution in nodejs and added some logging at each step. After few minutes and far from a solution I stopped it and disabled logging. I got the solution within seconds. The diff was few orders of magnitude, which was pretty surprising for me.
- arjie 11y agoSometimes writing to the terminal can be the slowest part because it's unbuffered by default in an interactive prompt, so even just redirecting stdout to /dev/null (or even a file) can be faster. Had that happen some ten years ago, so my explanation could be out of date or misremembered but the effect was real.
- btilly 11y agoYour explanation is correct. That is the semantics of Unix stdio buffering as implemented by glibc. So across a wide variety of languages, you'll see a significant performance difference if you are doing lots of writes to the terminal. On the other hand if you didn't do this, then interactive terminal programs would be entirely unusable.
- meric 11y agoThat's the biggest issue I have with the new npm. I wonder how many developer-days it has wasted already.
- nijiko 11y agoI'd also like to take the time to mention that the caching system when installing / other actions is extremely inefficient (for my times I have the progress bar turned off already and this is a project with around 80 modules shared between dev / prod): https://github.com/npm/npm/issues/10890 https://github.com/npm/npm/issues/10890 I started on a very (I would like to emphasize very a thousand times over) basic proof-of-concept to show how much faster it could be in the order of magnitudes: https://gist.github.com/nijikokun/2f1f16325f8ffe14b1b3 https://gist.github.com/nijikokun/2f1f16325f8ffe14b1b3 All this does is build a json of every package you currently have installed, and utilizes that as a lookup store the next time instead of rebuilding it every install; this was targeted towards installing / uninstalling existing packages. Not fresh installs. Fresh installs would benefit from bulk lookups via the API imo.
- voltagex_ 11y agoLooks good. I wonder if it'd be too much scope creep to add extension points for LAN/proxy caching. I know of one small dev team who had a single ADSL connection shared between ~10 people - NPM downloads would have been painful.
- vvoyer 11y agoI made an npm package that does this https://github.com/vvo/npm-pkgr https://github.com/vvo/npm-pkgr
- freshyill 11y agoI posted this below, but I'll post again here for visibility. What the heck is up with npm install on iTerm 2? I had no idea something was weird until recently, when I did an install with Termial. Here's a side-by-side video: https://youtu.be/odoVfHHBYVM https://youtu.be/odoVfHHBYVM
- Macha 11y agoWeird I'm seeing the left result in iterm 2 on my work mac
- freshyill 11y agoCheck Syrup-tan's comment above for an explanation.
- Syrup-tan 11y agoI had the same problem (and figured it was normal for a while), after some looking-into it I was able to fix it by unchecking "Treat ambiguous-width characters as double width" [0] [0] https://denpa.moe/~syrup/Screen%20Shot%202016-01-26%20at%206.39.12%20PM.png https://denpa.moe/~syrup/Screen%20Shot%202016-01-26%20at%206...
- freshyill 11y agoThat did the trick, thanks!
- ClayFerguson 11y agoDoing GUI calls is a notorious way to slow down apps. I sped up an app (major software from major large corporation) by a factor of 10x a few years back by removing GUI updates. What you have to do is create a thread that refreshes the GUI periodically and not let your main worker thread ever work on the GUI. This particular app was an Eclipse Plugin (a source control app) and the Eclipse SWT to update the console log window was being called after every file operation (of which there were 10s of thousands). All I did was move the GUI refreshing into it's own thread, and call it only every 250millis or so. I was hailed a hero, and deservedly so, for speeding up our app by 10X and was one of my most satisfying accomplishments in my software career.
- cdr 11y agoWhen I gave the Nexus Mod Manager a try a little while ago, I wondered how it managed to take 18 hours to copy a few hundred files from one directory to another - on a SSD. Wouldn't be surprised if this was at least part of it.
- beerbajay 11y agoNexus Mod Manager is impressively bad. The interface is insanely laggy; select a mod, wait a second, list starts to refresh, wait a second, list finishes refreshing, wait a second, the mod info shows up. It's really easy to do things by mistake because of the lagginess (and probably just generally poor quality of programming). It has multithreaded downloads, but even these are messed up. Every time I tried them about 50% of the downloads got corrupted.
- nicklo 11y agoThis strategy is also encouraged by best iOS development practices (main thread never works on GUI). Edit: 'main' as in the one doing the one doing the work, in the same spirit of the original post. In truth, the true main thread that runs in main() is where all the iOS GUI work happens. Apologies for the confusing wording.
- panic 11y ago
- ww520 11y agoScreen IO is slow. A trick to speed up unzip speed, do unzip -qq. That skips all the screen output and make it flies.
- dschiptsov 11y agoWell-designed server platform.
- mrmondo 11y agoDon't even get me started on the number of reasons why nodejs should not be running on servers, I'll be here all day. We're currently deploying a NodeJS app and we'll never do it again - the amount of problems with that ecosystem is quite amazing. As I tweeted earlier today "JS ppl tend to have marketing & UI/UX experience so JS crapcode gets popularised, becomes widespread, then we have to deal with the fallout".
- pixelHD 11y agoI'm genuinely interested. I've been using nodejs for hobby projects and scripts, so haven't seen too many issues. But I'd like to know how things are and what issues you've been seeing.
- mrmondo 11y agoHowdy, we've had lots of issues with single threaded performance, poor memory management especially around garbage collection, we had to modify our SELinux policies as it likes to execute memory off the stack, npm has been very unreliable and often unreasonably slow, error handling of the node index app server seems to have some interesting behaviour but I haven't dug into that one too deeply yet and the list goes on. As an experienced systems engineer it's very clear that this is browser technology that was never designed not fit to run on servers.
- teknologist 11y agoI work in China. Downloading from the outer Internet will never match the speed of writing to the console. Not too worried
- xiaq 11y agoNot to mention that when the binary you are downloading happens match particular keywords.
- teknologist 11y agoOh, that's fine. You can just download it from one of those mirror sites that host their own copies of the files. You can trust that it's genuine because of that big green checkmark on the page.
- beerbajay 11y agoYou just have to wait for a security worker to manually edit them out
- mannykannot 11y agoI used to joke, in the context of virus scanning, that I could give you a more accurate progress bar, but the scan would take twice as long...
- hharnisch 11y agoIt's all good, Laurie fixed it - https://twitter.com/seldo/status/692192238445711360 https://twitter.com/seldo/status/692192238445711360
- ferdamravenec 11y agoAnother product bitten by the dark side of progress indicators. One of the most often underestimated areas of product development, though certainly not among the most important underestimates. Easy to underestimate its cost and impact. After all, it's just a progress indicator.
- aidenn0 11y agosbcl's compilation output is so verbose that the compilation instructions mention that it builds faster on xterm than gnome-terminal
- bechampion 11y agoSomeone with more UI development can comment here , but i always thought progress bars where more like a signal to the user that "something is being done" , and not a representation of how much is done and how much is left.
- masklinn 11y agoThe whole point of a progressbar is to show work progress (as a fraction of total work). The issue is it's often difficult to map "work done" to a linear percentage, so either you do it on the cheap and your progressbar is just a souped-up spinner with no relation to wall-clock work time or you do lots of extra work to get more precise estimates but you end up taking more time overall.
- TazeTSchnitzel 11y agoTwo years ago, PHP's package manager, Composer, was sped up by disabling the garbage collector: http://blog.ircmaxell.com/2014/12/what-about-garbage.html http://blog.ircmaxell.com/2014/12/what-about-garbage.html (Or more accurately the cycle collector. You can't turn off reference counting.)
- dannypgh 11y agoYes, but, which download feels longer?
- BuckRogers 11y agoI don't know the exact cause here, but I can say I'm thankful for my systems background that I've gleamed for years before moving into programming. Most sysadmin types know that displaying output on a filecopy slows down transfers greatly because the screen vsync holds up the operation. When I write utilities today that do similar things, I only display output when writing and testing the program. Otherwise I'll write to memory then dump the results in a logfile at the end. In this case since there is probably a less intensive way of providing status updates, and can probably be resolved by doing intermittent checks.