19 ms·

Could it be that whenever coders today need some fairly trivial functionality, they tend to go out and find a library that contains it. So you end up with lots

Could it be that whenever coders today need some fairly trivial functionality, they tend to go out and find a library that contains it. So you end up with lots and lots of libraries where only a tiny bits of them are used. Just a hypothesis though.

deft 9y ago
I'm pretty sure this is part of it. Remember leftpad? For electron based apps they're just bundling all of chrome which explains the size too.
- allover 9y ago
  Leftpad is not an example of the issue mentioned. Leftpad is an example of a small module that was being used by a lot of projects. GPP is talking about pulling in larger libraries which contain code not actually used by the app. (Also Electron bundles 'just the rendering library from Chromium' [1], not 'all of Chrome'). [1] https://electron.atom.io/docs/tutorial/about/#core-philosophy https://electron.atom.io/docs/tutorial/about/#core-philosoph...
  - deft 9y ago
    Sorry for not being specific. As for leftpad I just meant the concept of using libraries when you really don't need to. Not the size.
    - allover 9y ago
      Well it wasn't that you weren't being specific, more that you were casually maligning the JS community unfairly, when the article isn't even about web/JS :) (I also don't agree with 'don't need to'. The main takeaway from the leftpad debarcle was the fixes to the npm module deletion policy, and hopefully people learning they shouldn't rely on an 'npm install' for production deployments! Whether people should use small modules is still up for debate, there are trade-offs [1] [2]). [1] https://github.com/sindresorhus/ama/issues/10#issuecomment-117766328 https://github.com/sindresorhus/ama/issues/10#issuecomment-1... [2] https://medium.com/@Rich_Harris/small-modules-it-s-not-quite-that-simple-3ca532d65de4 https://medium.com/@Rich_Harris/small-modules-it-s-not-quite...
AndrewStephens 9y ago
There are many reasons why apps are so big, but I think you are right about a major part of the problem. Development environments these days make it very easy to add in third party libraries for very little effort. At a previous job we had a monolithic java server that ended up at over 350Mb of compiled code simply because each development team had imported whatever libraries they thought they needed. In some cases, 2 or 3 versions of the same library were included.
- lsv1 9y ago
  This is why code coverage is so important as part of the general development cycle.
  - djKianoosh 9y ago
    Wait, code coverage? How does that address the number of libraries used or duplicate versions of the same library?
- wooter 9y ago
  i don't see why any of this is a problem
- slaymaker1907 9y ago
  How did they get multiple versions of the same library to work together? Java loads things via the class path so I would have thought that would cause some sort of error.
  - swsieber 9y ago
    Some libraries change the namespace (package names) between major versions, specifically to allow transition from one to another in a gradual manor, or to start following namespace guidelines better.
  - tedunangst 9y ago
    jarjar?
potatolicious 9y ago
I think this is likely the biggest culprit. Another user pointed out an analysis of the Facebook app: http://blog.timac.org/?p=1707 http://blog.timac.org/?p=1707 Most of it is actual code - not assets. For some types of apps (see: games) assets do take up a significant portion of total size, but for most everyday apps bloat by code over-inclusion is likely a bigger problem than asset-bloat. I wonder if it's possible to get major open-source libs to move towards more fine-grained build targets and internal dependency management, so that devs don't pull in a gigantic binary when they're only using a small slice of the functionality.
- Swizec 9y ago
  > I wonder if it's possible to get major open-source libs to move towards more fine-grained build targets and internal dependency management, so that devs don't pull in a gigantic binary when they're only using a small slice of the functionality. Fwiw, this is already a trend in JavaScript land. Libraries are moving towards many small packages so you can import only the parts you need either manually or using a tool like Webpack to throw away what isn't used.
  - cyphar 9y ago
    I don't think that's the right solution to the problem. Nobody wants libleftpad.so, and as someone who works on a distribution the very concept is horrific (making distribution packages for every three-line package does not make anyone happy). What I think GP was arguing for is that you have libstring.so which you can strip down to just having leftpad or w/e with Kconfig or similar configurations (preferably at link time not build time -- otherwise you still have the same problem).
    - RodericDay 9y ago
      I interpreted the comment in exactly the same way, and once again I'm baffled to see a perfectly polite and correct comment downvoted and greyed out.
      - 9y ago
ryandrake 9y ago
This would be true if the entirety of the library was used by the application. Wouldn't the linker throw out anything unused?
- teamhappy 9y ago
  > Wouldn't the linker throw out anything unused? There (usually) is no dead code elimination for libraries.
- mikeash 9y ago
  Depends on the language. If you're using something like C++, you can probably remove a lot of dead code. But with more dynamic languages, like Objective-C or Ruby or JavaScript, it becomes difficult or impossible to prove that a given chunk of code is never used. In Objective-C (which I'm most familiar with), the linker will keep all Objective-C classes and methods because it has no idea if you might be looking things up by name at runtime and invoking them that way. Even if you can throw away dead code, you can run into bloat because a library has a massive set of foundational APIs that the rest is built on, and using one little feature of the library ends up bringing in half the library code because it's used everywhere.
  - Retric 9y ago
    You could use static analysis to find out if anything calls objects dynamically by name at run time which could be a useful optimization. To be safe this would be one bool value for the entire code base, but with some care it would help. A larger issue is size is simply not considered a significant issue.
    - mikeash 9y ago
      With Objective-C, that will always come back "true." Even if you never use dynamic lookups, Apple's frameworks do. I suspect the same is true of other languages.
      - Retric 9y ago
joshvm 9y ago
This happens a lot, especially with dynamic linking. I have a project that uses Qt, OpenCV, CUDA, PCL and VTK - fairly standard stack for 3D imaging and visualisation. Since you normally need to bundle dependencies to account for different versions, this adds up quite fast. Qt adds 20MB for OpenGL, 15MB for the VC++ redist, about 30MB for other core libraries. Some stuff in OpenCV requires Nvidia's performance primitives (NPP), so in goes nppi64_80.dll - that's 100MB alone. opencv_cudaarithm310.dll is 70MB and even opencv_imgproc310.dll weighs in at 30MB. And on and on. So yes, one little call to cv::cuda::remap adds in a boatload of dependencies when all the algorithm is doing is using a lookup table to sample an image. https://github.com/opencv/opencv/blob/master/modules/cudawarping/src/cuda/remap.cu https://github.com/opencv/opencv/blob/master/modules/cudawar...
- maxxxxx 9y ago
  It's pretty ironic that dynamic linking's main motivation was to eliminate duplicate code and reduce code size. Deploy a DLL once, every software uses it and doesn't have to include it. Now it has turned out exactly the opposite way. If we went back to linking object code together we would get smaller sizes. Instead we have to include huge DLLs.
  - dfox 9y ago
    IIRC Windows has pretty involved infrastructure for deduplicating same executable images both in memory and on disk.
  - verytrivial 9y ago
    > dynamic linking's main motivation was to eliminate duplicate code and reduce code size I'm pretty sure the main motivation is/was to allow patching of dependencies independently from applications. Very popular shared libraries might save space overall, but that is a secondary effect.
    - maxxxxx 9y ago
      When I started out the main motivation was to get install sizes down. At least on Windows patching never really worked and ended in "DLL hell".
tylersmith 9y ago
I think this is exactly it. If you start vendoring dependencies you begin to notice how much of other's people's code your software is really using.
dingo_bat 9y ago
As I understand it, when you link your binary, the linker will only include the used parts of the libraries. Linking to a 10M library but calling only one function will not increase your binary size by 10M, probably just a few bytes.

deft 9y ago

I'm pretty sure this is part of it. Remember leftpad? For electron based apps they're just bundling all of chrome which explains the size too.

allover 9y ago
Leftpad is not an example of the issue mentioned. Leftpad is an example of a small module that was being used by a lot of projects. GPP is talking about pulling in larger libraries which contain code not actually used by the app. (Also Electron bundles 'just the rendering library from Chromium' [1], not 'all of Chrome'). [1] https://electron.atom.io/docs/tutorial/about/#core-philosophy https://electron.atom.io/docs/tutorial/about/#core-philosoph...
- deft 9y ago
  Sorry for not being specific. As for leftpad I just meant the concept of using libraries when you really don't need to. Not the size.
  - allover 9y ago
    Well it wasn't that you weren't being specific, more that you were casually maligning the JS community unfairly, when the article isn't even about web/JS :) (I also don't agree with 'don't need to'. The main takeaway from the leftpad debarcle was the fixes to the npm module deletion policy, and hopefully people learning they shouldn't rely on an 'npm install' for production deployments! Whether people should use small modules is still up for debate, there are trade-offs [1] [2]). [1] https://github.com/sindresorhus/ama/issues/10#issuecomment-117766328 https://github.com/sindresorhus/ama/issues/10#issuecomment-1... [2] https://medium.com/@Rich_Harris/small-modules-it-s-not-quite-that-simple-3ca532d65de4 https://medium.com/@Rich_Harris/small-modules-it-s-not-quite...

AndrewStephens 9y ago

There are many reasons why apps are so big, but I think you are right about a major part of the problem. Development environments these days make it very easy to add in third party libraries for very little effort. At a previous job we had a monolithic java server that ended up at over 350Mb of compiled code simply because each development team had imported whatever libraries they thought they needed. In some cases, 2 or 3 versions of the same library were included.

lsv1 9y ago
This is why code coverage is so important as part of the general development cycle.
- djKianoosh 9y ago
  Wait, code coverage? How does that address the number of libraries used or duplicate versions of the same library?
wooter 9y ago
i don't see why any of this is a problem
slaymaker1907 9y ago
How did they get multiple versions of the same library to work together? Java loads things via the class path so I would have thought that would cause some sort of error.
- swsieber 9y ago
  Some libraries change the namespace (package names) between major versions, specifically to allow transition from one to another in a gradual manor, or to start following namespace guidelines better.
- tedunangst 9y ago
  jarjar?

potatolicious 9y ago

I think this is likely the biggest culprit. Another user pointed out an analysis of the Facebook app: http://blog.timac.org/?p=1707 http://blog.timac.org/?p=1707 Most of it is actual code - not assets. For some types of apps (see: games) assets do take up a significant portion of total size, but for most everyday apps bloat by code over-inclusion is likely a bigger problem than asset-bloat. I wonder if it's possible to get major open-source libs to move towards more fine-grained build targets and internal dependency management, so that devs don't pull in a gigantic binary when they're only using a small slice of the functionality.

Swizec 9y ago
> I wonder if it's possible to get major open-source libs to move towards more fine-grained build targets and internal dependency management, so that devs don't pull in a gigantic binary when they're only using a small slice of the functionality. Fwiw, this is already a trend in JavaScript land. Libraries are moving towards many small packages so you can import only the parts you need either manually or using a tool like Webpack to throw away what isn't used.
- cyphar 9y ago
  I don't think that's the right solution to the problem. Nobody wants libleftpad.so, and as someone who works on a distribution the very concept is horrific (making distribution packages for every three-line package does not make anyone happy). What I think GP was arguing for is that you have libstring.so which you can strip down to just having leftpad or w/e with Kconfig or similar configurations (preferably at link time not build time -- otherwise you still have the same problem).
  - RodericDay 9y ago
    I interpreted the comment in exactly the same way, and once again I'm baffled to see a perfectly polite and correct comment downvoted and greyed out.
    - 9y ago

ryandrake 9y ago

This would be true if the entirety of the library was used by the application. Wouldn't the linker throw out anything unused?

teamhappy 9y ago
> Wouldn't the linker throw out anything unused? There (usually) is no dead code elimination for libraries.
mikeash 9y ago
Depends on the language. If you're using something like C++, you can probably remove a lot of dead code. But with more dynamic languages, like Objective-C or Ruby or JavaScript, it becomes difficult or impossible to prove that a given chunk of code is never used. In Objective-C (which I'm most familiar with), the linker will keep all Objective-C classes and methods because it has no idea if you might be looking things up by name at runtime and invoking them that way. Even if you can throw away dead code, you can run into bloat because a library has a massive set of foundational APIs that the rest is built on, and using one little feature of the library ends up bringing in half the library code because it's used everywhere.
- Retric 9y ago
  You could use static analysis to find out if anything calls objects dynamically by name at run time which could be a useful optimization. To be safe this would be one bool value for the entire code base, but with some care it would help. A larger issue is size is simply not considered a significant issue.
  - mikeash 9y ago
    With Objective-C, that will always come back "true." Even if you never use dynamic lookups, Apple's frameworks do. I suspect the same is true of other languages.
    - Retric 9y ago

joshvm 9y ago

This happens a lot, especially with dynamic linking. I have a project that uses Qt, OpenCV, CUDA, PCL and VTK - fairly standard stack for 3D imaging and visualisation. Since you normally need to bundle dependencies to account for different versions, this adds up quite fast. Qt adds 20MB for OpenGL, 15MB for the VC++ redist, about 30MB for other core libraries. Some stuff in OpenCV requires Nvidia's performance primitives (NPP), so in goes nppi64_80.dll - that's 100MB alone. opencv_cudaarithm310.dll is 70MB and even opencv_imgproc310.dll weighs in at 30MB. And on and on. So yes, one little call to cv::cuda::remap adds in a boatload of dependencies when all the algorithm is doing is using a lookup table to sample an image. https://github.com/opencv/opencv/blob/master/modules/cudawarping/src/cuda/remap.cu https://github.com/opencv/opencv/blob/master/modules/cudawar...

maxxxxx 9y ago
It's pretty ironic that dynamic linking's main motivation was to eliminate duplicate code and reduce code size. Deploy a DLL once, every software uses it and doesn't have to include it. Now it has turned out exactly the opposite way. If we went back to linking object code together we would get smaller sizes. Instead we have to include huge DLLs.
- dfox 9y ago
  IIRC Windows has pretty involved infrastructure for deduplicating same executable images both in memory and on disk.
- verytrivial 9y ago
  > dynamic linking's main motivation was to eliminate duplicate code and reduce code size I'm pretty sure the main motivation is/was to allow patching of dependencies independently from applications. Very popular shared libraries might save space overall, but that is a secondary effect.
  - maxxxxx 9y ago
    When I started out the main motivation was to get install sizes down. At least on Windows patching never really worked and ended in "DLL hell".

tylersmith 9y ago

I think this is exactly it. If you start vendoring dependencies you begin to notice how much of other's people's code your software is really using.

dingo_bat 9y ago

As I understand it, when you link your binary, the linker will only include the used parts of the libraries. Linking to a 10M library but calling only one function will not increase your binary size by 10M, probably just a few bytes.