4 ms·

What's the fundamental difference between code that is written as a part of the standard library and the code that is a part of a package? Both are open-source

What's the fundamental difference between code that is written as a part of the standard library and the code that is a part of a package? Both are open-sourced and are written by pretty much the same people.

johannes1234321 6y ago
My hope is that with a larger standard library there is more shared ownership of the library at large. With many small libraries there often is a single person working on it and others assuming "oh, somebody will have reviewed it" This can happen with a big library as well (as individual maintainers "own" specific parts) but there it is easier to establish a review culture. (Whether a review would have caught this specific case is yet another question, but again having an organisation could guarantee a timely response, is read if depending on a single contributor)
mekkkkkk 6y ago
Some amount of oversight by qualified people?
jadedtuna 6y ago
I would say the difference is, standard library goes through mode scrutiny/checks before an update is released. This of course depends on the library in question, as some non-std libraries are certainly subjected to more checks than others.
mikepurvis 6y ago
The standard libraries receive far, far more review, both at the API design and implementation level. Anyone can push literally anything to PyPI.
- golergka 6y ago
  Depends on the library in question. I don't think that lodash doesn't get enough eyeballs on it.
  - dmos62 6y ago
    You have a point there, but if you have a very well vetted third-party library that provides things as basic as IP address parsing, you'll probably see that library included into the official library package.
  - oblio 6y ago
    Lodash might, or might not, offer the same guarantees that a stdlib generally offers: backwards and forwards compatibility, a clear deprecation and removal schedule, etc. It's great to have high quality third party libraries but stuff like most of lodash should be in a stdlib, already.
  - mikepurvis 6y ago
    Well yes, but the point is that everything in the stdlib receives scrutiny and a long-term commitment to mainenance/upkeep. A small proportion of pypi/npm receives these things, but the vast majority is abandonware trash. You only have to briefly scroll through the main PyPI index to realise just how awful it is: https://pypi.org/simple/ https://pypi.org/simple/ (caution— multi-megabyte html)
- tveita 6y ago
  Interestingly the Python standard library behaves similarly (on Python 3.9.2): >>> import ipaddress >>> ipaddress.ip_address('010.0.0.1').is_private True Even though the documentation claims that "Leading zeroes are tolerated only for values less than 8 (as there is no ambiguity between the decimal and octal interpretations of such strings)." https://docs.python.org/3.9/library/ipaddress.html#ipaddress.IPv4Address https://docs.python.org/3.9/library/ipaddress.html#ipaddress...
  - qwertox 6y ago
    Same in 3.8. Versions below 3.8 throw a ValueError. But this is not an issue: >>> import ipaddress >>> ipaddress.IPv4Address('010.0.0.1') IPv4Address('10.0.0.1')
qwertox 6y ago
In addition to the very likely improved vetting, other libraries dependent on it will be coding against an API which is less likely to receive breaking changes, unless it is a major Python release. Even then, breaking changes will be clearly announced. You won't see any pinning against a specific and possibly outdated version.
simion314 6y ago
Trust. I can trust code that is part of Java, .Net or Python standard library is good or decent quality. With npm you can't trust anything, package can go away, malware could be inserted in them, backward compatibility could brake. Say you grab a 10 years old Java/.Net project source and a node one, and try to setup a developer environment to be able to create a fix. Would you bet that on average is 10X harder to get stuff to run on node ?
- JimDabell 6y ago
  > I can trust code that is part of Java, .Net or Python standard library is good or decent quality. I think you’re being a little optimistic there. Consider Java’s URL class, for instance. Its equals() implementation will resolve the hostname in both URLs and consider them equal if they resolve to the same IP address. So that’s a) completely wrong, b) a potential source of security problems (consider an attacker pointing their hostname to your IP address to control URLs that are “equal” to your URLs, for instance), c) horribly unpredictable (two URLs may or may not be the same depending on whether you are online or not), and d) awful for performance (introduces a network lookup even for hashcode(), for instance). That’s significantly worse than a bug like this, which is essentially an oversight that failed to consider an obscure notation nobody uses. > Say you grab a 10 years old Java/.Net project source and a node one, and try to setup a developer environment to be able to create a fix. Would you bet that on average is 10X harder to get stuff to run on node ? Java is probably okay, but I’ve tried this with .NET and it was extremely painful.
  - simion314 6y ago
    Funny enough you mention url, I had to fix a bug recently in JS code where our project was rejecting valid urls, the issue was that someone found somewhere a regex that validated urls but during the years the standards updated and now more characters were allowed. So copy pasting perfectly good code is bad because it might get outdated. I would bet that on average Java standard library code(that is not deprecated) will be higher quality then npm packages.
fsociety 6y ago
Attack surface, stdlib is one library... packages tends to be hundreds if not more of libraries. Much easier to take over a package than a stdlib.

johannes1234321 6y ago

My hope is that with a larger standard library there is more shared ownership of the library at large. With many small libraries there often is a single person working on it and others assuming "oh, somebody will have reviewed it" This can happen with a big library as well (as individual maintainers "own" specific parts) but there it is easier to establish a review culture. (Whether a review would have caught this specific case is yet another question, but again having an organisation could guarantee a timely response, is read if depending on a single contributor)

mekkkkkk 6y ago

Some amount of oversight by qualified people?

jadedtuna 6y ago

I would say the difference is, standard library goes through mode scrutiny/checks before an update is released. This of course depends on the library in question, as some non-std libraries are certainly subjected to more checks than others.

mikepurvis 6y ago

The standard libraries receive far, far more review, both at the API design and implementation level. Anyone can push literally anything to PyPI.

golergka 6y ago
Depends on the library in question. I don't think that lodash doesn't get enough eyeballs on it.
- dmos62 6y ago
  You have a point there, but if you have a very well vetted third-party library that provides things as basic as IP address parsing, you'll probably see that library included into the official library package.
- oblio 6y ago
  Lodash might, or might not, offer the same guarantees that a stdlib generally offers: backwards and forwards compatibility, a clear deprecation and removal schedule, etc. It's great to have high quality third party libraries but stuff like most of lodash should be in a stdlib, already.
- mikepurvis 6y ago
  Well yes, but the point is that everything in the stdlib receives scrutiny and a long-term commitment to mainenance/upkeep. A small proportion of pypi/npm receives these things, but the vast majority is abandonware trash. You only have to briefly scroll through the main PyPI index to realise just how awful it is: https://pypi.org/simple/ https://pypi.org/simple/ (caution— multi-megabyte html)
tveita 6y ago
Interestingly the Python standard library behaves similarly (on Python 3.9.2): >>> import ipaddress >>> ipaddress.ip_address('010.0.0.1').is_private True Even though the documentation claims that "Leading zeroes are tolerated only for values less than 8 (as there is no ambiguity between the decimal and octal interpretations of such strings)." https://docs.python.org/3.9/library/ipaddress.html#ipaddress.IPv4Address https://docs.python.org/3.9/library/ipaddress.html#ipaddress...
- qwertox 6y ago
  Same in 3.8. Versions below 3.8 throw a ValueError. But this is not an issue: >>> import ipaddress >>> ipaddress.IPv4Address('010.0.0.1') IPv4Address('10.0.0.1')

qwertox 6y ago

In addition to the very likely improved vetting, other libraries dependent on it will be coding against an API which is less likely to receive breaking changes, unless it is a major Python release. Even then, breaking changes will be clearly announced. You won't see any pinning against a specific and possibly outdated version.

simion314 6y ago

Trust. I can trust code that is part of Java, .Net or Python standard library is good or decent quality. With npm you can't trust anything, package can go away, malware could be inserted in them, backward compatibility could brake. Say you grab a 10 years old Java/.Net project source and a node one, and try to setup a developer environment to be able to create a fix. Would you bet that on average is 10X harder to get stuff to run on node ?

JimDabell 6y ago
> I can trust code that is part of Java, .Net or Python standard library is good or decent quality. I think you’re being a little optimistic there. Consider Java’s URL class, for instance. Its equals() implementation will resolve the hostname in both URLs and consider them equal if they resolve to the same IP address. So that’s a) completely wrong, b) a potential source of security problems (consider an attacker pointing their hostname to your IP address to control URLs that are “equal” to your URLs, for instance), c) horribly unpredictable (two URLs may or may not be the same depending on whether you are online or not), and d) awful for performance (introduces a network lookup even for hashcode(), for instance). That’s significantly worse than a bug like this, which is essentially an oversight that failed to consider an obscure notation nobody uses. > Say you grab a 10 years old Java/.Net project source and a node one, and try to setup a developer environment to be able to create a fix. Would you bet that on average is 10X harder to get stuff to run on node ? Java is probably okay, but I’ve tried this with .NET and it was extremely painful.
- simion314 6y ago
  Funny enough you mention url, I had to fix a bug recently in JS code where our project was rejecting valid urls, the issue was that someone found somewhere a regex that validated urls but during the years the standards updated and now more characters were allowed. So copy pasting perfectly good code is bad because it might get outdated. I would bet that on average Java standard library code(that is not deprecated) will be higher quality then npm packages.

fsociety 6y ago

Attack surface, stdlib is one library... packages tends to be hundreds if not more of libraries. Much easier to take over a package than a stdlib.