4 ms·

It's not that I didn't think "it was easy enough to use PyPI do analytics and analysis": it is near impossible for the layman to use PyPI to do analytics and an

by orf 3y ago

It's not that I didn't think "it was easy enough to use PyPI do analytics and analysis": it is near impossible for the layman to use PyPI to do analytics and analysis in it's current form. The volume of data is very unweildly, there are numerous quirks with extracting packages and no tooling exists to help you in any way. This means no analysis has been done on the contents of PyPI. In turn this means malicious packages are harder to detect (and for sure still present somewhere in there), it means people publish an absolutely crazy number of credentials to PyPI on a daily basis without ever knowing (+ no simple way to find concrete ways to improve this) and it means there is a lack of exploration on the impacts of language features/changes on the ecosystem. To me the GitHub aspect isn't important or interesting. Would it make any difference if it was distributed from a series of git repositories hosted on S3? It's the git apsect that is interesting, because it lowers the barrier for anyone to access the corpus of already public, already mirrored and already automatically-scanned-by-bad-actors code that is on PyPI. While this project is more "a number of things glued-together" than "a groundbreaking invention", I have to disagree with the triviality aspect. Most problems we deal with can be reduced to 'copying X from one place to another' (sorting?), and the devil is always in the details. > I don't like this, it's this kind of stunt that makes me reluctant to publish my code in general. Isn't this quite circular? People using code you publish publicly makes you reluctant to publish code publicly?

lizard 3y ago
> Isn't this quite circular? People using code you publish publicly makes you reluctant to publish code publicly? Not that I disagree with this project, but just to maybe help see it from a little different perspective... When people publish their code, I think they typically expect it's going to be used like import my_package do_something_cool() So it is a little weird when things like this come along and change that expectation. It's kind of like, "I scanned millions of Facebook photos for soda cans to see if people prefer Pepsi or Coke!" People didn't post those photos be be part of a project, they just wanted to share some pictures with their friends.
- orf 3y ago
  Thanks for that, I can definitely appreciate this perspective. I’d say it’s more akin to uploading photos to a shared public host like imgur rather than Facebook, but regardless I can see how someone’s expectations of who/what would use it might be different than mine.
  - lizard 3y ago
    Kinda like you say yourself, the service is probably the least interesting part. It doesn't really matter whether its a public repository or if you use thing your friends shared only within their network. When it comes to what people expect and how they'll feel about breaking those expectations, the only difference is that a smaller network of generally like-minded people _may_ already be cool with it, or at least it's easier to ask. I'm not even saying they're right to feel weird about it. Just that people are going to feel what they're going to feel, and doing something they didn't expect is a sure-fire way to get them to feel _something_.
- cmcaleer 3y ago
  It's not unusual to want to change certain behaviours of a project, e.g. by subclassing something within it. It's also worth at least having some idea of the code you're running before you run it, particularly if you don't know the developer, for many reasons but for e.g. [0]. I'm not really sold on the perspective that if you're a sophisticated enough developer to know+upload+publish on pypi that you wouldn't expect someone to read your code. In many ways that's kind of the point. Not to say such people don't exist, but they're probably a small minority. [0]: https://cyble.com/blog/over-45-thousand-users-fell-victim-to-malicious-pypi-packages/ https://cyble.com/blog/over-45-thousand-users-fell-victim-to...
  - lizard 3y ago
    According to the stats on the original link, there are over 25,000 identified secret ids/keys/tokens in the data. And it looks like that's just identifiable secrets, e.g. "Google API Keys" that I'm guessing are identifiable because they have a specific pattern, and may be missing other secrets that use less recognizable patterns. I mean, sure, compared to the 478,876 Projects claimed on https://pypi.org/ https://pypi.org/, that's a pretty small minority. On the other hand, I'd guess many Python packages don't use these particular services, or even need to connect to a remote service at all, so the area for this class of mistake should be smaller. And mistakes do happen, but that's a pretty big thing to miss if you are knowingly publishing your code with the expectation other people will be reading it.

by orf 3y ago

lizard 3y ago

> Isn't this quite circular? People using code you publish publicly makes you reluctant to publish code publicly? Not that I disagree with this project, but just to maybe help see it from a little different perspective... When people publish their code, I think they typically expect it's going to be used like import my_package do_something_cool() So it is a little weird when things like this come along and change that expectation. It's kind of like, "I scanned millions of Facebook photos for soda cans to see if people prefer Pepsi or Coke!" People didn't post those photos be be part of a project, they just wanted to share some pictures with their friends.

orf 3y ago
Thanks for that, I can definitely appreciate this perspective. I’d say it’s more akin to uploading photos to a shared public host like imgur rather than Facebook, but regardless I can see how someone’s expectations of who/what would use it might be different than mine.
- lizard 3y ago
  Kinda like you say yourself, the service is probably the least interesting part. It doesn't really matter whether its a public repository or if you use thing your friends shared only within their network. When it comes to what people expect and how they'll feel about breaking those expectations, the only difference is that a smaller network of generally like-minded people _may_ already be cool with it, or at least it's easier to ask. I'm not even saying they're right to feel weird about it. Just that people are going to feel what they're going to feel, and doing something they didn't expect is a sure-fire way to get them to feel _something_.
cmcaleer 3y ago
It's not unusual to want to change certain behaviours of a project, e.g. by subclassing something within it. It's also worth at least having some idea of the code you're running before you run it, particularly if you don't know the developer, for many reasons but for e.g. [0]. I'm not really sold on the perspective that if you're a sophisticated enough developer to know+upload+publish on pypi that you wouldn't expect someone to read your code. In many ways that's kind of the point. Not to say such people don't exist, but they're probably a small minority. [0]: https://cyble.com/blog/over-45-thousand-users-fell-victim-to-malicious-pypi-packages/ https://cyble.com/blog/over-45-thousand-users-fell-victim-to...
- lizard 3y ago
  According to the stats on the original link, there are over 25,000 identified secret ids/keys/tokens in the data. And it looks like that's just identifiable secrets, e.g. "Google API Keys" that I'm guessing are identifiable because they have a specific pattern, and may be missing other secrets that use less recognizable patterns. I mean, sure, compared to the 478,876 Projects claimed on https://pypi.org/ https://pypi.org/, that's a pretty small minority. On the other hand, I'd guess many Python packages don't use these particular services, or even need to connect to a remote service at all, so the area for this class of mistake should be smaller. And mistakes do happen, but that's a pretty big thing to miss if you are knowingly publishing your code with the expectation other people will be reading it.