6 ms·

All metrics will be gamed at some point. I don't know exactly how you could even fight this.

jasoneckert 2y ago
Neither do I. I believe the only thing anyone can do is take metrics of how the metrics are gamed, as this particular paper has done.
jazzyjackson 2y ago
there's various reasons webs-of-trust don't takeoff, but I can imagine a system where the metrics I see are only aggregated from friends-of-friends, and any other signal is just considered untrustworthy and therefor not worth observing
- drusepth 2y ago
  Do you still trust that system when your friends-of-friends are the ones gaming the system? Given the inherent network effects of manipulating webs of trust, I wouldn't be surprised if everyone had at least one friend-of-a-friend they shouldn't necessarily trust.
  - morkalork 2y ago
    Given all the obvious bots and sketchy recruiters that try to connect with me on LinkedIn, who all appear to have at least one mutual connection, it probably won't work.
    - jagged-chisel 2y ago
      Do we have a similar issue on GH? I think the nature of the service and its target audience affect this problem in a big way. You can follow anyone on GH, but there's no mutual connection option at all. LI has following and mutual connections. LI also has a much wider audience. How might a 'connection' look on GH? Will people freely connect, or will they appraise requests more closely?
- wruza 2y ago
  I can imagine access to raw data instead of some stupid come-on-game-me-able predefined indicator, and that I can run some private statistical analysis over it. People would use (and share) different algorithms and gamers will at least wander through this collectively created mud without any understanding except for the defaultest measures. But of course this is too complex and “no one will use it” (tm). So we’ll better have a screwed up recommendation system that doesn’t work at all, cause that’s simpler!
- codetrotter 2y ago
  I can only speak for me personally. For me the way that I use GitHub I don’t think the concept of “friends of friends” would be all that useful on GitHub. There are a handful of people that I know IRL that I follow on GitHub. And a few hundred that I follow in total. Out of the handful of people I know IRL, and who I follow on GitHub, only two or three of them are active there any given week. All of the other people I follow I have very little idea who they are. Usually I follow people I don’t know if I come across their profile and either the profile itself or their projects make me follow them. But I star way more different repos than the number of people I click follow on. For me, the main way of discovering new repos are: - Frontpage of HN, and comments in posts on HN. - Specific search results on Google when I have searched for libraries or programs that do specific things. - Libraries on crates.io that I think might be interesting to look into in the future. Maybe once or twice a month I happen to click on the main page of GitHub itself and see mentions of repos that have been committed to or starred or created by people I follow. So for me I don’t think “friends of friends” is a particularly great signal for things to look at. Most of the people I follow, I don’t know much about them. Likewise, for anyone that follows me it’s not necessarily any strong signal that I follow someone else in order to determine if activity from that someone else should be shown or weighted as more significant to my follower just because I happen to follow that other person. If you do want a strong signal for who to boost for my followers based on my own activity, go and look at the dependencies that I am using in my own projects. That’s a pretty good indicator that I put some amount of effort and interest into looking at something. This could be done by GitHub itself, parsing the Cargo.toml files of my projects and extracting the dependencies section and looking up which of those dependencies are hosted on GitHub.
- kube-system 2y ago
  Maybe so, but in this case, I don't think 'stars' is a good candidate for one of those metrics. I think the people worried about 'fake stars' are doing it wrong, and should just ignore the metric entirely.
- yencabulator 2y ago
  For my account, only count stars by the top 50% of the contributors to the projects I have starred?
begueradj 2y ago
It comes down to fighting against the human nature. And that's a lost battle. Set any law you want, our nature will push us to circumvent it even legally.
- thrance 2y ago
  Not nature no, it's all about incentives. Oftentimes it's financial, for github stars it's prestige and visibility.
- mentalgear 2y ago
  Most people are happy living in a fair ecosystem - it's only the 1-2% of the population that seek control, money and power that start trying to exploit the system. Only if we let that minority keep manipulating the system without consequences, it becomes the driving market force that the rest of the population also feels they have to comply to, to go along, as it already has happened in finance, academia, etc.
  - JumpCrisscross 2y ago
    > Most people are happy living in a fair ecosystem For varying and self-serving definitions of fair. (Almost everyone in the rich world is in an unfairly-advantaged minority.)
  - tcmart14 2y ago
    I'd push back against the 1-2%. I think the reality is, 1-2% is the group of people who will exploit the system and more importantly, have the means to do so. But the number of people who would exploit the system is probably quiet a bit higher, but it doesn't matter because they don't have the means to do so.
- vouaobrasil 2y ago
  I don't really think so. The Amish have a nice system. Their society has many fewer bad actors compared to general society. Actually one of the keys is repeated contact. People who have to interact again and again will try and game the system less. Not sure how to build that into a star system but why give up so easily? Do programmers give up when you say "this algorithm can't be made any faster?"
nwienert 2y ago
Only show "Active Developer Stars" by default: - Only accounts that have a decent amount of activity (pushing code, commenting, etc) - Has set up SSH - Older than 2 years - Account active consistently for at least a year - Must have 2-factor enabled - Filled out profile etc
- stevage 2y ago
  So now all the bots are pushing code, have SSH etc...
- zitterbewegung 2y ago
  I've heard of gaming GitHub stars by asking their friends to star their projects which would get around all of your bullets. Hence why I said it would be hard to game.
- eddythompson80 2y ago
  All of those are very, very, easy to automate. There are plenty of bot accounts that have unintentionally checked the full list.
  - nwienert 2y ago
    You can find a set of requirements that aren't. Eg 2-factor can include phone number. And activity requirements can be based on repo maturity (no just pushing to random empty repos). And while some boy accounts may have them, I doubt many have most. Also, you argue on semantics but the general idea of setting up a legitimacy test that factors in various things is very easily doable, the factors can be kept private, and you definitely can find ones that are generally hard to game.
    - gruez 2y ago
      >You can find a set of requirements that aren't. Eg 2-factor can include phone number. And activity requirements can be based on repo maturity (no just pushing to random empty repos). Then you have people complaining about being "shadowbanned" (because there's no recourse if you're a person and the algorithm thinks you're not active enough), or that github is being anti-privacy (by requiring phone number). It's hard to win here.
uludag 2y ago
I believe networks of human individuals can solve this to a good degree assuming a particular topology exists. Like, imagine a group of professionals of decent sized, all specializing in a similar field, and having lots of strong connections between each other where they have ample opportunities to share information. It would be hard for an outsider to come in and astroturf their product without immense effort (like hiring shills to attend conferences). In-person networks also obviously solve the problem stars as reputation: reputation spreads naturally in these sorts of networks. I think the problem comes with algorithmic scale. Maybe a solution would be to have more community building activities (maybe preferably offline).
aydyn 2y ago
Requiring real ID and showing _regional_ stars like Apple/Google would be a start.
- eddythompson80 2y ago
  > Requiring real ID Yeah, people would love that for sure. > showing _regional_ stars like Apple/Google would be a start. What does that mean? I thought regions only impact ranking not the net amount of stars (assuming we're talking about Apple/Google Maps). Which as far as I know, github doesn't do ranking.
  - aydyn 2y ago
    People already use github as a professional portfolio. Facebook uses real ID, and how popular is it? > What does that mean? I thought regions only impact ranking not the net amount of stars (assuming we're talking about Apple/Google Maps). Which as far as I know, github doesn't do ranking. At least on IOS reviews and ratings are by country, I dont actually know about google play though. (I dont have an android to check since I am not poor)
- stronglikedan 2y ago
  > Requiring real ID Sir, this is an HN.
  - aydyn 2y ago
    Oh thank god I thought we were at Wendy's
mentalgear 2y ago
doesn't mean why shouldn't fight back. That's exactly why we need research projects like these: to maintain the balance.
1propionyl 2y ago
Any metric that becomes a target ceases to be a good metric. The wrinkle is that measures that don't easily quantify are more resistant. For example, showing provable use by other reputable or trusted projects, or a significant amount of resources allocated to maintenance, or ... Really just anything that can't be reduced to a single number in a canonical way will in the long run prove far more useful for longer. This of course shifts some of the burden onto potential users to assess things more critically, and forecloses direct numerical comparison. But the idea that you could just look at a number and make such comparisons was faulty from the get go.
sedatk 2y ago
Prioritize the stars given by accounts you follow in the UI. Done.
- p1esk 2y ago
  I don’t want to follow anyone, but I do give stars to repos I like.
  - sedatk 2y ago
    Then you'll have to start following the creators of repos you like to build a web of trust.
awkward 2y ago
I can see github platform internals caring about this for anomaly detection, but as a developer, who cares? I suppose a botnet could be making fake stars on a malware project or supply chain attack, but the problem there doesn't seem like it's the number of stars.
ozim 2y ago
Make a website where you hire people to check out the libraries and publish scores. But people who have chops for that probably have high enough paying jobs not to care. As most likely no one would pay for reviews of libraries.

Neither do I. I believe the only thing anyone can do is take metrics of how the metrics are gamed, as this particular paper has done.

jazzyjackson 2y ago

there's various reasons webs-of-trust don't takeoff, but I can imagine a system where the metrics I see are only aggregated from friends-of-friends, and any other signal is just considered untrustworthy and therefor not worth observing

drusepth 2y ago
Do you still trust that system when your friends-of-friends are the ones gaming the system? Given the inherent network effects of manipulating webs of trust, I wouldn't be surprised if everyone had at least one friend-of-a-friend they shouldn't necessarily trust.
- morkalork 2y ago
  Given all the obvious bots and sketchy recruiters that try to connect with me on LinkedIn, who all appear to have at least one mutual connection, it probably won't work.
  - jagged-chisel 2y ago
    Do we have a similar issue on GH? I think the nature of the service and its target audience affect this problem in a big way. You can follow anyone on GH, but there's no mutual connection option at all. LI has following and mutual connections. LI also has a much wider audience. How might a 'connection' look on GH? Will people freely connect, or will they appraise requests more closely?
wruza 2y ago
I can imagine access to raw data instead of some stupid come-on-game-me-able predefined indicator, and that I can run some private statistical analysis over it. People would use (and share) different algorithms and gamers will at least wander through this collectively created mud without any understanding except for the defaultest measures. But of course this is too complex and “no one will use it” (tm). So we’ll better have a screwed up recommendation system that doesn’t work at all, cause that’s simpler!
codetrotter 2y ago
I can only speak for me personally. For me the way that I use GitHub I don’t think the concept of “friends of friends” would be all that useful on GitHub. There are a handful of people that I know IRL that I follow on GitHub. And a few hundred that I follow in total. Out of the handful of people I know IRL, and who I follow on GitHub, only two or three of them are active there any given week. All of the other people I follow I have very little idea who they are. Usually I follow people I don’t know if I come across their profile and either the profile itself or their projects make me follow them. But I star way more different repos than the number of people I click follow on. For me, the main way of discovering new repos are: - Frontpage of HN, and comments in posts on HN. - Specific search results on Google when I have searched for libraries or programs that do specific things. - Libraries on crates.io that I think might be interesting to look into in the future. Maybe once or twice a month I happen to click on the main page of GitHub itself and see mentions of repos that have been committed to or starred or created by people I follow. So for me I don’t think “friends of friends” is a particularly great signal for things to look at. Most of the people I follow, I don’t know much about them. Likewise, for anyone that follows me it’s not necessarily any strong signal that I follow someone else in order to determine if activity from that someone else should be shown or weighted as more significant to my follower just because I happen to follow that other person. If you do want a strong signal for who to boost for my followers based on my own activity, go and look at the dependencies that I am using in my own projects. That’s a pretty good indicator that I put some amount of effort and interest into looking at something. This could be done by GitHub itself, parsing the Cargo.toml files of my projects and extracting the dependencies section and looking up which of those dependencies are hosted on GitHub.
kube-system 2y ago
Maybe so, but in this case, I don't think 'stars' is a good candidate for one of those metrics. I think the people worried about 'fake stars' are doing it wrong, and should just ignore the metric entirely.
yencabulator 2y ago
For my account, only count stars by the top 50% of the contributors to the projects I have starred?

begueradj 2y ago

It comes down to fighting against the human nature. And that's a lost battle. Set any law you want, our nature will push us to circumvent it even legally.

thrance 2y ago
Not nature no, it's all about incentives. Oftentimes it's financial, for github stars it's prestige and visibility.
mentalgear 2y ago
Most people are happy living in a fair ecosystem - it's only the 1-2% of the population that seek control, money and power that start trying to exploit the system. Only if we let that minority keep manipulating the system without consequences, it becomes the driving market force that the rest of the population also feels they have to comply to, to go along, as it already has happened in finance, academia, etc.
- JumpCrisscross 2y ago
  > Most people are happy living in a fair ecosystem For varying and self-serving definitions of fair. (Almost everyone in the rich world is in an unfairly-advantaged minority.)
- tcmart14 2y ago
  I'd push back against the 1-2%. I think the reality is, 1-2% is the group of people who will exploit the system and more importantly, have the means to do so. But the number of people who would exploit the system is probably quiet a bit higher, but it doesn't matter because they don't have the means to do so.
vouaobrasil 2y ago
I don't really think so. The Amish have a nice system. Their society has many fewer bad actors compared to general society. Actually one of the keys is repeated contact. People who have to interact again and again will try and game the system less. Not sure how to build that into a star system but why give up so easily? Do programmers give up when you say "this algorithm can't be made any faster?"

nwienert 2y ago

Only show "Active Developer Stars" by default: - Only accounts that have a decent amount of activity (pushing code, commenting, etc) - Has set up SSH - Older than 2 years - Account active consistently for at least a year - Must have 2-factor enabled - Filled out profile etc

stevage 2y ago
So now all the bots are pushing code, have SSH etc...
zitterbewegung 2y ago
I've heard of gaming GitHub stars by asking their friends to star their projects which would get around all of your bullets. Hence why I said it would be hard to game.
eddythompson80 2y ago
All of those are very, very, easy to automate. There are plenty of bot accounts that have unintentionally checked the full list.
- nwienert 2y ago
  You can find a set of requirements that aren't. Eg 2-factor can include phone number. And activity requirements can be based on repo maturity (no just pushing to random empty repos). And while some boy accounts may have them, I doubt many have most. Also, you argue on semantics but the general idea of setting up a legitimacy test that factors in various things is very easily doable, the factors can be kept private, and you definitely can find ones that are generally hard to game.
  - gruez 2y ago
    >You can find a set of requirements that aren't. Eg 2-factor can include phone number. And activity requirements can be based on repo maturity (no just pushing to random empty repos). Then you have people complaining about being "shadowbanned" (because there's no recourse if you're a person and the algorithm thinks you're not active enough), or that github is being anti-privacy (by requiring phone number). It's hard to win here.

uludag 2y ago

I believe networks of human individuals can solve this to a good degree assuming a particular topology exists. Like, imagine a group of professionals of decent sized, all specializing in a similar field, and having lots of strong connections between each other where they have ample opportunities to share information. It would be hard for an outsider to come in and astroturf their product without immense effort (like hiring shills to attend conferences). In-person networks also obviously solve the problem stars as reputation: reputation spreads naturally in these sorts of networks. I think the problem comes with algorithmic scale. Maybe a solution would be to have more community building activities (maybe preferably offline).

aydyn 2y ago

Requiring real ID and showing _regional_ stars like Apple/Google would be a start.

eddythompson80 2y ago
> Requiring real ID Yeah, people would love that for sure. > showing _regional_ stars like Apple/Google would be a start. What does that mean? I thought regions only impact ranking not the net amount of stars (assuming we're talking about Apple/Google Maps). Which as far as I know, github doesn't do ranking.
- aydyn 2y ago
  People already use github as a professional portfolio. Facebook uses real ID, and how popular is it? > What does that mean? I thought regions only impact ranking not the net amount of stars (assuming we're talking about Apple/Google Maps). Which as far as I know, github doesn't do ranking. At least on IOS reviews and ratings are by country, I dont actually know about google play though. (I dont have an android to check since I am not poor)
stronglikedan 2y ago
> Requiring real ID Sir, this is an HN.
- aydyn 2y ago
  Oh thank god I thought we were at Wendy's

mentalgear 2y ago

doesn't mean why shouldn't fight back. That's exactly why we need research projects like these: to maintain the balance.

1propionyl 2y ago

Any metric that becomes a target ceases to be a good metric. The wrinkle is that measures that don't easily quantify are more resistant. For example, showing provable use by other reputable or trusted projects, or a significant amount of resources allocated to maintenance, or ... Really just anything that can't be reduced to a single number in a canonical way will in the long run prove far more useful for longer. This of course shifts some of the burden onto potential users to assess things more critically, and forecloses direct numerical comparison. But the idea that you could just look at a number and make such comparisons was faulty from the get go.

sedatk 2y ago

Prioritize the stars given by accounts you follow in the UI. Done.

p1esk 2y ago
I don’t want to follow anyone, but I do give stars to repos I like.
- sedatk 2y ago
  Then you'll have to start following the creators of repos you like to build a web of trust.

awkward 2y ago

I can see github platform internals caring about this for anomaly detection, but as a developer, who cares? I suppose a botnet could be making fake stars on a malware project or supply chain attack, but the problem there doesn't seem like it's the number of stars.

ozim 2y ago

Make a website where you hire people to check out the libraries and publish scores. But people who have chops for that probably have high enough paying jobs not to care. As most likely no one would pay for reviews of libraries.