5 ms·

Online Reputation Systems: how to design one that does what you need

67 points by Rod 16y ago

goatforce5 16y ago
Related: Just yesterday I noticed that OReilly recently published Building Web Reputation Systems (Randy Farmer, Bryce Glass) http://oreilly.com/catalog/9780596159801 http://oreilly.com/catalog/9780596159801 http://buildingreputation.com/ http://buildingreputation.com/ Randy was consulting with a company I was at a couple of years ago and while the two of us certainly didn't see eye-to-eye on a lot of things, it encouraged a lot of thought and debate. I'll be ordering the book today.
bmm6o 16y ago
So many sites get their reputation system slightly wrong, and end up incentivizing behaviors that they claim to not want. On discussion forums with reputation systems, the "echo chamber" effect is encouraged - it's been a running joke on slashdot since before the term "online reputation system" was coined. Also, early replies get viewed and upvoted more often, encouraging quick responses over more thoughtful (but later!) ones.
- notaddicted 16y ago
  Reddit has worked on this, here is a guest blog post by Randall Munroe explaining: http://blog.reddit.com/2009/10/reddits-new-comment-sorting-system.html http://blog.reddit.com/2009/10/reddits-new-comment-sorting-s...
- kevindication 16y ago
  I independently observed this on slashdot and made a few bucks exploiting the psychology of readers. I wrote up my thoughts a while back: http://squareone.pheared.net/2009/10/make-money-posting-to-slashdot.html http://squareone.pheared.net/2009/10/make-money-posting-to-s...
  - bmm6o 16y ago
    That's awesome. I love that the secret sauce turned out to be Simpson's quotes.
- krainboltgreene 16y ago
  Would a time-delayed showing of the comments help?
  - bmm6o 16y ago
    I don't think so. Or at least not at the cost of inhibiting discussion. I don't have any answers, and as the article discussed, you can't design a reputation scheme in a vacuum - you need to define your goals and align the incentives with them. That said, it seems to me that sites like HN vastly over-reward early posters to popular threads. On the other hand, this is only a bug in so far that it leads to problematic behavior. I don't see people taking advantage of it (yet). I'm more concerned with the problem of under-rewarding late, thoughtful replies. I think this is a critical problem with the Stack Exchange (StackOverflow) platform.
    - louislouis 16y ago
      The question is how can u get people to re-read the comments page for new comments? Maybe something like a "(5 new comments)" next to the comments link might work??
      - bmm6o 16y ago
        Metafilter does that, and I think that helps people continue a discussion that they are interested in, regardless of its effect on "reputation". HN does a similar thing with the "threads" link, but it only helps for conversations the user is active in (to encourage involvement?). My thinking on late (and hence low-exposure) contributions is more along the lines of weighting. If 10 people read a comment and 5 upvote it, it should be rewarded similarly to a comment with 100 reads and 50 upvotes.
Perceval 16y ago
It's not clear whether this book really grapples with the problem of pseudonymity on the web. What happens when a single user can start multiple accounts--one for good behavior and the other for trolling? When the trolling account gets banned, can they just start another one? There can be no 'reputation' without an answer to the problem of pseudonymity.
- krainboltgreene 16y ago
  The guys who created the 4chan web board code seem to think that the comment is what should have the reputation, not the user. This is one reason why they have a default-to-anon posting style (With options to force anon only posting). People then focus on the content of the comment, not the posters aura/name/backing.
  - lotharbot 16y ago
    One thing I initially hated about HN but have quickly grown to like: usernames are displayed in tiny, faint text. There are no avatar images, signatures, or other visual identifiers. This has a similar, though limited, effect -- it puts the focus on the content, not on the user. (My wife regularly reads my comments without realizing I'm the author; this would not happen on a board with avatars.) Yet it also allows users to build a reputation over time, encouraging us to consistently produce quality content.
    - bkudria 16y ago
      For anyone who might want to try the opposite - boldy highlight some users - try my Chrome extension: https://chrome.google.com/extensions/detail/mkdhfabjcebcgnpgnhefebefiabhmbfn/ https://chrome.google.com/extensions/detail/mkdhfabjcebcgnpg...
  - ryanelkins 16y ago
    When it comes to opinions though it does seem that who says something should play a part. Think of reviews - someone who says something is great and has a history of thinking the same things are great that I do is more relevant to me than an anonymous person thinking something is great. Reputation is a level of relevance that a user can build within a community.
jamesshamenski 16y ago
Here @ HN the simplicity of Karma allows for the most inciteful and productive community discussions on the net. Would any of these suggestions actually improve the way things work here? Also, as much as i hate blippy and swipely - if companies know who actually purchased a good - then their feedback becomes hugely valuable.

67 points by Rod 16y ago

Related: Just yesterday I noticed that OReilly recently published Building Web Reputation Systems (Randy Farmer, Bryce Glass) http://oreilly.com/catalog/9780596159801 http://oreilly.com/catalog/9780596159801 http://buildingreputation.com/ http://buildingreputation.com/ Randy was consulting with a company I was at a couple of years ago and while the two of us certainly didn't see eye-to-eye on a lot of things, it encouraged a lot of thought and debate. I'll be ordering the book today.

bmm6o 16y ago

So many sites get their reputation system slightly wrong, and end up incentivizing behaviors that they claim to not want. On discussion forums with reputation systems, the "echo chamber" effect is encouraged - it's been a running joke on slashdot since before the term "online reputation system" was coined. Also, early replies get viewed and upvoted more often, encouraging quick responses over more thoughtful (but later!) ones.

notaddicted 16y ago
Reddit has worked on this, here is a guest blog post by Randall Munroe explaining: http://blog.reddit.com/2009/10/reddits-new-comment-sorting-system.html http://blog.reddit.com/2009/10/reddits-new-comment-sorting-s...
kevindication 16y ago
I independently observed this on slashdot and made a few bucks exploiting the psychology of readers. I wrote up my thoughts a while back: http://squareone.pheared.net/2009/10/make-money-posting-to-slashdot.html http://squareone.pheared.net/2009/10/make-money-posting-to-s...
- bmm6o 16y ago
  That's awesome. I love that the secret sauce turned out to be Simpson's quotes.
krainboltgreene 16y ago
Would a time-delayed showing of the comments help?
- bmm6o 16y ago
  I don't think so. Or at least not at the cost of inhibiting discussion. I don't have any answers, and as the article discussed, you can't design a reputation scheme in a vacuum - you need to define your goals and align the incentives with them. That said, it seems to me that sites like HN vastly over-reward early posters to popular threads. On the other hand, this is only a bug in so far that it leads to problematic behavior. I don't see people taking advantage of it (yet). I'm more concerned with the problem of under-rewarding late, thoughtful replies. I think this is a critical problem with the Stack Exchange (StackOverflow) platform.
  - louislouis 16y ago
    The question is how can u get people to re-read the comments page for new comments? Maybe something like a "(5 new comments)" next to the comments link might work??
    - bmm6o 16y ago
      Metafilter does that, and I think that helps people continue a discussion that they are interested in, regardless of its effect on "reputation". HN does a similar thing with the "threads" link, but it only helps for conversations the user is active in (to encourage involvement?). My thinking on late (and hence low-exposure) contributions is more along the lines of weighting. If 10 people read a comment and 5 upvote it, it should be rewarded similarly to a comment with 100 reads and 50 upvotes.

Perceval 16y ago

It's not clear whether this book really grapples with the problem of pseudonymity on the web. What happens when a single user can start multiple accounts--one for good behavior and the other for trolling? When the trolling account gets banned, can they just start another one? There can be no 'reputation' without an answer to the problem of pseudonymity.

krainboltgreene 16y ago
The guys who created the 4chan web board code seem to think that the comment is what should have the reputation, not the user. This is one reason why they have a default-to-anon posting style (With options to force anon only posting). People then focus on the content of the comment, not the posters aura/name/backing.
- lotharbot 16y ago
  One thing I initially hated about HN but have quickly grown to like: usernames are displayed in tiny, faint text. There are no avatar images, signatures, or other visual identifiers. This has a similar, though limited, effect -- it puts the focus on the content, not on the user. (My wife regularly reads my comments without realizing I'm the author; this would not happen on a board with avatars.) Yet it also allows users to build a reputation over time, encouraging us to consistently produce quality content.
  - bkudria 16y ago
    For anyone who might want to try the opposite - boldy highlight some users - try my Chrome extension: https://chrome.google.com/extensions/detail/mkdhfabjcebcgnpgnhefebefiabhmbfn/ https://chrome.google.com/extensions/detail/mkdhfabjcebcgnpg...
- ryanelkins 16y ago
  When it comes to opinions though it does seem that who says something should play a part. Think of reviews - someone who says something is great and has a history of thinking the same things are great that I do is more relevant to me than an anonymous person thinking something is great. Reputation is a level of relevance that a user can build within a community.

jamesshamenski 16y ago

Here @ HN the simplicity of Karma allows for the most inciteful and productive community discussions on the net. Would any of these suggestions actually improve the way things work here? Also, as much as i hate blippy and swipely - if companies know who actually purchased a good - then their feedback becomes hugely valuable.