11 ms·

Collaborative filtering doesn't work for us

157 points by andreyk 5y ago

andyxor 5y ago
If I recall a few years ago Chatroulette really needed a "Hotdog/Not-Hotdog" classifier. I hope they solved that tiny problem using big data.
- munk-a 5y ago
  As hilariously as the comment is worded - it's honestly the truth and the fact that this blog post is focused on "creating conversations longer than 30 seconds" and not on reducing conversations quicker than 5 seconds (because that's obviously a dong I'm looking at) feels to me like they've really missed the mark. As uh, depressing as this may sound to the AI that needs to learn it, a hotdog focused neural network (let's just assume we'd call it "DongMind" because obviously we would) would probably greatly increase Chatroulette's value proposition.
  - willcipriano 5y ago
    I heard this a long time ago but if I recall correctly, the Diablo 2 devs had a way to find cheaters and used match making to put them in games with one another. If a similar strategy was applied here...
  - throwaway675309 5y ago
    Jinwang would make more sense (pun on the name of the character who makes the hotdog/not hotdog classifier in Silicon Valley which the gp is referencing)
- caramelazimuth 5y ago
  Why not combine the "hotdog" filter with the collaborative filtering to make much better predictions about mutual engagement by specific users?
  - nceasy 5y ago
    can you elaborate?
    - bagels 5y ago
      Some people want to see nudity. Some people don't. Could use presence of nudity and preference for nudity in the model.
mdoms 5y ago
Great to see them reporting a negative result. We don't get enough of that in our industry.
sanity31415 5y ago
I think it would be more accurate if they said that the types of collaborative filtering they tried didn't work. It seems like they only explored quite a small part of the solution space.
BarryMilo 5y ago
Has someone made Chatroulette with only audio yet? And no typing obviously. Seems it would solve the problem.
- austinjp 5y ago
  Everything old is new again. Phone chatrooms were like this in the 80s.
  - ampdepolymerase 5y ago
    We had Clubhouse in the 80s? Are you sure?
    - musicale 5y ago
      In the 1880s? They were called "party lines." https://en.wikipedia.org/wiki/Party_line_(telephony) https://en.wikipedia.org/wiki/Party_line_(telephony)
      - skinkestek 5y ago
        Extremely good HN comment :-D Funny and my history knowledge got a serious boost!
    - Geee 5y ago
      Yes, but it was called "club house". It had audio, video and haptics.
- deleted 5y ago
  [deleted]
disconcision 5y ago
nice to see some data but a little more qualitative analysis seems called for. like, i don't know what the current userbase of chatroulette is like, but from the reputation 10 years ago i would guess that a significant proportion of users are looking for someone they find attractive. if this is the case, and given that people are on-average heterosexual, then it would not be at all surprising that a 2-chain-based system would perform much worse than random. again, i'm not sure that this theory is well-founded, but it would be nice to see some demographically-informed discussion. also, saying that 2-chains didn't work doesn't seem to justify the title. collaborative filtering is broader than a single technique.
- mypalmike 5y ago
  That's very insightful yet kind of obvious. It would be interesting to see if 3-chain data fixed this, where a central meaningful conversation BC and meaningful conversations AB and CD tended to imply anything about AD.
shemnon42 5y ago
Colour me surprised: I can't read this article over Chick-fil-A wifi.
Straw 5y ago
This doesn't even come close to the power of (not even particularly) modern collaborative filtering algorithms. CF doesn't need to assume that A talks to B well and B talks to C well implies A talks to C well- rather, it typically tries to learn an embedding for A, B, C that predicts their interactions effectively. In the simplest form, this could be applying SVD or nonnegative matrix factorization to the user interaction matrix, truncating to get a low-rank symmetric approximation. More advanced forms would try to predict the interaction as some function of the two user embeddings, possibly even a neural network. This only shows that a particular feature based on 2-chains of interactions isn't useful, whereas the standard approaches I described can try to infer information across any number of connections, and don't necessarily assume that 2-chains as described are indicative of a particular result.
- sillysaurusx 5y ago
  Skip all that and cut to "a neural network." Presto. I wish I understood this simplicity when I first got into ML. I was always annoyed with claims like "a NN will solve your problems." Oh really? Yeah. Really. You can learn the theory later. Sure, applying SVD or nonnegative matrix factorization to the user interaction matrix, truncating to get a low-rank symmetric approximation makes sense now. (I have to pause and think about it for a little bit, but I do indeed understand each part.) The cool thing was, I didn't need to. The theory helps inform my future ML training runs. But I was able to get most of the magic without knowing a damn thing. Literally just throw Pytorch at the problem and screw around until it starts giving results. You'll be amazed and shocked and delighted how far that gets you. (And all the professional mathematicians and statisticians will be disgusted, but who cares?)
  - nestorD 5y ago
    Actually its one of those problems where it is worth not skipping. I have seen people applying neural networks to transform embeddings before doing a final dot-product where it could be proven (and indeed was confirmed in practice) that all of that was equivalent to a direct embedding. Furthermore, SVD type of algorithm are orders of magnitude faster than NN to train (which is why they become used to train word embeddings, see Glove).
tunesmith 5y ago
Is anyone else bothered by that font? The paragraphs feel really stretched out horizontally, and the headings seem one step removed from Comic Sans.
NilsIRL 5y ago
I might have misunderstand the blog post but it seems to me that everything that was tried used a "2-chain" which works on the assumption that the same type of people enjoy talking together. From my personal experience although this can often be true it can also often be completely wrong. Concretely the friends I have are not a copy of myself and I am not necessarily friends with those that are very similar to me. Another way to put it is to put people's personality as a binary, they're either of type I or type II. In the example given, person A and C both had a non-trivial conversation with person B. So person B is type I and person A and C type II. In this case type II can be interpreted as "has non-trivial conversations with type I". There is no reason for person A and C which are both type II to have a non-trivial conversation.
- exporectomy 5y ago
  Specifically, type I and II are probably male and female. Isn't it immediately obvious that model doesn't make sense if people are talking to the opposite sex? No, Bob and Carlie don't want to have a "meaningful chat" because they both chatted with Alice.
  - mattkrause 5y ago
    s/Carlie/Charlie/, I assume? (Carlie, where I grew up, is a woman's name so this threw me for a bit).
    - exporectomy 5y ago
      Yes, Charlie, sorry.
  - woko 5y ago
    With this in mind, the conclusion of the blog post is hilarious. Quotes from the blog post: > [The] rough idea is that if, say, Alice spoke to Bob for a long time and then Alice also spoke to Carol for a long time, then Bob and Carol are more likely than not to speak for a long time too. > [...] > The data have clearly shown us that, despite proving a successful technique for many other products across a range of contexts, collaborative filtering simply wasn’t a match for Chatroulette. Basically: 1) make an arbitrary assumption about your data, 2) apply a widely-used recommendation algorithm, 3) notice that the results are really bad, 4) reject the algorithm... rather than the assumption. Admittedly, I don't know whether the algorithm requires this assumption, or whether a slightly different one could be used. For instance, if Alice spoke to Bob and Charlie, then Bob and Charlie don't want to talk together. Or, if Alice spoke to Bob and Charlie, and Carol spoke to Bob, then Carol should be recommended to Charlie, etc.
elliekelly 5y ago
I’m shocked chatroulette even still exists. It makes me wonder what their dedicated user base is like. My gut says it would be either some of the coolest and most interesting people on the internet or some of the creepiest and most shocking trolls.
deleted 5y ago
[deleted]
qPM9l3XJrF 5y ago
>Unfortunately, the results showed a true-negative rate of 78%. i.e. most of the time the model failed to predict when a meaningful conversation was about to occur. >... >The model is significantly worse than a coin-flip. Obviously, this is not good; and given that the majority of conversations on the site are trivial 1. This isn't correct use of the term "true negative". A true negative is when your model predicts a negative, and the label is actually negative. "the model failed to predict when a meaningful conversation was about to occur" would correspond to a false negative (the model predicts negative, the the label is actually positive) 2. "The model is significantly worse than a coin-flip" - Talking about coin flips hardly makes sense if the majority of conversations on the site are trivial. A better way to think about this question would be from a Bayesian perspective. Define a random variable L for linkage. L=True if there exists a user B such users A and C both had non-trivial conversations with B. L=False otherwise. Then define a random variable T for whether the conversation is trivial or not. Then you can compute P(T=False|L=True) and P(T=False|L=False) from your data. It could be that what this article is trying to say is that P(T=False|L=False) > P(T=False|L=True), i.e. the presence of a linkage decreases the probability of a nontrivial conversation. This could occur if, for example, most users of Chatroulette are using it as a dating site, and since heterosexual people outnumber gay people, most nontrivial conversations occur between men and women. So if there's a linkage between A and C, that predicts that A and C are of the same gender, and they won't be interested in talking to each other. Anyway, I didn't read any further than that, but I would suggest just ranking users based on their propensity for nontrivial conversations and connecting users with other users who like to have nontrivial conversations (or perhaps even better, compute the mean/median conversation length for each user to get a sense of their preferred conversation length, then connect users to other users who have a similar preferred conversation length). I suspect there's room to use machine learning to improve Chatroulette a lot, even to the point where it could be the next Facebook. People already pay thousands of dollars to attend Harvard etc. for the networking. You just have to use machine learning tools correctly and find a way to capture some of that value.
- deleted 5y ago
  [deleted]
dannykwells 5y ago
Release the data and let the community figure this out! This particular analysis is rather limpid and, ummm, misses a key feature of the most common chatroulette use case.
bagels 5y ago
Would be cool if they published some data so that others could give it a try. I bet someone out there can build a model that is better than a coin flip.
robbrown451 5y ago
I think what doesn't work is their assumption that amount of time talking means it was a "good match." It's quite possible that it took some time for someone to figure out the other person was awful, or even that the entire conversation was negative, such as an argument. The algorithm doesn't know that because it is purely based on talk time, rather than taking into account such things as if the users gave each other good ratings.
- pyinstallwoes 5y ago
  Wouldn't that trend towards optimal over time? Perhaps with a decay or bias for recency. Valuable connection = what you put time into, most latest
- bagels 5y ago
  The hypothesis they chose is not the most obvious one to attack, and it ignores the most obvious application of collaborative filtering. Their hypothesis is that if A likes B, and A likes C then C will like B. They should instead attack this hypothesis (much more common application of collaborative filtering, think predicting what book you might buy on Amazon): A likes B, A likes C, D likes B -> D has higher probability of liking C as well
  - robbrown451 5y ago
    But to them, A liking B and B liking A are one and the same, since all they are measuring is talk time. Regardless, I think it would do much better if it took into account ratings. Even better if they allow you to rate them on several things and then give an overall rating. I personally hate it when things try to learn from my behavior rather than my explicit declarations. For instance, if I click on clickbait (hey, I'm human) it shouldn't assume I actually like that thing or want to see more of it.
shadilay 5y ago
I thought chatroullete was the premier platform in the 'exposing yourself to strangers' industry? Before you needed to take the city bus to experience this level of public exposure but now you can do so from the comfort of your own home.
sesuximo 5y ago
Maybe by and large guys like to talk to girls, but guys don’t like to talk to guys and girls don’t like to talk to girls. This would make a 2 chain model look bad I think. Maybe try a longer chain? i.e. “Bob liked Alice, and Charlie liked both Alice and Diane, so maybe Bob will like Diane” (Ik this is heteronormative — sorry)
DantesKite 5y ago
They could've saved a lot of time if they just read the Terence Tao blog (I'm being a bit cheeky here since it's a rather counterintuitive insight and doesn't necessarily hold for all cases). "If X correlates with Y and Y correlates with Z, then this does not imply X correlates with Z." https://terrytao.wordpress.com/2014/06/05/when-is-correlation-transitive/ https://terrytao.wordpress.com/2014/06/05/when-is-correlatio...
rl3 5y ago
This is Chatroulette. Burn the matchmaking algorithms and keep it true random? For example, if every dating site in existence did this the world would be a better place. Instead, we typically see layers of opaque algorithmic shit governing human interaction. The platform's interests are then in effect prioritized, not the user's. Did anyone ever stop and think: "Is aggressively matching what we believe to be compatible or like-minded people actually a good thing for society? Is it possible that's actually a pretty terrible thing? Do we even know?" There's a certain beauty to true randomness in a matchmaking context. It's a fair playing field for everyone. You start using highly imperfect and frankly arbitrary algorithms to govern who matches with who, and suddenly certain people aren't going to meet certain people for reasons they're probably not even aware of. "Hey, this person could have been your significant other or best friend, but you never met them because some engineers at some tech company decided to get fancy. Fuck you." Or, in so many dating app cases: fancy and greedy. Every time these algorithms are employed, there's always people that fall into the cracks; the dead zones. There they languish. Perhaps months or years later the algorithm is adjusted and affords them a reprieve. Perhaps never. --- This isn't meant to be an indictment of Chatroulette. On the contrary, good on them for reporting negative results. Obviously some filtering must be done for a myriad of reasons, such as curbing abuse. Their stated mission is even to match people "At random." They've thought about this problem a lot more than I have. That said, I thoroughly believe the road to hell is paved by matching users based on metrics such as projected compatibility or similar interests.
- zwaps 5y ago
  Just to answer that point: people (academically) look at whether it is a good idea to match similar people, what such a social structure means and how preferences for homophily, imposed or intrinsic, may affect the social structure. Do social media websites do think about it? I am sure some of then have. But then, their goals typically differ vastly from what you would consider good for society.
  - rl3 5y ago
    Thanks for this. Admittedly my tone was somewhat hyperbolistic, but you bring up an excellent point that I'd like to build off: It pains me to think there's large bodies of deep academic research in the social sciences that could guide certain tech platforms in a much more responsible direction. Instead, the drive for market domination and profit ensures that a mangled society is left in their wake (ironically, economics is also a social science). The hubris certain tech leaders have in making the decisions they do astounds me. Virtually every major tech platform has the money to have a veritable army of social scientists pondering the ethics of anything that company does with its platform. Certainly some of these companies do employ ethicists, though I can't help but feel more often than not it's a dog and pony show.
danbmil99 5y ago
TIL Chatroulette is still a thing
scotty79 5y ago
Why just 2-chain? If you wanted to predict matches and lengthy coversation between random tinder users (if there was no rigid separation for genders) you'd fail in similar fashion. Try 3-chain.
kangnkodos 5y ago
The basic assumption is that if M1 talks with F1 for a long time, and M2 talks with F1 for a long time then M1 and M2 will talk for a long time. Assuming it's based on hetero dating type situations, you can see why this wouldn't work. But there could be many other factors in play also. For example, if the M's above were talkers and the F's above were listeners, of any gender, you can see that two talkers or two listeners would not have a long conversation. But a talker and a listener is a good pair. Instead they should look at this. If M1 talks to F1 and F2 for a long time, and M2 talks to F1 for a long time, then will M2 talk with F2 for a long time? I think this would have a better chance of working.
- PaulDavisThe1st 5y ago
  I must have missed the mentions of gender in the actual paper. No wait, I didn't. There is no mention of gender in the actual paper. It has nothing to do with dating ... the paper is about chatroulette.
  - PoignardAzur 5y ago
    I think GP's point is that chatroulette's article is incomplete because it doesn't analyze how gender impacts chat duration. Like, the first question that came to me reading this was "Is there a pattern of men systematically skipping other men until they're connected with a woman?". It's a little weird the article doesn't ever address that kind of question.
abetusk 5y ago
This is really interesting but I'm highly skeptical. I'm not sure we're given enough information to really get at why they're getting the results they are. They report that if A-C have a non-trivial conversation and B-C have a non trivial conversation, then A-B have a "true-negative rate" of 78% of having a trivial conversation, by which I interpret a 78% over the base line of having A-B have a trivial conversation. The fact that it's so heavily biased against having a non-trivial conversation seems like it should be a huge red flag. How are they conducting the study? Are they informing participants A and B that they're being matched because they've had a non-trivial interaction with C? Is it random? If they are being informed and it turns out that there is a gender component to it, then that might be a big piece of the puzzle in that jealousy and competition start to come into play. Also, if gender is a big factor, then that might inform a "3-chain" collaborative filtering method in that if M0-F0 have a non-trivial interaction, M0-F1 have a non trivial interaction, M1-F0 have a non trivial interaction, then maybe M1-F1 have a higher chance of having a non trivial interaction. I think psychologists call these "confounding factors"? They get a massively strong signal of something but then disregard and and say "doesn't work for us". Something's going on, it's be nice to dig deeper into what it is instead of dismissing it offhand.
captainmuon 5y ago
I could repost what I recent said about onlyfriends here. I think for many people, the whole point of Chatroulette is "show me yours I'll show you mine". That's why the gender thing that others have commented on is not surprising (if Anna chatted with Bob, and Bob with Charlene, it does not follow that Anna wants to chat with Charlene - because people are looking to chat with the opposite sex.) If they want to pivot and sanitize it, they'll loose their core audience. Maybe it would be worthwhile to include two different populations in the model... One that looks for "meaningful conversations" and one that looks for nudity...
a_trn 5y ago
Hi, this is CR founder. Pleasantly surprised to see HN take interest in this. I hope you will not mind if I use this opportunity to say that I am looking for people who can help me on this and other interesting problems. at@chatroulette.com
SergeAx 5y ago
> the rough idea is that if, say, Alice spoke to Bob for a long time and then Alice also spoke to Carol for a long time, then Bob and Carol are more likely than not to speak for a long time too. This is not how collaborative filtration works. If Alice independently spoke to Bob and Carol for a long time, and Dwayne spoke to Bob for a long time, then chances are Dwayne may speak for a long time to Carol.

157 points by andreyk 5y ago

If I recall a few years ago Chatroulette really needed a "Hotdog/Not-Hotdog" classifier. I hope they solved that tiny problem using big data.

munk-a 5y ago
As hilariously as the comment is worded - it's honestly the truth and the fact that this blog post is focused on "creating conversations longer than 30 seconds" and not on reducing conversations quicker than 5 seconds (because that's obviously a dong I'm looking at) feels to me like they've really missed the mark. As uh, depressing as this may sound to the AI that needs to learn it, a hotdog focused neural network (let's just assume we'd call it "DongMind" because obviously we would) would probably greatly increase Chatroulette's value proposition.
- willcipriano 5y ago
  I heard this a long time ago but if I recall correctly, the Diablo 2 devs had a way to find cheaters and used match making to put them in games with one another. If a similar strategy was applied here...
- throwaway675309 5y ago
  Jinwang would make more sense (pun on the name of the character who makes the hotdog/not hotdog classifier in Silicon Valley which the gp is referencing)
caramelazimuth 5y ago
Why not combine the "hotdog" filter with the collaborative filtering to make much better predictions about mutual engagement by specific users?
- nceasy 5y ago
  can you elaborate?
  - bagels 5y ago
    Some people want to see nudity. Some people don't. Could use presence of nudity and preference for nudity in the model.

mdoms 5y ago

Great to see them reporting a negative result. We don't get enough of that in our industry.

sanity31415 5y ago

I think it would be more accurate if they said that the types of collaborative filtering they tried didn't work. It seems like they only explored quite a small part of the solution space.

BarryMilo 5y ago

Has someone made Chatroulette with only audio yet? And no typing obviously. Seems it would solve the problem.

austinjp 5y ago
Everything old is new again. Phone chatrooms were like this in the 80s.
- ampdepolymerase 5y ago
  We had Clubhouse in the 80s? Are you sure?
  - musicale 5y ago
    In the 1880s? They were called "party lines." https://en.wikipedia.org/wiki/Party_line_(telephony) https://en.wikipedia.org/wiki/Party_line_(telephony)
    - skinkestek 5y ago
      Extremely good HN comment :-D Funny and my history knowledge got a serious boost!
  - Geee 5y ago
    Yes, but it was called "club house". It had audio, video and haptics.
deleted 5y ago
[deleted]

disconcision 5y ago

nice to see some data but a little more qualitative analysis seems called for. like, i don't know what the current userbase of chatroulette is like, but from the reputation 10 years ago i would guess that a significant proportion of users are looking for someone they find attractive. if this is the case, and given that people are on-average heterosexual, then it would not be at all surprising that a 2-chain-based system would perform much worse than random. again, i'm not sure that this theory is well-founded, but it would be nice to see some demographically-informed discussion. also, saying that 2-chains didn't work doesn't seem to justify the title. collaborative filtering is broader than a single technique.

mypalmike 5y ago
That's very insightful yet kind of obvious. It would be interesting to see if 3-chain data fixed this, where a central meaningful conversation BC and meaningful conversations AB and CD tended to imply anything about AD.

shemnon42 5y ago

Colour me surprised: I can't read this article over Chick-fil-A wifi.

Straw 5y ago

This doesn't even come close to the power of (not even particularly) modern collaborative filtering algorithms. CF doesn't need to assume that A talks to B well and B talks to C well implies A talks to C well- rather, it typically tries to learn an embedding for A, B, C that predicts their interactions effectively. In the simplest form, this could be applying SVD or nonnegative matrix factorization to the user interaction matrix, truncating to get a low-rank symmetric approximation. More advanced forms would try to predict the interaction as some function of the two user embeddings, possibly even a neural network. This only shows that a particular feature based on 2-chains of interactions isn't useful, whereas the standard approaches I described can try to infer information across any number of connections, and don't necessarily assume that 2-chains as described are indicative of a particular result.

sillysaurusx 5y ago
Skip all that and cut to "a neural network." Presto. I wish I understood this simplicity when I first got into ML. I was always annoyed with claims like "a NN will solve your problems." Oh really? Yeah. Really. You can learn the theory later. Sure, applying SVD or nonnegative matrix factorization to the user interaction matrix, truncating to get a low-rank symmetric approximation makes sense now. (I have to pause and think about it for a little bit, but I do indeed understand each part.) The cool thing was, I didn't need to. The theory helps inform my future ML training runs. But I was able to get most of the magic without knowing a damn thing. Literally just throw Pytorch at the problem and screw around until it starts giving results. You'll be amazed and shocked and delighted how far that gets you. (And all the professional mathematicians and statisticians will be disgusted, but who cares?)
- nestorD 5y ago
  Actually its one of those problems where it is worth not skipping. I have seen people applying neural networks to transform embeddings before doing a final dot-product where it could be proven (and indeed was confirmed in practice) that all of that was equivalent to a direct embedding. Furthermore, SVD type of algorithm are orders of magnitude faster than NN to train (which is why they become used to train word embeddings, see Glove).

tunesmith 5y ago

Is anyone else bothered by that font? The paragraphs feel really stretched out horizontally, and the headings seem one step removed from Comic Sans.

NilsIRL 5y ago

I might have misunderstand the blog post but it seems to me that everything that was tried used a "2-chain" which works on the assumption that the same type of people enjoy talking together. From my personal experience although this can often be true it can also often be completely wrong. Concretely the friends I have are not a copy of myself and I am not necessarily friends with those that are very similar to me. Another way to put it is to put people's personality as a binary, they're either of type I or type II. In the example given, person A and C both had a non-trivial conversation with person B. So person B is type I and person A and C type II. In this case type II can be interpreted as "has non-trivial conversations with type I". There is no reason for person A and C which are both type II to have a non-trivial conversation.

exporectomy 5y ago
Specifically, type I and II are probably male and female. Isn't it immediately obvious that model doesn't make sense if people are talking to the opposite sex? No, Bob and Carlie don't want to have a "meaningful chat" because they both chatted with Alice.
- mattkrause 5y ago
  s/Carlie/Charlie/, I assume? (Carlie, where I grew up, is a woman's name so this threw me for a bit).
  - exporectomy 5y ago
    Yes, Charlie, sorry.
- woko 5y ago
  With this in mind, the conclusion of the blog post is hilarious. Quotes from the blog post: > [The] rough idea is that if, say, Alice spoke to Bob for a long time and then Alice also spoke to Carol for a long time, then Bob and Carol are more likely than not to speak for a long time too. > [...] > The data have clearly shown us that, despite proving a successful technique for many other products across a range of contexts, collaborative filtering simply wasn’t a match for Chatroulette. Basically: 1) make an arbitrary assumption about your data, 2) apply a widely-used recommendation algorithm, 3) notice that the results are really bad, 4) reject the algorithm... rather than the assumption. Admittedly, I don't know whether the algorithm requires this assumption, or whether a slightly different one could be used. For instance, if Alice spoke to Bob and Charlie, then Bob and Charlie don't want to talk together. Or, if Alice spoke to Bob and Charlie, and Carol spoke to Bob, then Carol should be recommended to Charlie, etc.

elliekelly 5y ago

I’m shocked chatroulette even still exists. It makes me wonder what their dedicated user base is like. My gut says it would be either some of the coolest and most interesting people on the internet or some of the creepiest and most shocking trolls.

deleted 5y ago

[deleted]

qPM9l3XJrF 5y ago

>Unfortunately, the results showed a true-negative rate of 78%. i.e. most of the time the model failed to predict when a meaningful conversation was about to occur. >... >The model is significantly worse than a coin-flip. Obviously, this is not good; and given that the majority of conversations on the site are trivial 1. This isn't correct use of the term "true negative". A true negative is when your model predicts a negative, and the label is actually negative. "the model failed to predict when a meaningful conversation was about to occur" would correspond to a false negative (the model predicts negative, the the label is actually positive) 2. "The model is significantly worse than a coin-flip" - Talking about coin flips hardly makes sense if the majority of conversations on the site are trivial. A better way to think about this question would be from a Bayesian perspective. Define a random variable L for linkage. L=True if there exists a user B such users A and C both had non-trivial conversations with B. L=False otherwise. Then define a random variable T for whether the conversation is trivial or not. Then you can compute P(T=False|L=True) and P(T=False|L=False) from your data. It could be that what this article is trying to say is that P(T=False|L=False) > P(T=False|L=True), i.e. the presence of a linkage decreases the probability of a nontrivial conversation. This could occur if, for example, most users of Chatroulette are using it as a dating site, and since heterosexual people outnumber gay people, most nontrivial conversations occur between men and women. So if there's a linkage between A and C, that predicts that A and C are of the same gender, and they won't be interested in talking to each other. Anyway, I didn't read any further than that, but I would suggest just ranking users based on their propensity for nontrivial conversations and connecting users with other users who like to have nontrivial conversations (or perhaps even better, compute the mean/median conversation length for each user to get a sense of their preferred conversation length, then connect users to other users who have a similar preferred conversation length). I suspect there's room to use machine learning to improve Chatroulette a lot, even to the point where it could be the next Facebook. People already pay thousands of dollars to attend Harvard etc. for the networking. You just have to use machine learning tools correctly and find a way to capture some of that value.

deleted 5y ago
[deleted]

dannykwells 5y ago

Release the data and let the community figure this out! This particular analysis is rather limpid and, ummm, misses a key feature of the most common chatroulette use case.

bagels 5y ago

Would be cool if they published some data so that others could give it a try. I bet someone out there can build a model that is better than a coin flip.

robbrown451 5y ago

I think what doesn't work is their assumption that amount of time talking means it was a "good match." It's quite possible that it took some time for someone to figure out the other person was awful, or even that the entire conversation was negative, such as an argument. The algorithm doesn't know that because it is purely based on talk time, rather than taking into account such things as if the users gave each other good ratings.

pyinstallwoes 5y ago
Wouldn't that trend towards optimal over time? Perhaps with a decay or bias for recency. Valuable connection = what you put time into, most latest
bagels 5y ago
The hypothesis they chose is not the most obvious one to attack, and it ignores the most obvious application of collaborative filtering. Their hypothesis is that if A likes B, and A likes C then C will like B. They should instead attack this hypothesis (much more common application of collaborative filtering, think predicting what book you might buy on Amazon): A likes B, A likes C, D likes B -> D has higher probability of liking C as well
- robbrown451 5y ago
  But to them, A liking B and B liking A are one and the same, since all they are measuring is talk time. Regardless, I think it would do much better if it took into account ratings. Even better if they allow you to rate them on several things and then give an overall rating. I personally hate it when things try to learn from my behavior rather than my explicit declarations. For instance, if I click on clickbait (hey, I'm human) it shouldn't assume I actually like that thing or want to see more of it.

shadilay 5y ago

I thought chatroullete was the premier platform in the 'exposing yourself to strangers' industry? Before you needed to take the city bus to experience this level of public exposure but now you can do so from the comfort of your own home.

sesuximo 5y ago

Maybe by and large guys like to talk to girls, but guys don’t like to talk to guys and girls don’t like to talk to girls. This would make a 2 chain model look bad I think. Maybe try a longer chain? i.e. “Bob liked Alice, and Charlie liked both Alice and Diane, so maybe Bob will like Diane” (Ik this is heteronormative — sorry)

DantesKite 5y ago

They could've saved a lot of time if they just read the Terence Tao blog (I'm being a bit cheeky here since it's a rather counterintuitive insight and doesn't necessarily hold for all cases). "If X correlates with Y and Y correlates with Z, then this does not imply X correlates with Z." https://terrytao.wordpress.com/2014/06/05/when-is-correlation-transitive/ https://terrytao.wordpress.com/2014/06/05/when-is-correlatio...

rl3 5y ago

This is Chatroulette. Burn the matchmaking algorithms and keep it true random? For example, if every dating site in existence did this the world would be a better place. Instead, we typically see layers of opaque algorithmic shit governing human interaction. The platform's interests are then in effect prioritized, not the user's. Did anyone ever stop and think: "Is aggressively matching what we believe to be compatible or like-minded people actually a good thing for society? Is it possible that's actually a pretty terrible thing? Do we even know?" There's a certain beauty to true randomness in a matchmaking context. It's a fair playing field for everyone. You start using highly imperfect and frankly arbitrary algorithms to govern who matches with who, and suddenly certain people aren't going to meet certain people for reasons they're probably not even aware of. "Hey, this person could have been your significant other or best friend, but you never met them because some engineers at some tech company decided to get fancy. Fuck you." Or, in so many dating app cases: fancy and greedy. Every time these algorithms are employed, there's always people that fall into the cracks; the dead zones. There they languish. Perhaps months or years later the algorithm is adjusted and affords them a reprieve. Perhaps never. --- This isn't meant to be an indictment of Chatroulette. On the contrary, good on them for reporting negative results. Obviously some filtering must be done for a myriad of reasons, such as curbing abuse. Their stated mission is even to match people "At random." They've thought about this problem a lot more than I have. That said, I thoroughly believe the road to hell is paved by matching users based on metrics such as projected compatibility or similar interests.

zwaps 5y ago
Just to answer that point: people (academically) look at whether it is a good idea to match similar people, what such a social structure means and how preferences for homophily, imposed or intrinsic, may affect the social structure. Do social media websites do think about it? I am sure some of then have. But then, their goals typically differ vastly from what you would consider good for society.
- rl3 5y ago
  Thanks for this. Admittedly my tone was somewhat hyperbolistic, but you bring up an excellent point that I'd like to build off: It pains me to think there's large bodies of deep academic research in the social sciences that could guide certain tech platforms in a much more responsible direction. Instead, the drive for market domination and profit ensures that a mangled society is left in their wake (ironically, economics is also a social science). The hubris certain tech leaders have in making the decisions they do astounds me. Virtually every major tech platform has the money to have a veritable army of social scientists pondering the ethics of anything that company does with its platform. Certainly some of these companies do employ ethicists, though I can't help but feel more often than not it's a dog and pony show.

danbmil99 5y ago

TIL Chatroulette is still a thing

scotty79 5y ago

Why just 2-chain? If you wanted to predict matches and lengthy coversation between random tinder users (if there was no rigid separation for genders) you'd fail in similar fashion. Try 3-chain.

kangnkodos 5y ago

The basic assumption is that if M1 talks with F1 for a long time, and M2 talks with F1 for a long time then M1 and M2 will talk for a long time. Assuming it's based on hetero dating type situations, you can see why this wouldn't work. But there could be many other factors in play also. For example, if the M's above were talkers and the F's above were listeners, of any gender, you can see that two talkers or two listeners would not have a long conversation. But a talker and a listener is a good pair. Instead they should look at this. If M1 talks to F1 and F2 for a long time, and M2 talks to F1 for a long time, then will M2 talk with F2 for a long time? I think this would have a better chance of working.

PaulDavisThe1st 5y ago
I must have missed the mentions of gender in the actual paper. No wait, I didn't. There is no mention of gender in the actual paper. It has nothing to do with dating ... the paper is about chatroulette.
- PoignardAzur 5y ago
  I think GP's point is that chatroulette's article is incomplete because it doesn't analyze how gender impacts chat duration. Like, the first question that came to me reading this was "Is there a pattern of men systematically skipping other men until they're connected with a woman?". It's a little weird the article doesn't ever address that kind of question.

abetusk 5y ago

This is really interesting but I'm highly skeptical. I'm not sure we're given enough information to really get at why they're getting the results they are. They report that if A-C have a non-trivial conversation and B-C have a non trivial conversation, then A-B have a "true-negative rate" of 78% of having a trivial conversation, by which I interpret a 78% over the base line of having A-B have a trivial conversation. The fact that it's so heavily biased against having a non-trivial conversation seems like it should be a huge red flag. How are they conducting the study? Are they informing participants A and B that they're being matched because they've had a non-trivial interaction with C? Is it random? If they are being informed and it turns out that there is a gender component to it, then that might be a big piece of the puzzle in that jealousy and competition start to come into play. Also, if gender is a big factor, then that might inform a "3-chain" collaborative filtering method in that if M0-F0 have a non-trivial interaction, M0-F1 have a non trivial interaction, M1-F0 have a non trivial interaction, then maybe M1-F1 have a higher chance of having a non trivial interaction. I think psychologists call these "confounding factors"? They get a massively strong signal of something but then disregard and and say "doesn't work for us". Something's going on, it's be nice to dig deeper into what it is instead of dismissing it offhand.

captainmuon 5y ago

I could repost what I recent said about onlyfriends here. I think for many people, the whole point of Chatroulette is "show me yours I'll show you mine". That's why the gender thing that others have commented on is not surprising (if Anna chatted with Bob, and Bob with Charlene, it does not follow that Anna wants to chat with Charlene - because people are looking to chat with the opposite sex.) If they want to pivot and sanitize it, they'll loose their core audience. Maybe it would be worthwhile to include two different populations in the model... One that looks for "meaningful conversations" and one that looks for nudity...

a_trn 5y ago

Hi, this is CR founder. Pleasantly surprised to see HN take interest in this. I hope you will not mind if I use this opportunity to say that I am looking for people who can help me on this and other interesting problems. at@chatroulette.com

SergeAx 5y ago

> the rough idea is that if, say, Alice spoke to Bob for a long time and then Alice also spoke to Carol for a long time, then Bob and Carol are more likely than not to speak for a long time too. This is not how collaborative filtration works. If Alice independently spoke to Bob and Carol for a long time, and Dwayne spoke to Bob for a long time, then chances are Dwayne may speak for a long time to Carol.