7 ms·

Twitter is in a dire situation. As a fun project I wrote a Lua - Torch bot to search for certain tweets and hit like on them based on sentiment analysis. I re

Twitter is in a dire situation. As a fun project I wrote a Lua - Torch bot to search for certain tweets and hit like on them based on sentiment analysis. I realized that API query results were mostly news bots, retweet bots, corporate PR bots, social media aggregator platforms like Buffer, and just plain old spam bots. How bad was it? After filtering 1,000 tweets per query, I barely found 10-20 real human users. That signal to noise ratio is dismal, and detrimental to the core product experience. Twitter must be forced to maintain this fake high activity to prop up the share price. BONUS: Guess who else is spamming their post feed: Tumblr. Tumblr didn't allow any adult content or keyword search; since Marissa Mayer took over she seems to have loosened that policy to fluff the numbers. Tumblr today is drowning in porn.

komali2 9y ago
What's crazy is that bots are not violating Twitter TOS. I reported a bunch of fake accounts I found and was told they aren't breaking any rules, even ones just posting spammy links.
- petercooper 9y ago
  Well, yeah. If they're just tweeting and not @ing at other users, they can't really "spam" anyone since only their followers, who choose to see them, will see those messages. Twitter does take a pretty dim view of DM and @reply spam though because it is genuinely annoying and not opt-in.
  - Pxtl 9y ago
    Is follow spam targeted? Because most of my interaction with obvious bots is follow-churn.
ForrestN 9y ago
A lot of Tumblr porn is actually very specifically (and astutely) curated. The few times I've visited, it's actually a much nicer experience than porn websites. If anything, I think Tumblr would be wise to lean into being, among other things, a space for the sharing of erotic content. Maybe there's some spam but it's pretty easy to find your way into non-spam on Tumblr in my experience.
- ionforce 9y ago
  Isn't that a bit thorny since most of the content is copyrighted...
  - devonkim 9y ago
    My understanding of the Tumblr porn content is that they’re usually small clips and I believe under so many seconds doesn’t count under copyright infringement laws. However, nothing would be a problem if it was original content that had waivers and age of consent forms and such on file.
    - kibwen 9y ago
      > I believe under so many seconds doesn’t count under copyright infringement laws Careful here. This appears to be referring to the concept of fair use, but firstly there are multiple criteria used to judge fair use (https://en.wikipedia.org/wiki/Fair_use#U.S._fair_use_factors https://en.wikipedia.org/wiki/Fair_use#U.S._fair_use_factors), secondly these criteria are (intentionally) subjective and up to a judge's interpretation, and thirdly fair use is US-only.
      - wyager 9y ago
        > thirdly fair use is US-only. That seems fine. In the interest of their userbase, companies should aim for the least copyright-damaged user experience possible; this means picking a single country (ideally one with liberal copyright law) and ignoring copyright law in other countries they aren’t based out of. If countries want to force their censorship standards, they can at least be honest about it and block the website (rather than silently deflecting the responsibility of censorship to the website itself).
      - ryanplant-au 9y ago
        The term fair use is US-only but many nations have similar concepts or concepts that overlap with it. Australia's 'fair dealing' policy allows for the use of copyrighted material without seeking approval if its purpose is in satire, research, reviewing, media criticism, or news reporting, for example. What's notable there is that length or amount used aren't as important, which has some positive effects but also some important negative ones (it would not be possible to create Google in Australia because taking the summary snippets and image thumbnails has no legal justification). Interestingly in the last big debate over loosening Australian copyright law and adopting broader fair use, the American MPAA was the biggest funder of opposition efforts.
nkkollaw 9y ago
I've yet to figure out why anyone would consume any kind of content on Twitter. I've used it for a while, and what I got is that it's goos for (and people use it) to spam others about your projects or show off. However, if you try to use it to get news or updates on anything it is the least efficient, most stressful thing I've ever used. I see Twitter as a good tool for outages, natural disasters, and protests. That's pretty much it.
- JoshTriplett 9y ago
  > However, if you try to use it to get news or updates on anything it is the least efficient, most stressful thing I've ever used. Depends heavily on the set of people you follow. I've found it to be a great source of news, and I typically see news show up there hours to days before I see it show up in places like HN.
  - 7dare 9y ago
    For instance, if you follow sports, it's an extremely efficient and direct way of keeping up-to-date with teams and players.
- majos 9y ago
  I use it to follow a relatively small (a few dozen) group of mostly computer scientists with a few cooks/chefs and basketball writers. At this level it takes maybe 1-2 minutes to read an entire day's feed, and I usually end up with a few interesting things to read that I might not have seen otherwise. I have also found it useful at conferences to follow what's happening. I have no idea how people follow more than say a hundred people profitably.
  - nkkollaw 9y ago
    Yes, no idea. What I find awful is that HN and Reddit save me time by sorting out low-quality content, while on Twitter you're the one that has to work hard to do it (if you can), and Twitter itself just adds more noise in the meantime.
- 9y ago
madeofpalk 9y ago
> Tumblr today is drowning in porn. 'Drowning' in porn? In the sense that that's a bad thing?
chimeracoder 9y ago
> BONUS: Guess who else is spamming their post feed: Tumblr. Tumblr didn't allow any adult content or keyword search; since Marissa Mayer took over she seems to have loosened that policy to fluff the numbers. Tumblr today is drowning in porn. ...what? Tumblr was known for porn long before it sold to Yahoo. If anything, Tumblr started cracking down on blogs with adult content afterwards (for example, requiring users to log in before visiting them). There was a huge backlash from artists and bloggers with non-pornographic gay-themed content, because many of them were caught in the ripple effects from these changes. I had friends who worked at Tumblr before 2012, and the running joke was that Tumblr was 50% porn and 30% pictures of cats. (Don't take those figures too seriously, but clearly porn was a large part of Tumblr long before the acquisition). Also, Tumblr definitely did allow adult keywords in searches. I know this because of a rather unfortunate incident at a student hackathon my company sponsored, in which a student thought it would be a "funny" idea to search for a risque phrase when demonstrating his weekend hack (an aggregator of Tumblr posts).
- fwip 9y ago
  The porn bots have only gotten worse, though. In 2012 it was at least mostly human-run porn blogs, and they didn't start following you if you were just a regular blog.
  - kakarot 9y ago
    That's true and it needs to be dealt with, but OP definitely is speaking outside of their domain of knowledge because Tumblr was always drowning in porn. OP didn't specifically mention bots which leads me to believe they are speaking of porn in general. At boarding school many moons ago, we all used Tumblr to get our rocks off because most porn sites were blocked.
- FussyZeus 9y ago
  One of Tumblr's differentiators is allowing adult content. It's very hard to find places friendly to that when you're looking to distribute your artwork and get a following going. Thankfully more and more websites are based out of other countries now and don't need to swim in the puritan current of the US.
wpietri 9y ago
Or it could be that Twitter cares about fake account impact mainly in terms of user experience. If a fake bot likes a tweet from another fake bot, does any human actually care? I used to have a much bigger problem on Twitter with fake accounts and bot likes than I did, but despite my decades of willingness to bitch about spam, I have to concede that they've gotten better lately. There's also a user benefit to letting bots run when they're not hurting anyone: you don't give hints to the spammers on how you're finding and nuking them. Indeed, if you identify them but don't block them, you can get a lot of data on what is spam. For example, on my mail server, I noticed I was getting a lot of dictionary attacks. So I took the hundred most common first names not in use on my domain and fed them all into the spam training system. That means odds are very good that my spam trainer will have seen a piece of spam before they try my actual account.
- TillE 9y ago
  You get to see a flood of spambots if you make the mistake of clicking any of the trending hashtags, but otherwise yeah it's a non-issue for the vast majority of Twitter usage.
- linkmotif 9y ago
  > If a fake bot likes a tweet from another fake bot, does any human actually care? Very nice!
icognito293765 9y ago
I occasionally fetch some works from japanese artists (SFW and not) from twitter since not all of them use pixiv and as far as my browsing experience goes their retweets, conversations, mentions and so on all seem to point to other humans. Likes, Followers and accounts yelling into some hashtags can be dominated bots. But if you look at the feeds of individual people and those that they talk with that's a totally different experience. > Tumblr today is drowning in porn. Tumblr has had NSFW for a long time now. If anything some of it may have migrated to patreon now that it is easier to monetize.
oiuewr88 9y ago
I don't know why this is surprising Large companies almost entirely maintain themselves with some level of clever accounting of their assets, if not outright falsehoods Hollywood studios, MS, Apple, Amazon, GE, GM, Ford, Johnson & Johnson... all companies with well documented histories of exploitative thuggery and shady GAAP practices. Everyone is hiding their numbers more and more these days so they can spin doctor them. Little needs to be said about Wall Street welfare queen status
Pxtl 9y ago
The trick with Twitter is that basically these bots are invisible unless you search hashtags or read the replies to a popular person who is spammed by them. It's not like anybody sane would follow these bots. Twitter's model is basically "shouting at the universe" but if nobody listens it doesn't matter.
- lugg 9y ago
  Problem being investors do listen to these metrics and nobody really knows how bad the noise is in that data being shared. I 100% guarantee twitter knows the exact ratio of real to unreal users and their content. They're also trying to slowly crack down on bots. A year ago they insta locked your account if it looked like automated liking of content. Now they're insta locking if you look like a follow/unfollow bot. Its a balancing act and I feel like its analogous to getting out of extreme debt. They're trying to replace bots as their real user base grows internally.
anigbrowl 9y ago
I would love to get copies of your script if you wanted to email a link. I'm not so interested in bots but I am interested in social network analysis. I don't know about Tumblr wrt porn - my impression is that there's a lot of sexual material there but that it's community-driven rather than commercial, partly due to the demographics of its user base.
marenkay 9y ago
Isn't that kind of expected? The rate of real users pretty much matches the rate of real users on websites. Would also be interesting to run similar analysis in FB, I'd expect similar rates here.
ErikAugust 9y ago
I have spent several years working on a product (https://www.rapidcrowd.co https://www.rapidcrowd.co) that cuts through the noise (bots, fake accounts, inactives) of Twitter to find real users that fit related topics - and your rough estimate of 20 real users in 1000 tweets isn't too far off. For this reason, trending topics and keyword search are essentially hijacked features. However, because I believe there are many useful bots that have organic followings in the millions - I don't believe they need to be simply removed from the ecosystem. Instead, my suggestion would be a 'bots' account type. Some ideas: - A robot version of the 'blue checkmark'. This would allow users to quickly identify a tweet as sent from a bot. - This account type could be linked to a real owners account, much like Twitter apps are. Accounts flagged and failed to register as a bot could be subject to deletion. - Bots would automatically receive low ranking in search queries, and trending topics. Perhaps they would be completely delisted. - Bots cannot follow other users. - Bots cannot tweet at* other users. More extreme: - Bots cannot tweet without some sort of spend. Maybe they can only tweet in some ratio from real Likes they receive. This is a bit extreme, but would mitigate a lot of problems. I really believe a happy median could be found - and currently think that a well-curated Twitter timeline is amazing, but as I stated search results and trending topics are completely broken.
- MBCook 9y ago
  When I first joined Twitter a few years ago I tried the ‘search near me’ feature a few times. Weather bots. For any city within 100 miles of where I am. Plus bots posting job listings. Plus companies posting those same job listings. There was basically no signal to find, it was all noise. The few ‘legitimate’ ones I found were from local PD/FD. I think they just need to ban all bots/automated postings. Or make them filters le and require a $100/mo account and $1/tweet. Something to discourage the absolute garbage.
  - notahacker 9y ago
    But these bots exist because some people actually use Twitter as a newsfeed for that sort of thing. And being partly an RSS substitute is surely part of Twitter's business model these days. Not suggesting that many of these accounts weren't still absolute garbage, but the signal/noise ratio isn't always great with obviously human-run accounts either. The other issue is that the distinction between a bot and a human isn't clear-cut: there's plenty of shades of grey in between something which spits out badly-scraped listings all day and an actual human having noncommercial conversations with Twitter friends. It's more classifying what's bad behaviour (using the pornbot tactic of mass follow/unfollow to attract attention even if you're a human marketer tweeting actual content, and even if you haven't written a script to do it) and what's perfectly acceptable automation like tweet schedulers that could use some work
sbarre 9y ago
Using Twitter for search/discovery by hashtag or keyword is totally pointless. I follow specific people that I've discovered outside of the network, or by referral, and as a result my feed is basically "all signal". This seems like the only reasonable way to use Twitter now..
throwawaypanda 9y ago
>After filtering 1,000 tweets per query, I barely found 10-20 real human users. I am kind of surprised the number of human users is so high. I know a lot of bloggers of various sizes. To the best of my knowledge virtually all of them have hooked up one of the available services to post on their behalf. They either spend a little time scheduling out their tweets for the next week/month then forget about twitter until their schedule runs dry OR they set something up to randomly pick from some pool of blog posts and spam links. Either way they then essentially never go on twitter again once things are up and running. The whole thing is full of bots talking to each other.

komali2 9y ago

What's crazy is that bots are not violating Twitter TOS. I reported a bunch of fake accounts I found and was told they aren't breaking any rules, even ones just posting spammy links.

petercooper 9y ago
Well, yeah. If they're just tweeting and not @ing at other users, they can't really "spam" anyone since only their followers, who choose to see them, will see those messages. Twitter does take a pretty dim view of DM and @reply spam though because it is genuinely annoying and not opt-in.
- Pxtl 9y ago
  Is follow spam targeted? Because most of my interaction with obvious bots is follow-churn.

ForrestN 9y ago

A lot of Tumblr porn is actually very specifically (and astutely) curated. The few times I've visited, it's actually a much nicer experience than porn websites. If anything, I think Tumblr would be wise to lean into being, among other things, a space for the sharing of erotic content. Maybe there's some spam but it's pretty easy to find your way into non-spam on Tumblr in my experience.

ionforce 9y ago
Isn't that a bit thorny since most of the content is copyrighted...
- devonkim 9y ago
  My understanding of the Tumblr porn content is that they’re usually small clips and I believe under so many seconds doesn’t count under copyright infringement laws. However, nothing would be a problem if it was original content that had waivers and age of consent forms and such on file.
  - kibwen 9y ago
    > I believe under so many seconds doesn’t count under copyright infringement laws Careful here. This appears to be referring to the concept of fair use, but firstly there are multiple criteria used to judge fair use (https://en.wikipedia.org/wiki/Fair_use#U.S._fair_use_factors https://en.wikipedia.org/wiki/Fair_use#U.S._fair_use_factors), secondly these criteria are (intentionally) subjective and up to a judge's interpretation, and thirdly fair use is US-only.
    - wyager 9y ago
      > thirdly fair use is US-only. That seems fine. In the interest of their userbase, companies should aim for the least copyright-damaged user experience possible; this means picking a single country (ideally one with liberal copyright law) and ignoring copyright law in other countries they aren’t based out of. If countries want to force their censorship standards, they can at least be honest about it and block the website (rather than silently deflecting the responsibility of censorship to the website itself).
    - ryanplant-au 9y ago
      The term fair use is US-only but many nations have similar concepts or concepts that overlap with it. Australia's 'fair dealing' policy allows for the use of copyrighted material without seeking approval if its purpose is in satire, research, reviewing, media criticism, or news reporting, for example. What's notable there is that length or amount used aren't as important, which has some positive effects but also some important negative ones (it would not be possible to create Google in Australia because taking the summary snippets and image thumbnails has no legal justification). Interestingly in the last big debate over loosening Australian copyright law and adopting broader fair use, the American MPAA was the biggest funder of opposition efforts.

nkkollaw 9y ago

I've yet to figure out why anyone would consume any kind of content on Twitter. I've used it for a while, and what I got is that it's goos for (and people use it) to spam others about your projects or show off. However, if you try to use it to get news or updates on anything it is the least efficient, most stressful thing I've ever used. I see Twitter as a good tool for outages, natural disasters, and protests. That's pretty much it.

JoshTriplett 9y ago
> However, if you try to use it to get news or updates on anything it is the least efficient, most stressful thing I've ever used. Depends heavily on the set of people you follow. I've found it to be a great source of news, and I typically see news show up there hours to days before I see it show up in places like HN.
- 7dare 9y ago
  For instance, if you follow sports, it's an extremely efficient and direct way of keeping up-to-date with teams and players.
majos 9y ago
I use it to follow a relatively small (a few dozen) group of mostly computer scientists with a few cooks/chefs and basketball writers. At this level it takes maybe 1-2 minutes to read an entire day's feed, and I usually end up with a few interesting things to read that I might not have seen otherwise. I have also found it useful at conferences to follow what's happening. I have no idea how people follow more than say a hundred people profitably.
- nkkollaw 9y ago
  Yes, no idea. What I find awful is that HN and Reddit save me time by sorting out low-quality content, while on Twitter you're the one that has to work hard to do it (if you can), and Twitter itself just adds more noise in the meantime.
9y ago

madeofpalk 9y ago

> Tumblr today is drowning in porn. 'Drowning' in porn? In the sense that that's a bad thing?

chimeracoder 9y ago

> BONUS: Guess who else is spamming their post feed: Tumblr. Tumblr didn't allow any adult content or keyword search; since Marissa Mayer took over she seems to have loosened that policy to fluff the numbers. Tumblr today is drowning in porn. ...what? Tumblr was known for porn long before it sold to Yahoo. If anything, Tumblr started cracking down on blogs with adult content afterwards (for example, requiring users to log in before visiting them). There was a huge backlash from artists and bloggers with non-pornographic gay-themed content, because many of them were caught in the ripple effects from these changes. I had friends who worked at Tumblr before 2012, and the running joke was that Tumblr was 50% porn and 30% pictures of cats. (Don't take those figures too seriously, but clearly porn was a large part of Tumblr long before the acquisition). Also, Tumblr definitely did allow adult keywords in searches. I know this because of a rather unfortunate incident at a student hackathon my company sponsored, in which a student thought it would be a "funny" idea to search for a risque phrase when demonstrating his weekend hack (an aggregator of Tumblr posts).

fwip 9y ago
The porn bots have only gotten worse, though. In 2012 it was at least mostly human-run porn blogs, and they didn't start following you if you were just a regular blog.
- kakarot 9y ago
  That's true and it needs to be dealt with, but OP definitely is speaking outside of their domain of knowledge because Tumblr was always drowning in porn. OP didn't specifically mention bots which leads me to believe they are speaking of porn in general. At boarding school many moons ago, we all used Tumblr to get our rocks off because most porn sites were blocked.
FussyZeus 9y ago
One of Tumblr's differentiators is allowing adult content. It's very hard to find places friendly to that when you're looking to distribute your artwork and get a following going. Thankfully more and more websites are based out of other countries now and don't need to swim in the puritan current of the US.

wpietri 9y ago

Or it could be that Twitter cares about fake account impact mainly in terms of user experience. If a fake bot likes a tweet from another fake bot, does any human actually care? I used to have a much bigger problem on Twitter with fake accounts and bot likes than I did, but despite my decades of willingness to bitch about spam, I have to concede that they've gotten better lately. There's also a user benefit to letting bots run when they're not hurting anyone: you don't give hints to the spammers on how you're finding and nuking them. Indeed, if you identify them but don't block them, you can get a lot of data on what is spam. For example, on my mail server, I noticed I was getting a lot of dictionary attacks. So I took the hundred most common first names not in use on my domain and fed them all into the spam training system. That means odds are very good that my spam trainer will have seen a piece of spam before they try my actual account.

TillE 9y ago
You get to see a flood of spambots if you make the mistake of clicking any of the trending hashtags, but otherwise yeah it's a non-issue for the vast majority of Twitter usage.
linkmotif 9y ago
> If a fake bot likes a tweet from another fake bot, does any human actually care? Very nice!

icognito293765 9y ago

I occasionally fetch some works from japanese artists (SFW and not) from twitter since not all of them use pixiv and as far as my browsing experience goes their retweets, conversations, mentions and so on all seem to point to other humans. Likes, Followers and accounts yelling into some hashtags can be dominated bots. But if you look at the feeds of individual people and those that they talk with that's a totally different experience. > Tumblr today is drowning in porn. Tumblr has had NSFW for a long time now. If anything some of it may have migrated to patreon now that it is easier to monetize.

oiuewr88 9y ago

I don't know why this is surprising Large companies almost entirely maintain themselves with some level of clever accounting of their assets, if not outright falsehoods Hollywood studios, MS, Apple, Amazon, GE, GM, Ford, Johnson & Johnson... all companies with well documented histories of exploitative thuggery and shady GAAP practices. Everyone is hiding their numbers more and more these days so they can spin doctor them. Little needs to be said about Wall Street welfare queen status

Pxtl 9y ago

The trick with Twitter is that basically these bots are invisible unless you search hashtags or read the replies to a popular person who is spammed by them. It's not like anybody sane would follow these bots. Twitter's model is basically "shouting at the universe" but if nobody listens it doesn't matter.

lugg 9y ago
Problem being investors do listen to these metrics and nobody really knows how bad the noise is in that data being shared. I 100% guarantee twitter knows the exact ratio of real to unreal users and their content. They're also trying to slowly crack down on bots. A year ago they insta locked your account if it looked like automated liking of content. Now they're insta locking if you look like a follow/unfollow bot. Its a balancing act and I feel like its analogous to getting out of extreme debt. They're trying to replace bots as their real user base grows internally.

anigbrowl 9y ago

I would love to get copies of your script if you wanted to email a link. I'm not so interested in bots but I am interested in social network analysis. I don't know about Tumblr wrt porn - my impression is that there's a lot of sexual material there but that it's community-driven rather than commercial, partly due to the demographics of its user base.

marenkay 9y ago

Isn't that kind of expected? The rate of real users pretty much matches the rate of real users on websites. Would also be interesting to run similar analysis in FB, I'd expect similar rates here.

ErikAugust 9y ago

I have spent several years working on a product (https://www.rapidcrowd.co https://www.rapidcrowd.co) that cuts through the noise (bots, fake accounts, inactives) of Twitter to find real users that fit related topics - and your rough estimate of 20 real users in 1000 tweets isn't too far off. For this reason, trending topics and keyword search are essentially hijacked features. However, because I believe there are many useful bots that have organic followings in the millions - I don't believe they need to be simply removed from the ecosystem. Instead, my suggestion would be a 'bots' account type. Some ideas: - A robot version of the 'blue checkmark'. This would allow users to quickly identify a tweet as sent from a bot. - This account type could be linked to a real owners account, much like Twitter apps are. Accounts flagged and failed to register as a bot could be subject to deletion. - Bots would automatically receive low ranking in search queries, and trending topics. Perhaps they would be completely delisted. - Bots cannot follow other users. - Bots cannot tweet at* other users. More extreme: - Bots cannot tweet without some sort of spend. Maybe they can only tweet in some ratio from real Likes they receive. This is a bit extreme, but would mitigate a lot of problems. I really believe a happy median could be found - and currently think that a well-curated Twitter timeline is amazing, but as I stated search results and trending topics are completely broken.

MBCook 9y ago
When I first joined Twitter a few years ago I tried the ‘search near me’ feature a few times. Weather bots. For any city within 100 miles of where I am. Plus bots posting job listings. Plus companies posting those same job listings. There was basically no signal to find, it was all noise. The few ‘legitimate’ ones I found were from local PD/FD. I think they just need to ban all bots/automated postings. Or make them filters le and require a $100/mo account and $1/tweet. Something to discourage the absolute garbage.
- notahacker 9y ago
  But these bots exist because some people actually use Twitter as a newsfeed for that sort of thing. And being partly an RSS substitute is surely part of Twitter's business model these days. Not suggesting that many of these accounts weren't still absolute garbage, but the signal/noise ratio isn't always great with obviously human-run accounts either. The other issue is that the distinction between a bot and a human isn't clear-cut: there's plenty of shades of grey in between something which spits out badly-scraped listings all day and an actual human having noncommercial conversations with Twitter friends. It's more classifying what's bad behaviour (using the pornbot tactic of mass follow/unfollow to attract attention even if you're a human marketer tweeting actual content, and even if you haven't written a script to do it) and what's perfectly acceptable automation like tweet schedulers that could use some work

sbarre 9y ago

Using Twitter for search/discovery by hashtag or keyword is totally pointless. I follow specific people that I've discovered outside of the network, or by referral, and as a result my feed is basically "all signal". This seems like the only reasonable way to use Twitter now..

throwawaypanda 9y ago

>After filtering 1,000 tweets per query, I barely found 10-20 real human users. I am kind of surprised the number of human users is so high. I know a lot of bloggers of various sizes. To the best of my knowledge virtually all of them have hooked up one of the available services to post on their behalf. They either spend a little time scheduling out their tweets for the next week/month then forget about twitter until their schedule runs dry OR they set something up to randomly pick from some pool of blog posts and spam links. Either way they then essentially never go on twitter again once things are up and running. The whole thing is full of bots talking to each other.