15 ms·
"If the average length of a tweet (minus headers) is ~100 bytes of text, that’s only 50GB. You could fit it on a USB stick." https://twitter.com/elonmusk/status
by ceejayoz 4y ago
"If the average length of a tweet (minus headers) is ~100 bytes of text, that’s only 50GB. You could fit it on a USB stick." https://twitter.com/elonmusk/status/1534939289653592065 https://twitter.com/elonmusk/status/1534939289653592065
Having worked with the Twitter API, I think he's rather underestimating how much volume this is gonna wind up being, the amount of processing it'll entail, and the difficulties of determining "this is a bot" from just the Tweet data.
- d23 4y agoIt's imbecilic and baffling to think this is all one would need. Edit: to be clear, I'm saying this about what Elon said, not you.
- rvense 4y ago"Tech world in shock after self-absorbed middle aged man speaks before thinking"
- mcintyre1994 4y agoI'm curious how he thinks you can detect bots by analysing just the tweet text. How would you even tell the difference between a retweet and a bunch of bots copy-pasting stuff without headers?
- micromacrofoot 4y agoProbably a couple hundred metadata fields for each tweet, right? he's at least one order of magnitude off right out of the gate
- mikkergp 4y agoAlso, Isn’t the metadata the important part for determine not vs non bot? I don’t think bots start there tweet, “bot here: “
- sudosysgen 4y agoNot to mention the graph that you'll have to build between messages and users not trivially related. It's going to be far from trivial. If it was easy to find bots everyone would do it.
- tmp_anon_22 4y agoThe metadata (ip, client headers, timestamps, etc) alone is going to be over 100 bytes. How pretty do people think real world data sets are? Now scale that to billions of records. What a terrible cultural icon Musk has become.
- deleted 4y ago[deleted]
- meatsauce 4y agoGood thing we have icons like AOC and George Floyd to put Musk's success in perspective. We should all aspire to be fentanyl dropping, pregnant women torturing street trash. We should all aspire to accomplish great things like AOC when she ...uh she did ...um ...drop Adderall and start a podcast about putting on makeup! Yeah! That podcast alone employs 50,000 engineers and has saved or created 100,000,000 jobs a month, at least.
- deleted 4y ago[deleted]
- falcolas 4y agoHaving worked with the Twitter firehose, I can say with some certainty that the volume of data is measured in TBs per day. Sifting through a subset of the data to make correlations is working with PBs of data. Source: Former employee of one of the few companies who got the "push" firehose and not just the API.
- gregsadetsky 4y agoAgreed. Back in a distant 2011: "1 terabyte per day of data [...] - that is how much is produced by the firehose of 250 million tweets a day" [0] i.e., ~4kb per tweet. So, 2ish Tb/day of text + metadata at current rates? [0] https://techcrunch.com/2011/12/07/datasift-historical-tweets/ https://techcrunch.com/2011/12/07/datasift-historical-tweets...
- sudosysgen 4y agoDefinitely much more. I guarantee the amount of metadata per tweet increased massively, and also now there's things like pictures and videos. More like 40-60TiB if I had to guess.
- gregsadetsky 4y agoRight, I was excluding media from the equation. This other 2010 source [0] said "Twitter generates 8 terabytes of data per day". And that's 12 years ago... [0] https://techcrunch.com/2010/09/17/twitter-seeing-6-billion-api-calls-per-day-70k-per-second/ https://techcrunch.com/2010/09/17/twitter-seeing-6-billion-a...
- deleted 4y ago[deleted]
- abathur 4y agoEven if you get reliable bot ID, there's a difference between a zero- or negative-value bot that just spams or tries to inflame social tensions and positive-value bots that automatically propagate things like weather or traffic updates. Some accounts are a mix of human and "bot" updates--and you can get a very different impression of their nature depending on your sample. https://twitter.com/JeffLindner1 https://twitter.com/JeffLindner1, for example, posts automatic alerts from the Harris County flood warning system (such as https://twitter.com/JeffLindner1/status/1529386618938920960 https://twitter.com/JeffLindner1/status/1529386618938920960). If you look at his account when it isn't pouring outside, you'll see human updates and retweets. If you look during a three-day downpour, you may see dozens of automatic tweets per human post.
- nightski 4y agoIt feels like bots should be treated differently entirely than users. They should not look like a normal user and it should be completely up to the user to turn bots on/off or subscribe to specific ones. Maybe with it's own UI and feed entirely. NOTE: For the ones operating legitimately I mean.
- ceejayoz 4y ago(Legitimate) bot accounts are labeled. https://twitter.com/TwitterSupport/status/1494041387880304645 https://twitter.com/TwitterSupport/status/149404138788030464...
- nightski 4y agoOh good to know. I think it's amusing that I have been using Twitter for over a decade and never knew this. Maybe it's because there are so few legitimate bots. Although I will say that UI hint is extremely subtle.
- abathur 4y agoI hadn't heard of it either. Only for the last ~9 months, it seems. And from the comments, it required proactive action.
- elisharobinson 4y agoi dont think so , why do you need a bot in the first place ... its probably to run a trend of some sort, correlate that with when the users were created should give a good base .. current MAU's stand around 0.46 B users , 50 M tweets on the user base would avg 10 tweets a month that sounds right ... if new account which stays under this limit would result in a False negative
- ceejayoz 4y agoI can assure you bot detection is more than "how many posts a month?"
- deleted 4y ago[deleted]
- wp381640 4y agoI found this tweet very naive and an indication that Musk really doesn't know what he's getting into. I don't see how he thinks tweet content alone will be enough to verify the 5% of monthly monetizable users The other poor indicator was his tweet about sampling _100_ of his own replies to detect bots. That's a horribly small sample and horrible selection bias The firehose is terabytes a day, add storage + indexes and it's pretty far from "fitting on an USB stick" in order to analyze.
- oefrha 4y agoWhy would you assume ignorance instead of malice when a very smart person advised by smart people with domain knowledge makes an obviously nonsensical claim, while billions are on the line?
- shp0ngle 4y agoBecause it’s Elon Musk.
- Barrin92 4y agobecause Musk isn't some technical genius with amazing domain knowledge, he's the world's best salesman with a unique talent of attracting followers.
- coffeeblack 4y agoThere are lots of great technical interviews with him on Yt that show the opposite is true.
- deleted 4y ago[deleted]
- waffleiron 4y agoCould you link a few? I'd be interested, cause it's not the feeling I get from him.
- lentil_soup 4y agoGenuinely curious, would a day of tweets be enough of a sample to do a bot count analysis? Would it be the correct type of dataset?
- srvmshr 4y agoAs a small company working with Twitter firehose covering 10% of traffic just in Japan, we are clocking between 40-45GB of JSON data per day. (i.e about 1.5 TB per month)
- stingraycharles 4y agoAs someone who worked with Twitter’s APIs a lot, and the metric shitton of metadata that is sent with every tweet, him thinking it’s 100 bytes is hilarious. I know Twitter typically does assessments of a client’s infrastructure before providing them access to the Firehose, to ensure they are actually able to keep up with the stream. It’s that much data. He’ll likely have to sample the data anyway before he can do any meaningful analysis, so the whole exercise is pointless.
- mgiannopoulos 4y agoHis reply is obviously a joke as he was replying to a joke.
- csmpltn 4y agoThe core product is a 280-character buffer. The hypothesis is that sampling those buffers at scale should be enough to detect significant bot activity on the platform, without needing any additional metadata. That may very well be the case, so why the snide?
- rkargon 4y agoMy guess is that that will work to his advantage -- since it's so hard to tell who is a bot from the raw data, he'll be able to massage his calculations and come up with pretty much any figure he wants, allowing him to claim that Twitter has significantly more bots than claimed