8 ms·
X updates its Terms to prohibit crawling/scraping of its data
- o0banky0o 3y ago[flagged]
- deleted 3y ago[deleted]
- LinuxBender 3y agoMy scripts have not opted-in to accepting any terms of use nor have they signed any contracts. [Edit] I should add that nobody has paid me for accessing my site that has a ToS/AUP requiring everyone to pay me 10% of their monthly income.
- flutas 3y agoHence why you need to sign in to access content from most feeds at least, so that you've agreed to the ToS.
- BitwiseFool 3y agoWhenever I visit someone's Twitter profile without being logged in, I see an assortment of years old tweets and nothing current. Do other people experience this too? I assumed that I was being A/B tested with some limited view after the forced login modal stopped appearing.
- asicsp 3y agoDoes it happen on all profiles? Perhaps some of them are using features likes pins/highlights to show curated tweets?
- BitwiseFool 3y agoYes, it happens to all profiles I visit and I can rule out the pins/highlights feature. There's a weatherman in Texas who is a prolific tweeter and his page shows a random assortment of years old retweets (~2015), some tweets from last month, and no current weather forecasts. They don't change between page visits and none of them are in chronological order. What do you see when you visit https://twitter.com/wfaaweather https://twitter.com/wfaaweather ?
- asicsp 3y agoYeah, it shows some recent tweets mixed with old ones (including 2015)...
- cma 3y agoI'm seeing it too, not pinned/highlighted and different when logged in.
- duskwuff 3y ago> Whenever I visit someone's Twitter profile without being logged in, I see an assortment of years old tweets and nothing current. Do other people experience this too? Yes. Views of logged-out profiles display statuses sorted by most to least likes, rather than chronologically. It's incredibly dumb, and makes many accounts (like government announcement accounts - @NWS et al) useless while logged out.
- BitwiseFool 3y agoWhat a horrible decision. With this change I get the sense that the user has a dead account, or the page is broken. Definitely not some compelling "top hits" compilation that would make me want to sign up for 'X'. What a disaster.
- sangnoir 3y ago> What a disaster Some people like(d) to compare Musk to Jobs; but when it comes to Product instincts - they couldn't be further apart. Musk is the anti-Steve-Jobs, his ability to make the worst product decisions is preternatural.
- Fnoord 3y agoThey changed this a couple of months ago. Twitter is no longer chronological. On top of that Blue get promoted, having more reach. I deleted my Twitter account before all that, all too often I cannot even see Twitter deeplinks anymore.
- themagician 3y agoMight want to fix that.
- meepmorp 3y agoWhy?
- mtmail 3y agoDo the scripts respect the https://twitter.com/robots.txt https://twitter.com/robots.txt ? It contains User-agent: * Disallow: /
- jraph 3y agoHardly legally enforceable, no? robots.txt is nothing more than a gentle request. edit: even Twitter don't really have much faith in those rules: # Every bot that might possibly read and respect this file # ======================================================== User-agent: * Disallow: / "might possibly". lol. Really sounds like "probably won't work but can't hurt, eh?"
- bayindirh 3y agoI love some people “just declare” something legally unenforceable if that thing is not something they like.
- falcolas 3y ago> Hardly legally enforceable, no? Up to the lawyers and the courts. And I can definitely see how it could be interpreted to be a legitimate "bots are not authorized to view this site" directive since robots.txt is a standard which is respected across the industry.
- jraph 3y agoI could write that people wearing yellow pants should not view my website on my website prominently. I sure would hope it would not cut it in courts.
- doctorpangloss 3y agoWhy not? If you wrote that on your cake shop’s banner, you’d prevail with more egregious policies.
- notjoemama 3y ago
- ilyt 3y agoWrote a TOS that requires sites opting out of scraping to pay you for non-scraped link
- deng 3y agoYou don't need to opt-in to terms of services. The service provider can simply refuse to service you if you violate the ToS. Violating ToS is not a crime. However, if you actively try to circumvent this refusal of the service provider, this could very well be a crime, depending on what you are doing.
- maxwell 3y ago"Let them sue" https://arstechnica.com/tech-policy/2023/09/musks-unpaid-bills-dozens-of-lawsuits-try-to-force-twitter-aka-x-to-pay-up https://arstechnica.com/tech-policy/2023/09/musks-unpaid-bil...
- coding123 3y agoFrom what I've heard many of those lawsuits have let him settle for less, including the rent
- itsoktocry 3y ago>From what I've heard many of those lawsuits have let him settle for less, including the rent Of course, that's how settlements work. Elon literally has more wealth to bully people than anyone else on the planet, and is willing to do things out of spite.
- grumple 3y agoWon't this robots.txt update stop major search engines from indexing Twitter? Seems like a great way to kill revenue as Twitter vanishes from search results.
- jraph 3y agoI wonder if search engines actually use rules for Googlebot when parsing some (specific?) robots.txt. "Googlebot" might as well be the "Mozilla/5.0" of search index bot user agents.
- wiktor-k 3y agoTwitter is benefiting from network effects by now. I'm not even sure when have I encountered a tweet in search results last time.
- deleted 3y ago[deleted]
- jeroenhd 3y agoI encounter tweets all the time when I'm trying to find out more about recent events.
- thisismyhna 3y agoDoes twitter rely on that? Most of the time I go to twitter is from news stories, comments or other social media.
- sundaeofshock 3y agoAny revenue damage from this will be faulty minor compared to the damage that has already been done.
- dahfizz 3y agoThey have a section in the robots.txt that allows Google. They must think no other search engine matters.
- 0cf8612b2e1e 3y agoHow does this sit after the Linked In scraping decision?
- rchaud 3y agoCould someone keep scraping data and claim it's a negotiating tactic to reduce the cost of the API? Musk is a businessman, he'll get it.
- freitzkriesler2 3y agoIIRC there have been supreme court rulings about the legality of web scraping but they were all publicallu facing services like Amazon or other e-commerce sites. Twitter is locked behind a login wall now so the rules in theory might be different. Of course, bots can still get in through the human door.
- atemerev 3y agoAnd yet, Twitter makes it easy to mass register accounts in bulk.
- thih9 3y agoHow? 1-2 years ago they used to automatically suspend new accounts without a phone number within 1h after registration; after that you had to go through a manual account recovery process. Did something change here?
- atemerev 3y agoYes, they stopped requiring phone numbers and opted instead for email confirmation sequence, which can be relatively easily automated.
- Fnoord 3y agoBurner phone number.
- cmeacham98 3y agoMost people would not consider it easy to acquire burner phones in bulk (especially with the restriction they've never been used to sign up for Twitter).
- Fnoord 3y agoI wrote 'Burner phone number'; not burner phone. If the phone number has been previously in use, you can probably hijack the account.
- Karunamon 3y agoMaybe to register, but in my experience (trying to set up an official account for a project) they are quickly locked the moment you try to actually do anything with them, requiring a email link followed by a unique multi-step captcha.
- thih9 3y agoIf anything, I’m surprised it wasn’t there already.
- moomoo11 3y agoSad that people hate use Twitter. Just don’t use it and ignore it. It’s just an app. Now that it has a login gate it’s even better to avoid.
- cmrdporcupine 3y agoI hope services like HN and others rapidly go through their database of links that point to Twitter and mirror the content off into actually-public archives. Because it's only a matter of time before work-arounds like Nitter and others go belly-up. There's unfortunately massive amount of content produced and reproduced by people who imagined they were doing so on a public forum. But it's increasingly going behind a wall of 403s and obfuscation. For a few days in July non-Twitter members couldn't see anything there. Now they can only see the primary post and not the threads. That's Musk's right to do that, but the right thing to do is for people to clue in that this isn't a public resource, and if they want their content to be broadly available they'll need to find other places to put it. And not just for posterity, but for the present as well. One thing I don't understand is how the advertisers who nominally fund Twitter (or did) are fine with this. Seems like deliberately kneecapping the reach and impressions.
- dehrmann 3y agoWhat is his obsession with bots? Is it really costing them that much? Maybe shut off more expensive discovery features for bot users?
- temporalparts 3y agoAs long as it's not behind a login, and is publicly available, I think there's US precedent that this is unenforceable: https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
- dplgk 3y agoThey ended up settling https://www.natlawreview.com/article/court-finds-hiq-breached-linkedin-s-terms-prohibiting-scraping-mixed-ruling-declines https://www.natlawreview.com/article/court-finds-hiq-breache...
- kube-system 3y agoNot enforceable in a US court but it is enforceable through technical means.
- lossolo 3y ago> but it is enforceable through technical means What you mean by that? Currently it's not enforceable by any technical means, as I can scrape Twitter without any limitations.
- kube-system 3y agoif(requests > bigNumber) banHammer(); Or whatever alternative bot detection they want to do. There are plenty of solutions for these types of problems.
- HAL3000 3y ago> if(requests > bigNumber) banHammer(); If requests from who? or what? Account? IP? Because no one is doing it from one account or one IP. > There are plenty of solutions for these types of problems. Actually not a lot, it's a really hard problem to solve, unless you're referring to unsophisticated scrapers that have no idea what they're doing and are following some online tutorials on how to scrape Twitter. Out of desperation, they even implemented blanket bans on entire legitimate mobile carriers in some EU countries, but it didn't have a lot of impact on the business of Twitter scraping.
- gameoverhumans 3y ago"its data" - nothing on Twitter, except the pathetic memes from a billionaire wannabe edgelord should "belong" to Twitter. It's all user-generated content. I know it'll never happen because we live in the reality where everything must be awful all the time, but I'll continue dreaming of a day where large internet platforms are regarded and regulated as public utilities.
- safety1st 3y agoI've come to feel that regulating a platform like Twitter as a public utility could be a well intentioned but unfathomably evil act. Tweeting is the ultimate example of an activity that's designed to be engaging but is otherwise awful. I think it may be the very nature of popularity-driven ultra-short-form content that it encourages online mobs, cancel culture, dunks and hot takes. If there's even a chance this is true then it would be criminal to calcify this medium in its currently known form as a permanent pillar of our public square - which regulating it as a public utility would do. We could be inflicting suffering on future generations that might have escaped it. No. It's better to hope that we discover better ways of communicating, and tweeting becomes irrelevant and dies.
- mschuster91 3y ago> I think it may be the very nature of popularity-driven ultra-short-form content that it encourages online mobs, cancel culture, dunks and hot takes. What "cancel culture"? Who got "cancelled" for real? Literally everyone of those whining about "cancel culture" who hasn't actually broken laws or is being suspected having done so (Tate) is still running around and making tons of money whining around to everyone.
- LightHugger 3y agoFunnily enough people on the right are largely immune to cancel culture because they have their own support structures. Meanwhile innocent people on the left get their lives ruined by cancel culture twitter zombies that think harassing and stalking people makes them righteous. And just the usual, people who want to be assholes and just pick the 'socially acceptable' way to hurt other people. "The Left" is extremely cannibalistic among those who subscribe to a central "Left" dogma.
- MightyBuzzard 3y ago[dead]
- nvr219 3y ago“I wonder who that’s for.”