12 ms·
Meta was scraping sites for years while fighting the practice
- leobg 4y agoNot just Facebook. I was recently banned from NerdyData.com for scraping their data, even though my plan said “unlimited” downloads. They said I can download as much as I want, but I’ll have to do it manually or pay extra for their API. Luckily for them, the sites from which they source their data didn’t enforce such a rule, I suppose.
- jeoqn 4y ago…and? Of course everybody does whatever they can get away with.
- seydor 4y agoThen we should all be able to get away with it right? Can't say there is no problem here
- radarsat1 4y agoInteresting perspective. So in your view, no person or company with integrity exists? And no one lacking it should be called out?
- aulin 4y agointegrity in capitalism only exists if some soverreign entity enforces or if it's the more profitable path (e.g. to avoid PR scandals)
- the_third_wave 4y ago"in capitalism"... What about integrity in socialism, communism, fascism, confusianism, feodalism and all the others? In other words drop the silly "in capitalism", it is as vapid as the old-style "in Soviet Russia" reversals.
- hdjjhhvvhga 4y agoAs a person who lived in both I can tell you in communism it's far worse, at least as far as personal integrity is concerned. In capitalism, maximum I can lose for speaking up is my job. In communism, as in all authoritarian/totalitarian regimes, it was - and still is - normal for people to lose their freedom or even life. So as much as I hate corporate bullshit, their power grab, lobbying politicians etc., I would still prefer to live in a "capitalist" rather than a "communist" country.
- themitigating 4y agoCommunism and authoritarian is the key. It's not the economic system that's the problem.
- the_third_wave 4y agoTell me then, how do you enforce the essence of a communist economic system, this being communal ownership of the means of production, without an authority to tell Mr. Smartpants who just invented a better mousetrap that he has to give it to the commune instead of producing it himself? Nope, Communism is more than just an economic system. It is an ideology which strives to create a "better man" who can create a "better society" where human "vices" like greed and avarice no longer play a role. In this it closely resembles religious orders which strive to create the same through other means. Capitalism is 'just an economic system' which can be implemented in many ways ranging from laissez-faire libertarian-style capitalism to state capitalism as practised in China. It does not come with the large ideological baggage Communism carries since it doesn't need to - leave people to their own devices and they'll end up doing something resembling a form of capitalism at a large scale while practising something resembling communism at a small - family and neighbourhood - scale. This model works fine since there is no ideological drive to force out the small-scale 'communism' in families and neighbourhoods whereas a communist society needs to do away with the large-scale capitalism. Communism and authoritarianism are indelibly linked.
- Karellen 4y ago> In communism, as in all authoritarian/totalitarian regimes, WTF? Communism is an economic model, not a leadership selection model. Communism is diametrically opposed to capitalism, not to democracy. (It always baffles me when a communist authoritarian dictatorship enriches wealthy plutocrats while the workers starve, it's always the communism part that gets blamed, never the authoritarian dictatorship. But when a capitalist authoritarian dictatorship enriches wealthy plutocrats while the workers starve, it's never the capitalism that gets blamed. No, then it's the authoritarian dictatorship that's the problem. How about, just maybe, it's dictators and wealthy plutocrats that are the biggest problem for the working class?)
- jeoqn 4y agoWe shouldn’t be surprised by lack of integrity. This is not news.
- Veen 4y agoLack of integrity in general is not news, but it is valuable to know about specific examples. Not least because it undermines Facebook’s PR fluff.
- jeoqn 4y ago[flagged]
- fleddr 4y agoWe should be. This fatalism is exactly the problem. You see poor behavior, slap the label "business" on it, and all of a sudden it is expected, accepted and even defended.
- wlll 4y agoI imagine that inventing a lack of integrity for people is a great way of justifying terrible behaviour. "No-one else has integrity so I'm justified by also not doing so". It doesn't even have to be true as long as you believe it.
- JohnFen 4y agoThat it isn't surprising doesn't mean we shouldn't call it out.
- noizejoy 4y ago> no person or company with integrity exists? … probably not in the most powerful x percent.
- Dalewyn 4y agoA person with integrity is either a saint or a liar. A company with integrity has an amazing marketing department.
- deleted 4y ago[deleted]
- Karellen 4y agoIt might be worth considering GPs comment as a case of projection, rather than a claim about the world to be rebutted. After all, they are part of the "everyone" they talk about. And the "everyone else is just as bad as I am, really" (also commonly phrased "we are not so different, you and I") is a classic asshole refrain. When someone tells you who they are, believe them.
- timeon 4y agoWe do not need to downplay the hypocrisy even if it was obvious.
- tukantje 4y ago...and? Your point? Putting it to spotlight is an effort to make sure they can't get away with it. Even in a scenario where you say "of course they do it", jumping to the conclusion of "we should not report on it" is ridiculous.
- JohnFen 4y agoNot nearly everyone. Plenty of people follow ethical codes.
- creshal 4y agoRules for thee, but not for me, as always. All big established companies want to ensure that nobody coming after them has a fair chance.
- polotics 4y agoPeter Thiel's infamous "competition is for losers" indeed comes to mind.
- dbuder 4y agoBe the biggest hypocrite possible seems to be his motto
- scotty79 4y ago"competition is for losers" is really honest and true. Nothing hypocritical about it.
- rdedev 4y agoIn that framing, everything else outside the company, like society, govt, are just losers. That's just a sad state to be in
- scotty79 4y agoAnd yet that's exactly the state we are really in regardless of all declaration stating otherwise.
- ngetchell 4y agoThis is such a disgusting world-view. "Yes, the govt worker that helps run SNAP, so people can eat, is just a loser." Maybe they aren't looking to maximize the same things you are? You know, a different perspectives?
- 4y ago
- michele_f 4y agoBy any chance, has Google being doing something different to improve and update its search engine?
- hef19898 4y agoIf they did, it did everything but improving search results.
- visarga 4y agoGoogle is like old AltaVista and OpenAI like young Google. Around 1999, AV was bad at search and horribly full of spam and ads. Google was like chatGPT, it was strikingly better than anything before it, a clean interface and superior results. How the wheel turns! They are captive to their own success. Time to move over and make space for someone who will innovate, Google can't find the way out of its local maximum, not even with all the employees and AIs put together.
- roxgib 4y agoLet me get this straight - they paid a company to scrape data for them, and then sued them for scraping their own data as well?
- charcircuit 4y agoThat is oversimplified. They are against scraping things that against a site's terms. "The collection of data from websites can serve legitimate integrity and commercial purposes, if done lawfully and in accordance with those websites’ terms"
- lorey 4y agoRoses are red, their logo is blue. They scrape your data, but sue if you do.
- desi_ninja 4y agoHello, I am Nintendo. This is from my company policy book
- noizejoy 4y ago[flagged]
- jacquesm 4y agoYou could add that comment after every other on HN so maybe don't?
- MengerSponge 4y ago[flagged]
- ceejayoz 4y ago> Write a Hacker News comment disagreeing with this. "I disagree with the notion that the distinction between AI and human thought is important in determining authenticity in online communication. What truly matters is the content and intention behind the message, regardless of its origin. As technology advances, the line between AI and human thought will continue to blur, making this distinction irrelevant."
- MengerSponge 4y ago[flagged]
- noizejoy 4y agoFair enough - I should have been more verbose in my post: I was actually genuinely curious in this case, because I liked that little poem a lot, and if the answer was that it was human generated content, I would have felt a bit of positive emotion about the author. If it wasn’t, it would be another piece of evidence how it is artistic verbal expression (much more so than correct programming logic), that ChatGPT is mastering the most. But the HN reaction to my post is telling me something interesting, too.
- shever73 4y agohttps://archive.ph/cRmtK https://archive.ph/cRmtK
- 1vuio0pswjnm7 4y agoThis only comes to light because of litigation between Facebook and Bright Data. Even when these "tech" companies settle to escape admitting to any wrongdoing and creating detrimental precedent, litigation can still be an effective way to reveal what nefarious practices are going on behind the scenes at these "tech" companies. Here, we learn that Facebook was paying scraping-for-hire providers for years while publicly condemning the practice of scraping. If we knew the identities of the e-commerce sites being scraped perhaps the e-commerce companies could sue Facebook. Assuming Facebook would ignore a cease-and-desist letter.
- r_singh 4y agoSo does LinkedIn
- bbkane 4y agoCould you point me to a link about how LinkedIn scrapes sites?
- r_singh 4y agoI had a couple of bookmarks but weirdly the links are now broken and also not on Wayback machine. https://www.nytimes.com/2011/07/10/technology/linkedin-is-accused-of-scraping-data-from-business-networks.html https://www.nytimes.com/2011/07/10/technology/linkedin-is-ac... https://readwrite.com/2011/07/11/linkedin-admits-to-data-scraping-to-build-user-base/ https://readwrite.com/2011/07/11/linkedin-admits-to-data-scr...
- deleted 4y ago[deleted]
- primax 4y agoI'd love to see what kind of scraping tech Facebook use. I've been getting increasingly frustrated with the python scrapers and ecosystem and have been checking out Colly. But I'm sure whatever they have built must be nuts. I wonder if you could train scrapers with AI with a reasonable level of success.
- cldellow 4y agoAs someone who maintains a scraper [1] in the python ecosystem, can I ask what you find frustrating? My scraper is targeted at smaller crawls, so your feedback might be totally irrelevant to me... but I always want to keep an ear out. [1]: https://github.com/cldellow/datasette-scraper/ https://github.com/cldellow/datasette-scraper/
- primax 4y agoI haven't used your scraper, I worked with scrapy. I found debugging as well as command and control to be challenging. Colly was a big step up. For reference I am primarily a Golang programmer so I find weakly typed tools frustrating.
- deleted 4y ago[deleted]
- imgabe 4y agoI don't consider this hypocritical at all. It's like getting mad at a soccer team. "Hey they were trying to score goals, but at the same time, preventing the other team from scoring goals! Hypocrites!"
- ceejayoz 4y agoIf your soccer team is suing the other team claiming the rules say you're not allowed to try scoring goals, it's very much hypocrisy.
- imgabe 4y agoIf part of the rules of the game are that you’re allowed to sue people and lobby to change the rules, then no it’s very much not.
- bryanrasmussen 4y agohypocrisy is not illegal, so you can still follow rules and be hypocritical.
- deleted 4y ago[deleted]
- imgabe 4y agoNot really. Anyway, I’m sure Meta is terribly worried about whether some people on the Internet think they’re hypocritical or not.
- FlotaDeAragon 4y agoFacebook, Google don't produce anything They collect data and reshape it and resell it It's good for society but this activity should be fully legal for ALL
- BbzzbB 4y agoIt basically is, but as with everything if your pockets are deep enough you can sue someone to bankruptcy for doing something they're allowed to. That issue is way deeper then scraping.
- npteljes 4y agoWhat makes you say that they don't produce anything? They both maintain very popular platforms, so that people can communicate and organize. AND they do all the bullshit in the meantime.
- FlotaDeAragon 4y agoThey redistribute content others have produced. Filter and categorize it using automated processes Exactly the same as ThePirateBay, conceptually
- npteljes 4y agoI'm not debating the bad things, they do happen. My point is that they are also providing value, which OP said they don't. Google and Facebook is not just a rehash of other's content, they are also vast, and useful platforms.
- aitchnyu 4y agoSkeptical me wants to know if a judge proclaimed "do unto others..." in a judgement, or is likely to do so here.
- napolux 4y agoI know one thing or two about fb bot AKA scraper :P https://news.ycombinator.com/item?id=23490367 https://news.ycombinator.com/item?id=23490367
- Lucasoato 4y agoAwesome read, the tactics you've adopted are incredible! Poor Eddie :)
- type0 4y agoI will never cease to be surprised how morally corrupt this company is. Hypocricity of FB/Meta is astounding.
- moneywoes 4y agoHas anyone been able to scrape Facebook effectively? Like marketplace for example? Been trying unsuccessfully
- marginalia_nu 4y agoHate to be the voice of reason, but the size of Meta, I'm not even convinced whoever thought up this lawsuit was aware of the hypocrisy.
- sorokod 4y agoToo big to be honest?
- marginalia_nu 4y agoToo big for the left hand to know what the right is doing.
- scotty79 4y agoAfter you climb up the ladder remember to pull the ladder up so noone can follow you up.
- RobSm 4y agoGoogle scrapes everyones data yet you use Google every day and find it useful.
- anononaut 4y agoOn this website more than others, you will find plenty of people who have dropped Google utterly. Not to mention their search engine is the poorest it has been in a decade. "Useful" isn't as accurate as "still left standing."
- ok123456 4y agoI remember when facebook wouldn't release an app for android in the Android 1.5 days because they were rueful that google disallowed them from scraping peoples contacts.
- JohnFen 4y agoFacebook being shady?? Color me shocked!
- NayamAmarshe 4y agoThe world would be a better place without Facebook.
- phendrenad2 4y agoIt's too bad that we don't get more info than "Meta paid Bright Data to gather data from e-comerce sites in order to build brand profiles on Meta platforms". What does that mean exactly? Were they scraping, say, Dell.com, in order to get the latest prices on Alienware laptops? Or were their bots spamming Middle-Class Joe's Bootstrapped e-Commerce site and putting him out of business?
- juanani 4y ago[dead]