12 ms·
Dear AI Companies, instead of scraping OpenStreetMap, how about a $10k donation?
- mouse_ 2y agoAI = IP thieves I don't think they're going to donate...
- RicoElectrico 2y agoThe irony is that as indicated in the comments it's far more easy to just download the data dump for the whole planet. It's 70-ish GB right now.
- klyrs 2y agoIf their scraper is sufficiently diligent, they will also download the data dump. The ultimate hope is that when the AI wakes up, it will realize that its training data has both a fragmented dataset and also a giant tarball, and delete the fragments. This sounds like one of those situations where people prefer amortized cost analysis to avoid looking like fools.
- foverzar 2y agoWell it's nice to see the whole concept of IP finally collapsing as it should.
- asddubs 2y agothat's a little optimistic
- foverzar 2y agoIs it really? It's all boils down to powerplay, really. And lately lots of powerful entities, from nations to corporations, seem to be poised on disrupting this institution.
- loceng 2y agoI noticed too that now that the old dinosaur incumbents in various industrial complexes can no longer compete, they want to get rid of non-compete clauses - so they can instead poach talent or at least those who had access to the latest technologies and process of actually innovative companies.
- pessimizer 2y agoExcept it's not collapsing. The only legal changes have been to allow billionaires to do whatever they want whenever they want, and have been made by judges and not legislatively. You're still going to get sued to oblivion. edit: if we let them, they're just going to merge with the media companies and cross-license to each other.
- foverzar 2y agoThere is a whole wide world beyond just obsessing over "billionaires" in a tiny us-centric corner of the world. One could point out China, who totally has so much respect for someone's notion of legality. Or France that had never cared for software patents. Or one could point out good old pirates, who were always relatively successful at giving a middle finger to the notion of "intellectual property". "Billionaires" are simply another straw, peculiar only in the sense that they were the pillar upholding this institution. And speaking of "not legislatively, but judges": 1. Not every country has a common law legal system 2. Just take a look at Japan's AI legislations.
- tonetegeatinst 2y agoI mean is it though? Even before gpt4 llm's existed....and not just from openAI. I get not liking crawling and I hate openAI for how they ruined the term open source, but this is not new. Iv had stuff scraped before and iv done web scrapping as well. Hell even excel will help you scrape web data. While some of the increase of training data has helped models like gpt4, its not just a factor of more data.
- 0xEF 2y agoFrom my perspective as a small blogger, the general sentiment is yes, web scraping was a thing before the AI boom, but now it is an even bigger and far more careless thing. Just me two cents, though, mostly formed from chatting with other small bloggers about it. I suppose it could be argued that the size of the vehicle does not matter to the roadkill; they still get run over. But it honestly feels like the various AI platforms suddenly made web scraping much, much more fashionable.
- lnxg33k1 2y agoThe people working for these companies are just clueless, arrogant, ignorant, unaware of others, just trying to hit some productivity target to get promoted, of course they're not going to bother checking whether there are other ways to do something avoiding annoying open source projects
- rqtwteye 2y agoIn my company it’s easier to buy commercial software for $10000 than it is to donate $100 for open source voluntarily. I think they need to open up a store where you can buy donations disguised as licenses so the bean counters don’t even realize this could be free.
- resource_waste 2y agoThe difference in quality is apparent though. $10,000 and you have an account manager that will actually follow up on issues. I recently paid $5k for software and its incredible the difference. Its like I have a part time contractor and software.
- persnickety 2y agoIt's not clear from your comment, did you pay for commercial software or paid for an open source contributor's time? And, regardless of the answer, what was your experience with the other option, for comparison?
- rqtwteye 2y agoThat’s not my experience. We have systems that cost way into the six digits. You get an account manager that follows up and schedules a lot of meetings. But this doesn’t mean that difficult problems actually get resolved. Just a lot of talk. Even MS super premium support so far has never worked for me. They want an enormous amount of data and then go quiet.
- resource_waste 2y ago
- snehk 2y agoYou can literally set up your own OpenStreetMap instance in ten minutes. It's a simple 'docker run'-command. Sure, indexing will take a bit but even that can't take that long given their resources. That's just ridiculously greedy.
- dawnerd 2y agoLink then? Because last time I tried it was a bit more complex than that.
- claytonjy 2y agoI used OSMRouter maybe 7 or 8 years ago to process a couple billion routes and it was about as simple as GP described. Just need Docker and an index file. the US one was massive so I kept needing a bigger VM just to load it, but once I did I was able to make HTTP calls over the network to get what I needed. Took a few days to get a working setup but only a few hours to rip through billions of requests, and I was making them synchronously with R; could have been much faster if I was smarter then.
- shironandon 2y agohttps://github.com/openstreetmap/openstreetmap-website/blob/master/INSTALL.md https://github.com/openstreetmap/openstreetmap-website/blob/...
- dawnerd 2y agoExactly. This isn't just a 10 minute project. And thats just for the website, not the tiles, etc.
- orblivion 2y agoA while ago I very briefly tried Headway out of curiosity. This is the easiest Docker based option for the "full stack". It didn't work out of the box. Things went wrong. Which is no surprise, there's a ton of moving parts. And maybe it's not a big deal to work around but I highly doubt that it's 10 minutes of work to get everything working reliably.
- jackienotchan 2y agoAffected companies are becoming increasingly frustrated with the army of AI crawlers out there as they won't stick to any scraping best practices (respect robot.txt, use public APIs, no peak load). It's not necessarily about copyright, but the heavy scraping traffic also leads to increased infra costs. What's the endgame here? AI can already solve captchas, so the arms race for bot protection is pretty much lost.
- MisterBastahrd 2y agoHow long before companies start putting AI restrictions on new account creation simply because of the sheer amount of noise and storage issues associated with bot spam?
- candiddevmike 2y agoInvite only authenticated islands based on trust. Which seems like the end result of the rampant centralization of the internet.
- zeroCalories 2y agoThe open web is on a crash course. I don't necessarily believe in copyright claims, but I think it makes sense to aggressively prosecute scrapers for DDOSing.
- danielmarkbruce 2y agoBars and coffee shops?
- infecto 2y agoHonest question, what are "AI Companies" scraping from OSM?
- edent 2y agoBecause - and I cannot stress this enough - they are both ignorant and greedy. Whenever I've traced back an AI bot scraping my sites, I've tried to enter into a dialogue with them. I've offered API access and data dumps. But most of them are barely above the level of "script kiddies". They've read a tutorial on scraping so that's the only thing they know. They also genuinely believe that any public information is theirs for the taking. That's all they want to do; consume. They have no interest in giving back.
- indymike 2y ago> ignorant and greedy. This is going to be the title of my book on AI that I totally need to write.
- noah_buddy 2y agoAvarice and ignorance ;)
- carimura 2y agoi think you answered the "why", not the "what". :)
- BizarroLand 2y agoThe what is "everything they can get". They are the modern equivalent of torrent users who don't seed.
- briandear 2y agoTorrent users who do seed (assuming it’s copyrighted material) are no better. They’re just stealing someone else’s content and facilitating its theft. If a company scrapes data, and then publishes the data for others to scrape.. they are still part of the problem — the altruism of letting other piggyback from their scraping doesn’t negate that they essentially are stealing data. Stealing from grocery store and giving away some of what you steal doesn’t absolve the original theft.
- exabrial 2y agoOnce again: Silicon Valley does not understand the concept of willful consent.
- JohnFen 2y agoOh, it understands. It just rejects anything that might interfere with income generation.
- pessimizer 2y agoAnd like most bad things that companies do, it happens inevitably. The person who doesn't reject anything that might interfere with income generation will be fired and replaced with someone who will. Meanwhile, the owners will maintain and carefully curate their ignorance about any of those subjects.
- tonetegeatinst 2y agoCounter argument, its just an excuse to get rid of scraping. Google and every search engine scrapes websites, internet archive scrapes websites to archive stuff, and I scrape data when using excel to import data. Also their are people who want to archive everything. Iv had my own stuff be scrapped. My biggest issue was bandwidth but I wasn't a big site so it wasn't a big issue.
- ThatsANewOne 2y ago[flagged]
- deleted 2y ago[deleted]
- bofadeez 2y agoStill too expensive. Why not just sell it at the prevailing rate per GB for residential IP bandwidth? A thousand max.
- Aachen 2y agoThe joke is that you can already download it for free, no donation or bandwidth reimbursement needed https://wiki.openstreetmap.org/wiki/Planet.osm https://wiki.openstreetmap.org/wiki/Planet.osm I guess since it's posted to osm.town Mastodon, this is assumed to be known. Was surprised to see it without context here on HN; I can understand the confusion. Apparently most people here are already aware that one can download the full OpenStreetMap data without scraping
- deely3 2y agoIn your opinion, profits from selling data should go to OpenStreetMap contributors or OpenStreetMap Foundation?
- nailer 2y agoSomeone recently pointed out the Aaron Schwartz was threatened with going to prison for scraping, meanwhile there's hundred of billion of dollars right now invested in AI LLMs build from... scraping.
- ToucanLoucan 2y agoRules for thee and not for me. Same as it ever was.
- startupsfail 2y agoAnd Markdown is the primary format.
- bdjsiqoocwk 2y agoWhy go to AI LLMs? Scraping and indexing is all Google search does.
- stavros 2y agoThat's because the megacorps can scrape you, but you can't scrape the megacorps.
- jahewson 2y agoJSTOR is a non-profit run by academics. Indeed, they have plenty of money, but they’re no megacorp.
- SJC_Hacker 2y agoGenerally it should be "More powerful entities can scrape you, but you can't scrape them back" Google scraping JSTOR (hey, don't they do that already with Google Scholar?" is much less of a problem then JSTOR attempting to scrape Google.
- TiredOfLife 2y agoWasn't he scrapping paywalled stuff?
- Satam 2y agoI needed osm data at one point. Never managed to figure out how to do it the proper way. To get data you need, you need to download massive 100Gb files, in obscure formats, and use obscure libraries. Info is scattered, there are HTTP APIs but they’re limited or rate-limited and it’s not clear if you’re supposed to use them. I know I’m ignorant and I’m happy the project exists, but the usability in the era where devs expect streamlined APIs is not great. I ended up using some free project that had pre-transformed osm data for what i needed.
- Aachen 2y agoWhat non-obscure formats or libraries would you suggest for a planet's worth of geographic data? I've also downloaded planet.osm before and parsed it on my desktop with iirc osmosis. Never used that format or tool anywhere else but it's not like OSM has so many competitors offering you large amounts of geospatial data in a freely usable way. What do you considered established mechanisms for this?
- Doctor_Fegg 2y agoThat's kind of by design. Providing streamlined APIs requires a funding model to both host those APIs and pay an army of devops to maintain them. The OSM Foundation is intentionally small and doesn't do that. Rather, it encourages a decentralised ecosystem where anyone can take the data and build services on it - some commercial, some hobbyist, some paid-for, some free. It works really well, and IMO better than the big-budget maximalist approach of the Wikimedia Foundation.
- spywaregorilla 2y agohttps://wiki.openstreetmap.org/wiki/OSM_JSON https://wiki.openstreetmap.org/wiki/OSM_JSON Looks pretty sensible to me?
- ks2048 2y agoOn https://www.openstreetmap.org/ https://www.openstreetmap.org/, click "Export" (upper-left). It lets you choose a small rectangle (click "Manually select a different area"). It gives you a .osm right from the browser. For literally single point, on the map icons on the right, one is arrow with question mark ("Query features"). With this you can click on single features and get their data.
- butz 2y agoPut planet.osm on torrent. Allow "scraping" only through torrent. Now scrapers are sharing the network load between themselves. Not to mention improved network speed, as probably they all sit on the same AWS instance.
- Firefishy 2y agoOur data is already publish via torrent, see https://planet.openstreetmap.org https://planet.openstreetmap.org Our data, including minutely updates is available via public S3 buckets (EU & US) supported by the AWS Open Data Sponsorship Program.
- butz 2y agoAs expected. Now for the hard part: how to redirect scrapers to use torrents, and keep them uploading data? Should we write some "data scraping tutorials" specifically targeting OSM and describe using torrents as the best method?
- ks2048 2y agoKind of sad that CommonCrawl, or something like it, has not removed the need for tons of different companies hitting all the servers in the world. I guess part of it is wanting more control (more frequent visits, etc) and part is simply having lots of VC money and doing something they can do to try and impress more investors - "We have proprietary 5 PB dataset!" (literally adds nothing to commoncrawl).
- Workaccount2 2y agoI can hear the AI groaning about regular humans suddenly caring a lot about IP protection and discussing ways to add DRM to protect it. I really hope the irony isn't lost on everyone.
- acd 2y agoContent attribution where content providers get paid.
- deleted 2y ago[deleted]
- nashashmi 2y agoHow about a honey pot for AI companies? Endless loop of stupidly generated content. Imagine twitter posts with artificial tweets at the end.
- AshamedCaptain 2y agoNot dissimilar to how instead of just cloning my pre-compressed repos in a simple few seconds operation, "AI" scrappers prefer to request every single revision of every single .c file through the web interface with all the useless (for them) bells an whistles. Web interface which I have set up as cgi and therefore it will take them longer to finish scrapping than the age of the universe. But in the meanwhile they waste me power and resources.
- Firefishy 2y agoOP here. The toot was my sarcastic response after having to rate-limit and block another set of abusive scrapers aggressively hitting our website and mapping API. robots.txt be damned. OpenStreetMap data is free to download. We published minutely on https://planet.openstreetmap.org/ https://planet.openstreetmap.org/ and the data available via AWS S3 + torrent. If you just starting out, best to start with a smaller regional extract: https://wiki.openstreetmap.org/wiki/Planet.osm https://wiki.openstreetmap.org/wiki/Planet.osm
- gosub100 2y agoHow come, in this era of no privacy, pixel trackers, data brokers, etc can they not easily stop scraping? Somehow bots have a way to be anonymous online yet consumers have to fight an uphill battle?
- SergeAx 2y agoOnce, during a tech interview, the interviewer asked me to design a system for daily scraping English Wikipedia. I started with "Let's download the gzipped archive...". It turned out that the interviewer didn't know about that possibility and was waiting for the description of a complex system to download it page by page with multi-threading, canonical URLs, checking of visited pages, retries and so on. To their credit, they gave me "A" for this assignment. In the end, I got the job.
- stereo 2y agoOpenStreetMap Foundation chairperson here. OpenStreetMap's data is available for free in bulk from https://planet.openstreetmap.org https://planet.openstreetmap.org. We encourage using these instead of scraping our site. Scraping puts a high load on our donated resources. We block scraping IPs, but even that takes us work and time. Respecting our time and resources helps us keep the service free and accessible for everyone.
- Self-Perfection 2y agoAnd how exactly do you block scraping IPs? I suspect some of scrapers are just confused and are not aware of better ways to get OSM data. Responding with a 403 error code will only lead to them changing their IP addresses. A more effective approach might be to provide a response containing instructions on where to download data in bulk or a link to a guide that explains how to process OSM dumps.
- stereo 2y agoYou make a good point. I already have a PR to add text to our robots.txt but the 403 page is an excellent idea.
- miohtama 2y agoIf it is Russian and Chinese AI companies, they are unlikely to care. If it is Western companies, they might care, because they have more respect to the rules. Faked user agent is usually unfortunately the indication of the former.
- sofixa 2y ago> they might care, because they have more respect to the rules Do they? Didn't OpenAI scrape everything regardless of licence forbidding reuse without attribution or for commercial interest?
- jaakl 2y agoJust an idea: https://news.ycombinator.com/item?id=41111145 https://news.ycombinator.com/item?id=41111145