6 ms·
This is fantastic. I would like to see wider legislation allowing scraping of IMDB, Genius, Reddit, Facebook, and Google made legal. These services receive free
by echelon 7y ago
This is fantastic. I would like to see wider legislation allowing scraping of IMDB, Genius, Reddit, Facebook, and Google made legal. These services receive free input from users. The data should remain free.
Edit (sort of off topic): There's still value in the building and providing services at scale, but this lowers the barrier to cross the moat for small players. The first step is data liberation. Then we can work to bring down the other cost barriers. It's a lot easier to build services that scale in 2019 than it was in 2005.
The semantic web was misguided in 200X, but we might want to take another swing at it in the future.
- OrgNet 7y agofor IMDb, they have a lot of data that is easily accessible, not sure what is missing though: https://datasets.imdbws.com/.. https://datasets.imdbws.com/...
- tdhoot 7y agoOnly for personal and non-commercial use, which is probably not what startups need.
- JustSomeNobody 7y agoWhy would they have to make it available to startups in an easily accessible manner?
- nitrogen 7y agoIMDB was originally crowdsourced, wasn't it?
- bewuethr 7y agoIt started as lists and shell scripts to query them in the newsgroup rec.arts.movies.
- JustSomeNobody 7y agoSo, have the data available. But I see no reason why they should have to go out of their way to change their site to make it easy for someone to get it.
- OrgNet 7y agoprocess it on your personal computer and use the output in your startup
- psv1 7y agoAnother side of this is that the entity doing the scraping is more often than not another company. Which means that if your proposal is implemented, a user can voluntarily give their personal data to Google/Reddit/Facebook etc but that company then has to make the user's personal data available to another company.
- bobthepanda 7y agoIt only applies to data displayed publicly, though. If Facebook and such started requiring logins to see personal data would that be such a bad thing?
- jjeaff 7y agoI'm not certain, but it kind of sounds like even things behind a login are still scrapable. Assuming the general public can get a login easily anyway. Basically, just requiring an account is not enough to forbid scraping.
- ptero 7y agoIt's not quite like that. The first company cannot prevent scraping by individuals or another company of information that it already shows to everyone. Which, to me, is a good thing. My 2c.
- gkoberger 7y agoEh. I want my picture and name uploaded to LinkedIn, since it's a professional network and people use it to find me for good reasons. It may seem dumb, however not having a LinkedIn with a good picture can genuinely hurt your career. I do NOT want my picture run through facial recognition software, or my name/email sold to marketers who will add it to a drip campaign.
- rch 7y agoExactly. I don't like or "enjoy" LinkedIn but I do find it useful professionally. Now it sounds like this ruling implies that by creating a profile on one platform, I have to accept that every company that comes along can include me in their corpus. Maybe I should be able to set a pass-through GDPR flag on my profile such that third parties (subject to that regulation) will have to exclude me from their datasets.
- jakeogh 7y agoIt's already legal. Adding law adds restrictions.
- pharrington 7y agoYou're not wrong. In the general case though, adding law can mandate that a certain already occurring activity must be done.
- polygot 7y agoIf you add .json to the end of a Reddit URL, it will return JSON data. For example: https://www.reddit.com/r/ubuntu.json https://www.reddit.com/r/ubuntu.json . It also works with comment threads and posts.
- Raidion 7y agoPRAW is also a great python reddit "scraper" that allows you to pull data via their API very easily.
- ludamad 7y agoNow that is an ergonomic API.
- avip 7y agoWonderful feature also used by Trello https://trello.com/b/rq2mYJNn/public-trello-boards.json https://trello.com/b/rq2mYJNn/public-trello-boards.json
- polygot 7y agoAlso, it outputs XML and RSS too: https://www.reddit.com/r/ubuntu.rss https://www.reddit.com/r/ubuntu.rss and https://www.reddit.com/r/ubuntu.xml https://www.reddit.com/r/ubuntu.xml
- ahbyb 7y agoxml and rss seem to be the same exact output
- diminoten 7y agoYeah no need to scrape Reddit, their content is accessible via their API.
- sneak 7y agoI have adopted this in other projects and added the functionality there as well; it is a brilliant idea.
- sireat 7y agoReddit has a decent API The golden rule is to use the API before you start raw scraping.
- lonelappde 7y agoWhat gives you a rightful claim to information that I gave to someone else, if neither I nor they consent?
- austhrow743 7y agoYou did consent. This is about information you gave to LinkedIn and told them to give to the general public.
- tekknik 7y agoWhen you talk about data here these are people. This HiQ software is actually a bit scary. What if it gives a false signal which ends in an employees termination? Data on people should not be freely attainable, the person should give explicit access. If I don’t want HiQ processing my information (I don’t) then they shouldn’t be able to. Especially now with some employers requiring a LinkedIn profile.