10 ms·

Yahoo Releases the Largest-Ever Machine Learning Dataset for Researchers

397 points by tchalla 11y ago

glxc 11y ago
13.5 TB - that is pretty huge! Great to get some truly "Big Data" sets out there. I consider "Big Data" to be data that can't be conventionally processed on a commodity machine, else it's just analytics Yahoo must be applauded for supplying various data sets and helping progress machine learning research
- collyw 11y ago
  I saw a course advertised in my email yesterday. Big data with MySQL. The description talked about queries and aggregate functions. That isn't big data - that's just "using a database" before the term "big data" appeared in the mainstream.
  - sbarre 11y ago
    Ugh.. I had a boss years back who insisted on using "big data" to refer to our analytics and reporting work (which was nowhere near big data in terms of data size - we had maybe a million rows in our database across all our tables), and I fruitlessly tried for months to explain to him that anyone who really knows what "big data" means would immediately see through his bullshit..
    - collyw 11y ago
      I really wish we could get rid of the hipster / buzzword / fashionista aspect of our industry. Way too much churn as a result. I would far rather spend time honing SQL skills to perfection rather than having to learn another NoSQL database. Unfortunately job descriptions prefer the latter.
  - collyw 11y ago
    https://www.coursera.org/learn/analytics-mysql?utm_medium=email&utm_source=marketing&utm_campaign=1RcwcLmkEeW6i-NodUB9Qw https://www.coursera.org/learn/analytics-mysql?utm_medium=em... Ok, they are comparing MySQL to Excel, so in relative terms the data could be bigger... :(
frik 11y ago
Sad the future of "Yahoo!" (the tech company, not the Alibaba stock) is uncertain. They were always very open with their research. Thinking back to 2008/09 they had the biggest Hadoop clusters, etc. even the first edition of O'Reilys Hadoop books says "Yahoo press".
- magicmu 11y ago
  As a brand, I'm honestly impressed that they've lasted through so many sea changes.
- alceufc 11y ago
  Flickr was -- and maybe still is -- very useful for the computer vision research community.
  - tedd4u 11y ago
    This is old news but we put together a 100M item dataset from Flickr as well, all Creative Commons licensed, including lots of metadata from users as well as some pre-computed features. It's called the YFCC100M. http://yahoolabs.tumblr.com/post/89783581601/one-hundred-million-creative-commons-flickr-images http://yahoolabs.tumblr.com/post/89783581601/one-hundred-mil...
    - mapmap 11y ago
      Is a Creative Commons license for research only or can it be used for commercial purposes if you attribute the original author?
  - TeMPOraL 11y ago
    How do they use it? Does it have a good programmatic access? Because UX-side, I'm surprised they still exist. I don't know of a single photo-related web site that has worse UI and is more annoying to use than Flickr.
    - PaulHoule 11y ago
BetaCygni 11y ago
I'm torn. I love open data, but I fully expect that someone will (partially) deanonymize this.
- rectang 11y ago
  I share your concern. Once data like this is deanonymized, it's out there forever -- there's no going back to fix it like you would a software bug. So you need perfect understanding and provable security at release time to guaranteed safety into the indefinite future. That's not an easy constraint to satisfy.
  - joatmon-snoo 11y ago
    The thing that terrifies me specifically is that there's been work done - I believe it was a branch of the US military studying network traffic patterns - showing that you can reconstruct profiles based on behavior patterns and link them back to the original user with high success rates.
- fweespeech 11y ago
  Yep, and this is why stuff like this should have a formal opt-in process.
- japaw 11y ago
  Probobly. Hopefully so have they learned from the fallout from the AOL search log case ( https://en.wikipedia.org/wiki/AOL_search_data_leak https://en.wikipedia.org/wiki/AOL_search_data_leak ). That case was certainly a big mess.
  - jacquesm 11y ago
    I'm not sure why you're downvoted, the AOL search log case was a huge mistake on the part of AOL and I'm quite surprised that Yahoo! would take a risk like this. The real risk is never in just this data but in combining it with other public datasets. I haven't looked at what is in this particular dump in detail but if there is data that had to be anonymized (as they claim) then you can bet that there will be people already busy trying to reverse that.
wahsd 11y ago
Can someone please explain to me why this dataset needs to be one big file? They couldn't have broken it down? I need to download the full 1.5TB? Also, they couldn't have simply made the data available on one of the "big-data" services? Seems to redundant and inefficient.
rodionos 11y ago
Note that they're sharing this dataset only with *.edu, which is unfortunate for the rest of us. I wish they would allow access to a fraction of the dataset, e.g. 5% of records, for the rest of the community.
- boomzilla 11y ago
  Would it be suffice with an .edu email, or does one need a formal document from university officials? I tried to click through from the sandbox link, but a Yahoo account is required.
  - nolok 11y ago
    Do you mean to technically get access, or to be legally allowed to use it ? Because for the latter, merely having a .edu isn't enough.
    - cinquemb 11y ago
      It's probably more complicated than that, because I assume there are places in the world where you can legally ignore someone trying to enforce such claim where they are outside of such jurisdiction.
  - Namrog84 11y ago
    Most places only care that you were a student with active edu. I graduated years ago and I still use the edu without problem. Most things say you need to be a student from university. Which I am/was. Very few places say actively enrolled student. It's a legal gray area in my opinion.
    - dexterdog 11y ago
      Many schools will give alumni edu addresses as well
  - rodionos 11y ago
    I have a lifelong edu address but it's not what they mean of course. My reading is that you have to be an enrolled student or researcher.
mooreds 11y ago
It is actually 1.5TB compressed. Direct link to the dataset: http://webscope.sandbox.yahoo.com/catalog.php?datatype=r&did=75 http://webscope.sandbox.yahoo.com/catalog.php?datatype=r&did... "The dataset may be used by researchers to validate recommender systems, collaborative filtering methods, context-aware learning, large-scale learning algorithms, transfer learning, user behavior modeling, content enrichment and unsupervised learning methods." Edit: added quote
astazangasta 11y ago
I am so sick of the implication that all data is equivalent, and there is some generic notion of "big data" that we generic "data scientists" can learn how to "mine" using some generic technique called "deep learning" that will give us all the answers we need like some kind of oracle. I study biology. The shape of the data, the way it is structured, the problems we face in analyzing it, are quite different than the ones faced in user-news interaction data. Techniques that are useful for reshaping and summarizing one dataset are not necessarily applicable to another.
- blazespin 11y ago
  Word2Vec!
jedberg 11y ago
It's too bad they aren't publishing this as an EBS snapshot. That would probably be the most useful way their intended audience could consume it given that most universities get a ton of free Amazon credits for exactly this type of research.
- lqdc13 11y ago
  My university had no Amazon credits (2 years ago). I did have access to several supercomputers though, which would work out much better for this type of data. Yahoo is also somewhat closer to Microsoft than to Amazon.
IshKebab 11y ago
Not the most interesting dataset though.
inglor 11y ago
People who downloaded this - does this contain any form of tagging of the data? For example, do news articles contain visit counts? Article sentiment? Any form of structured information? Otherwise, what benefit does this have over scarping news sites?
- GrantS 11y ago
  The interesting data here aren't the news articles themselves, but the news-browsing history of 20 million people over a 4 month period. To answer your first question, though, according to the official description of the dataset [1], "On the item side, we are releasing the title, summary, and key-phrases of the pertinent news article." [1] http://webscope.sandbox.yahoo.com/catalog.php?datatype=r&did=75 http://webscope.sandbox.yahoo.com/catalog.php?datatype=r&did...
- redrummr 11y ago
  Did you read the article? It seems like they are providing data on the stories people clicked, and at what time, so you can draw temporal and recommendation hypotheses. Some device and location specifics are provided. Scraping can only tell the scraper's story. This data tells millions of people's stories.
boltzmannbrain 11y ago
It's unfortunate Yahoo assumes only those with .edu email addresses make up "the research community".
- rovr138 11y ago
  No they don't. http://webscope.sandbox.yahoo.com http://webscope.sandbox.yahoo.com
  - boltzmannbrain 11y ago
    You're kidding... "TO BE ELIGIBLE TO RECEIVE WEBSCOPE DATA, UNLESS SPECIFIED IN A PARTICULAR DATASET, YOU MUST: - Be a faculty member, research employee or student from an accredited university - Send the data request from an accredited university .edu or domain name (for international universities) email address"
fsaintjacques 11y ago
I really need a yahoo account with verified sms to download this?
wdr1 11y ago
Is it possible to get the readme w/o downloading the entire thing? They state "The readme file for this dataset is located in part 1 of the download. Please refer to the readme file for a detailed overview of the dataset.", but I only see an option to get the full 1.5T.
jonesb6 11y ago
1) Begin registration to a community college. 2) Get .edu email address 3) Profit
satyajeet23 11y ago
Released Publicly? You need an .edu mail address, a yahoo account with verified sms to download this! Very unfortunate.
- zo1 11y ago
  It is unfortunate. But who knows what sort of restriction have to be imposed by the various sources of the data and other various contractual obligations? I'd imagine most of us would feel quite differently if we knew that we were sources for certain parts of the dataset.

397 points by tchalla 11y ago

glxc 11y ago

13.5 TB - that is pretty huge! Great to get some truly "Big Data" sets out there. I consider "Big Data" to be data that can't be conventionally processed on a commodity machine, else it's just analytics Yahoo must be applauded for supplying various data sets and helping progress machine learning research

collyw 11y ago
I saw a course advertised in my email yesterday. Big data with MySQL. The description talked about queries and aggregate functions. That isn't big data - that's just "using a database" before the term "big data" appeared in the mainstream.
- sbarre 11y ago
  Ugh.. I had a boss years back who insisted on using "big data" to refer to our analytics and reporting work (which was nowhere near big data in terms of data size - we had maybe a million rows in our database across all our tables), and I fruitlessly tried for months to explain to him that anyone who really knows what "big data" means would immediately see through his bullshit..
  - collyw 11y ago
    I really wish we could get rid of the hipster / buzzword / fashionista aspect of our industry. Way too much churn as a result. I would far rather spend time honing SQL skills to perfection rather than having to learn another NoSQL database. Unfortunately job descriptions prefer the latter.
- collyw 11y ago
  https://www.coursera.org/learn/analytics-mysql?utm_medium=email&utm_source=marketing&utm_campaign=1RcwcLmkEeW6i-NodUB9Qw https://www.coursera.org/learn/analytics-mysql?utm_medium=em... Ok, they are comparing MySQL to Excel, so in relative terms the data could be bigger... :(

frik 11y ago

Sad the future of "Yahoo!" (the tech company, not the Alibaba stock) is uncertain. They were always very open with their research. Thinking back to 2008/09 they had the biggest Hadoop clusters, etc. even the first edition of O'Reilys Hadoop books says "Yahoo press".

magicmu 11y ago
As a brand, I'm honestly impressed that they've lasted through so many sea changes.
alceufc 11y ago
Flickr was -- and maybe still is -- very useful for the computer vision research community.
- tedd4u 11y ago
  This is old news but we put together a 100M item dataset from Flickr as well, all Creative Commons licensed, including lots of metadata from users as well as some pre-computed features. It's called the YFCC100M. http://yahoolabs.tumblr.com/post/89783581601/one-hundred-million-creative-commons-flickr-images http://yahoolabs.tumblr.com/post/89783581601/one-hundred-mil...
  - mapmap 11y ago
    Is a Creative Commons license for research only or can it be used for commercial purposes if you attribute the original author?
- TeMPOraL 11y ago
  How do they use it? Does it have a good programmatic access? Because UX-side, I'm surprised they still exist. I don't know of a single photo-related web site that has worse UI and is more annoying to use than Flickr.
  - PaulHoule 11y ago

BetaCygni 11y ago

I'm torn. I love open data, but I fully expect that someone will (partially) deanonymize this.

rectang 11y ago
I share your concern. Once data like this is deanonymized, it's out there forever -- there's no going back to fix it like you would a software bug. So you need perfect understanding and provable security at release time to guaranteed safety into the indefinite future. That's not an easy constraint to satisfy.
- joatmon-snoo 11y ago
  The thing that terrifies me specifically is that there's been work done - I believe it was a branch of the US military studying network traffic patterns - showing that you can reconstruct profiles based on behavior patterns and link them back to the original user with high success rates.
fweespeech 11y ago
Yep, and this is why stuff like this should have a formal opt-in process.
japaw 11y ago
Probobly. Hopefully so have they learned from the fallout from the AOL search log case ( https://en.wikipedia.org/wiki/AOL_search_data_leak https://en.wikipedia.org/wiki/AOL_search_data_leak ). That case was certainly a big mess.
- jacquesm 11y ago
  I'm not sure why you're downvoted, the AOL search log case was a huge mistake on the part of AOL and I'm quite surprised that Yahoo! would take a risk like this. The real risk is never in just this data but in combining it with other public datasets. I haven't looked at what is in this particular dump in detail but if there is data that had to be anonymized (as they claim) then you can bet that there will be people already busy trying to reverse that.

wahsd 11y ago

Can someone please explain to me why this dataset needs to be one big file? They couldn't have broken it down? I need to download the full 1.5TB? Also, they couldn't have simply made the data available on one of the "big-data" services? Seems to redundant and inefficient.

rodionos 11y ago

Note that they're sharing this dataset only with *.edu, which is unfortunate for the rest of us. I wish they would allow access to a fraction of the dataset, e.g. 5% of records, for the rest of the community.

boomzilla 11y ago
Would it be suffice with an .edu email, or does one need a formal document from university officials? I tried to click through from the sandbox link, but a Yahoo account is required.
- nolok 11y ago
  Do you mean to technically get access, or to be legally allowed to use it ? Because for the latter, merely having a .edu isn't enough.
  - cinquemb 11y ago
    It's probably more complicated than that, because I assume there are places in the world where you can legally ignore someone trying to enforce such claim where they are outside of such jurisdiction.
- Namrog84 11y ago
  Most places only care that you were a student with active edu. I graduated years ago and I still use the edu without problem. Most things say you need to be a student from university. Which I am/was. Very few places say actively enrolled student. It's a legal gray area in my opinion.
  - dexterdog 11y ago
    Many schools will give alumni edu addresses as well
- rodionos 11y ago
  I have a lifelong edu address but it's not what they mean of course. My reading is that you have to be an enrolled student or researcher.

mooreds 11y ago

It is actually 1.5TB compressed. Direct link to the dataset: http://webscope.sandbox.yahoo.com/catalog.php?datatype=r&did=75 http://webscope.sandbox.yahoo.com/catalog.php?datatype=r&did... "The dataset may be used by researchers to validate recommender systems, collaborative filtering methods, context-aware learning, large-scale learning algorithms, transfer learning, user behavior modeling, content enrichment and unsupervised learning methods." Edit: added quote

astazangasta 11y ago

I am so sick of the implication that all data is equivalent, and there is some generic notion of "big data" that we generic "data scientists" can learn how to "mine" using some generic technique called "deep learning" that will give us all the answers we need like some kind of oracle. I study biology. The shape of the data, the way it is structured, the problems we face in analyzing it, are quite different than the ones faced in user-news interaction data. Techniques that are useful for reshaping and summarizing one dataset are not necessarily applicable to another.

blazespin 11y ago
Word2Vec!

jedberg 11y ago

It's too bad they aren't publishing this as an EBS snapshot. That would probably be the most useful way their intended audience could consume it given that most universities get a ton of free Amazon credits for exactly this type of research.

lqdc13 11y ago
My university had no Amazon credits (2 years ago). I did have access to several supercomputers though, which would work out much better for this type of data. Yahoo is also somewhat closer to Microsoft than to Amazon.

IshKebab 11y ago

Not the most interesting dataset though.

inglor 11y ago

People who downloaded this - does this contain any form of tagging of the data? For example, do news articles contain visit counts? Article sentiment? Any form of structured information? Otherwise, what benefit does this have over scarping news sites?

GrantS 11y ago
The interesting data here aren't the news articles themselves, but the news-browsing history of 20 million people over a 4 month period. To answer your first question, though, according to the official description of the dataset [1], "On the item side, we are releasing the title, summary, and key-phrases of the pertinent news article." [1] http://webscope.sandbox.yahoo.com/catalog.php?datatype=r&did=75 http://webscope.sandbox.yahoo.com/catalog.php?datatype=r&did...
redrummr 11y ago
Did you read the article? It seems like they are providing data on the stories people clicked, and at what time, so you can draw temporal and recommendation hypotheses. Some device and location specifics are provided. Scraping can only tell the scraper's story. This data tells millions of people's stories.

boltzmannbrain 11y ago

It's unfortunate Yahoo assumes only those with .edu email addresses make up "the research community".

rovr138 11y ago
No they don't. http://webscope.sandbox.yahoo.com http://webscope.sandbox.yahoo.com
- boltzmannbrain 11y ago
  You're kidding... "TO BE ELIGIBLE TO RECEIVE WEBSCOPE DATA, UNLESS SPECIFIED IN A PARTICULAR DATASET, YOU MUST: - Be a faculty member, research employee or student from an accredited university - Send the data request from an accredited university .edu or domain name (for international universities) email address"

fsaintjacques 11y ago

I really need a yahoo account with verified sms to download this?

wdr1 11y ago

Is it possible to get the readme w/o downloading the entire thing? They state "The readme file for this dataset is located in part 1 of the download. Please refer to the readme file for a detailed overview of the dataset.", but I only see an option to get the full 1.5T.

jonesb6 11y ago

1) Begin registration to a community college. 2) Get .edu email address 3) Profit

satyajeet23 11y ago

Released Publicly? You need an .edu mail address, a yahoo account with verified sms to download this! Very unfortunate.

zo1 11y ago
It is unfortunate. But who knows what sort of restriction have to be imposed by the various sources of the data and other various contractual obligations? I'd imagine most of us would feel quite differently if we knew that we were sources for certain parts of the dataset.