5 ms·

How To Get Experience Working With Large Datasets

36 points by m3mb3r 16y ago

ashtophoenix 16y ago
What a silly article - When it said how to get experience working with large datasets I was expecting it would explain more about storage/scalability/design/caching issues etc. There are myriad ways to get (or generate) data to play with...
andrewjshults 16y ago
Along the same lines, NYC's Big Apps 2.0 competition is going on right now (http://nycbigapps.com/ http://nycbigapps.com/). Not affiliated, but I went to NYTM last year where they demoed the winners and there are some interesting (and impressively large) datasets to play with. One of my favorites was the mobile app, CabSense, that crunched the TLC data to determine the best corners to catch a cab on depending on the time of day
wrath 16y ago
One nice and free dataset which you can play with is the BestBuy open data. You can download the full catalog of products from BestBuy in JSON and XML format. http://developer.bestbuy.com http://developer.bestbuy.com Simply register for a key and you'll have access to the data.
drblast 16y ago
US Census data is multiple gigabytes, and well documented. If you want to run a database through its paces beyond the point where all the data fits in memory, that's a good place to start.
- lsb 16y ago
  Census data's definitely interesting. If you have the cash, you can rent a box with 68GB of memory for $0.70/hr on EC2, as a spot instance.
earl 16y ago
What's with the recent fetishization of Big Data? I'm moving to Dziuba's camp -- its a developer dick size contest.
- elai 16y ago
  An interview I had asked my experience with large data sets and high traffic web sites and such and what I thought about them. I got an impression they were a bit shell shocked by the workload of a high traffic website. Many software paying positions and startup problems deal with large data sets vs. the small easy ones you usually deal with in client applications, many games, iPhone apps, etc.
- bobds 16y ago
  It's because we need the techniques and tools to handle the ever-increasing amount of information in our lives.
fmw 16y ago
They might be relatively small, but http://www.grouplens.org/node/12 http://www.grouplens.org/node/12 has some interesting datasets that can be used to experiment with recommendation systems, e.g. book and movie reviews.

36 points by m3mb3r 16y ago

What a silly article - When it said how to get experience working with large datasets I was expecting it would explain more about storage/scalability/design/caching issues etc. There are myriad ways to get (or generate) data to play with...

andrewjshults 16y ago

Along the same lines, NYC's Big Apps 2.0 competition is going on right now (http://nycbigapps.com/ http://nycbigapps.com/). Not affiliated, but I went to NYTM last year where they demoed the winners and there are some interesting (and impressively large) datasets to play with. One of my favorites was the mobile app, CabSense, that crunched the TLC data to determine the best corners to catch a cab on depending on the time of day

wrath 16y ago

One nice and free dataset which you can play with is the BestBuy open data. You can download the full catalog of products from BestBuy in JSON and XML format. http://developer.bestbuy.com http://developer.bestbuy.com Simply register for a key and you'll have access to the data.

drblast 16y ago

US Census data is multiple gigabytes, and well documented. If you want to run a database through its paces beyond the point where all the data fits in memory, that's a good place to start.

lsb 16y ago
Census data's definitely interesting. If you have the cash, you can rent a box with 68GB of memory for $0.70/hr on EC2, as a spot instance.

earl 16y ago

What's with the recent fetishization of Big Data? I'm moving to Dziuba's camp -- its a developer dick size contest.

elai 16y ago
An interview I had asked my experience with large data sets and high traffic web sites and such and what I thought about them. I got an impression they were a bit shell shocked by the workload of a high traffic website. Many software paying positions and startup problems deal with large data sets vs. the small easy ones you usually deal with in client applications, many games, iPhone apps, etc.
bobds 16y ago
It's because we need the techniques and tools to handle the ever-increasing amount of information in our lives.

fmw 16y ago

They might be relatively small, but http://www.grouplens.org/node/12 http://www.grouplens.org/node/12 has some interesting datasets that can be used to experiment with recommendation systems, e.g. book and movie reviews.