4 ms·

I don't know - maybe I'm just irrationally optimistic but I don't believe anyone has "locked up all of the data". You can download the whole Wikipedia data dum

I don't know - maybe I'm just irrationally optimistic but I don't believe anyone has "locked up all of the data". You can download the whole Wikipedia data dump at under 100GB uncompressed (text only - with media is around a TB). The entire common crawl with 3bn pages is only around 250 TB. While the Wikipedia dataset is too large to fit in ram for most people and the common crawl is too big to fit on a single disk, you can process these in your own local cluster quite easily and relatively cheaply. Honestly I think the real breakthroughs to be made will be algorithmic and I don't believe those are out of reach for "civilians" outside of the tech giants.

pas 9y ago
While I think OP is a bit too pessimistic, in a way s/he is right. Facebook can design a simple experiment, implement the changes in their whole platform, roll out, gather data, and iterate on it. You are stuck with 25TBs of wikipedia for the rest of forever. And "academic datasets". Because the important data is probably not inferable from web links or otherwise semi-passive Internet structures. That said, I think the missing component is still a good mind theory.
- alexcnwy 9y ago
  Humans don't run split tests on alternate realities but they can learn from wikipedia alone - I think we just don't know quite how yet. We need a lot of data currently because algorithms don't generalize like people do. I'm not convinced you need to be a tech giant to be able to make that breakthrough - I think it's a problem of approach, not a lack of data.
  - pas 9y ago
    People ingest a lot of data. Especially when they are young. They touch everything, look at everything in funny ways, chew on things, and so on. Try to interact with everything to gain data. It's just not textual.
AndrewKemendo 9y ago
Great, so who are the people going to make those breakthroughs? Last I checked they all work for...AFGAM (or are teaching)

pas 9y ago

While I think OP is a bit too pessimistic, in a way s/he is right. Facebook can design a simple experiment, implement the changes in their whole platform, roll out, gather data, and iterate on it. You are stuck with 25TBs of wikipedia for the rest of forever. And "academic datasets". Because the important data is probably not inferable from web links or otherwise semi-passive Internet structures. That said, I think the missing component is still a good mind theory.

alexcnwy 9y ago
Humans don't run split tests on alternate realities but they can learn from wikipedia alone - I think we just don't know quite how yet. We need a lot of data currently because algorithms don't generalize like people do. I'm not convinced you need to be a tech giant to be able to make that breakthrough - I think it's a problem of approach, not a lack of data.
- pas 9y ago
  People ingest a lot of data. Especially when they are young. They touch everything, look at everything in funny ways, chew on things, and so on. Try to interact with everything to gain data. It's just not textual.

AndrewKemendo 9y ago

Great, so who are the people going to make those breakthroughs? Last I checked they all work for...AFGAM (or are teaching)