5 ms·

Wolfram has got to be a liar, right? Hand-curating trillions of pieces of data?

andrewljohnson 18y ago
"Wolfram says that trillions of pieces of data were selected and managed by a team of experts at Wolfram Research, and that these experts also tweak the information to ensure that it can be read and displayed by the system." If 100 experts each curated 1000 pieces of data a day, it would take them 20,000,000 days to curate 2 trillion pieces of data. Has WA been in development for longer than I suspect, or am I missing something here?
- noodle 18y ago
  why does all new data have to be hand-curated and why just 1000 pieces of data a day? it says it makes use of lots of public data sets. i'm sure there's some implicit trust there. thats not to say that there isn't exaggeration happening. probably is. i'm just saying, lets not throw them under the bus just yet, either.
  - andrewljohnson 18y ago
    Well, I am giving them credit for 1000 pieces of data per person. Even if one person is curating 100,000 a day, it's still way too long of a time frame. I just object to someone throwing around the word trillion, related to something people do by hand. I understand it's probably not a straight out lie, but it's a dumb statement regardless.
- ableal 18y ago
  Let's take a power-amplifier-design approach to that (work back from output). Let's also say that by 'piece-of-data', we're talking about the famous pirate unit, the piece-of-eight, a.k.a. byte. So, we need an output of 2e12 bytes, produced by 100 experts, in, let's say, 200 days (about a metric work-year ;-). That's 10e8 bytes/expert.day. A 100 MB table of reliable non-trivial data per day. Not totally out of the question, but finding twenty thousand of those ...
mr_eel 18y ago
Just think about it; it’s likely that a lot of the information was already structured and comes from vetted sources. Of course they don’t inspect every single bit of data! Instead they’re likely to do doing some checks, editorialisation and general shepherding to make sure that the data is accurate.
dsil 18y ago
Think of it this way. Say someone was trying to get and use as much GPS data they could find to map the world's trails. Each source and format might require a different algorithm to "curate" it into a usable form for your data structures, but you certainly wouldn't have to tweak each individual bit of data.

"Wolfram says that trillions of pieces of data were selected and managed by a team of experts at Wolfram Research, and that these experts also tweak the information to ensure that it can be read and displayed by the system." If 100 experts each curated 1000 pieces of data a day, it would take them 20,000,000 days to curate 2 trillion pieces of data. Has WA been in development for longer than I suspect, or am I missing something here?

noodle 18y ago
why does all new data have to be hand-curated and why just 1000 pieces of data a day? it says it makes use of lots of public data sets. i'm sure there's some implicit trust there. thats not to say that there isn't exaggeration happening. probably is. i'm just saying, lets not throw them under the bus just yet, either.
- andrewljohnson 18y ago
  Well, I am giving them credit for 1000 pieces of data per person. Even if one person is curating 100,000 a day, it's still way too long of a time frame. I just object to someone throwing around the word trillion, related to something people do by hand. I understand it's probably not a straight out lie, but it's a dumb statement regardless.
ableal 18y ago
Let's take a power-amplifier-design approach to that (work back from output). Let's also say that by 'piece-of-data', we're talking about the famous pirate unit, the piece-of-eight, a.k.a. byte. So, we need an output of 2e12 bytes, produced by 100 experts, in, let's say, 200 days (about a metric work-year ;-). That's 10e8 bytes/expert.day. A 100 MB table of reliable non-trivial data per day. Not totally out of the question, but finding twenty thousand of those ...

mr_eel 18y ago

Just think about it; it’s likely that a lot of the information was already structured and comes from vetted sources. Of course they don’t inspect every single bit of data! Instead they’re likely to do doing some checks, editorialisation and general shepherding to make sure that the data is accurate.

dsil 18y ago

Think of it this way. Say someone was trying to get and use as much GPS data they could find to map the world's trails. Each source and format might require a different algorithm to "curate" it into a usable form for your data structures, but you certainly wouldn't have to tweak each individual bit of data.