4 ms·

Hooray! Some sense at last. I have worked for at least 3 different employers that claimed to be using "Big Data". Only one of them was really telling the truth

Hooray! Some sense at last. I have worked for at least 3 different employers that claimed to be using "Big Data". Only one of them was really telling the truth. All of them wanted to feel like they were doing something special. The sad thing is, they were all special, each in their own particular way, but none of what made each company magic and special had anything to do with the size of the data that they were handling. Hadoop was required in exactly zero of these cases. Funnily enough, after I left one of them, they started to build lots of Hadoop-based systems for reasons which, as far as I could fathom, had more to do with the resumes of the engineers involved than the actual technical merits of the case. Sad, but 'tis the way of the world.

netcan 13y ago
Tricky catch 22. You want the kind of employees that are interested in what they do enough to want to learn new skills and try new exciting technologies. But, you want them to actually use something boring.
- Blahah 13y ago
  There's nothing boring about pandas
  - dalke 13y ago
    It's even more exciting mixing pandas with the IPython notebook.
    - edraferi 13y ago
      heh. Until you actually have to repeat something! It's quite useful for iterative refinements like tweaking pyplot graphics, but it's all to easy to lose track of what versions of which blocks executed to create the current state. Maybe I just need more discipline..
      - dalke 13y ago
        Walking on a highwire is also exciting. :)
  - freyrs3 13y ago
    Until you run out of memory.
    - Blahah 13y ago
      that hardly makes it boring... it's just inappropriate for some kinds of task
    - estebank 13y ago
      That's when the fun starts!
dalke 13y ago
Agreed, but I'm not optimistic that the view will become wide-spread. There was a phrase I learned long ago: "if it fits into cache it's not a supercomputing problem." This essay promotes the related view: "If it fits on a desktop, it's not Big Data." BTW, you might like Greg Wilson's 2008 essay "HPC Considered Harmful". I can't find the slides anymore, but there's a lovely interview with him about this at http://itc.conversationsnetwork.org/shows/detail3682.html http://itc.conversationsnetwork.org/shows/detail3682.html .
- jerf 13y ago
  Just because it's easy to lose track, I just checked, and "Stereotypical Grandma walks into Best Buy and buys their best desktop because she doesn't know any better" "(and probably got bamboozled by the sales guy)" ends up with Grandma having 16GB of RAM for $900.
harrytuttle 13y ago
Yes I've worked for a couple of outfits that did this over the years. It's usually down to who sold them the solution though. It's worst if the salesperson is external. The funniest has to be the engineering company which I used to work with (not for thank fuck) back in the 90's (when 100Gb was big data!). They managed to bag 2x full 42U racks full of HP N-Class HPUX UNIX kit, PDU systems and separate disk arrays for just over £1,000,000 supplied with one full time UNIX monkey to keep the plates spinning. Turns out they had only 10 CAD/engineering staff and about 8Gb of data in an Oracle DB which a single Sun Ultra desktop (<£10,000) could have handled without a problem at the time.
- lukeschlather 13y ago
  When in the 90's? I'd expect it would have taken the full 2x 42U racks to keep the whole dataset in memory at one time. Not to say it's required, but that doesn't sound totally crazy for the time period. Also, having an order of magnitude more hardware than you ought to need is totally understandable when hardware capability is increasing by an order of magnitude faster than hardware replacement cycles. It's when you're off by several orders of magnitude that it starts to become suspect.
  - harrytuttle 13y ago
    1998. It had 4Gb of RAM in each node which wasn't that scary. Most of it was UPS and disk. I had a Sun 1000E [1] for 3 years as a workstation from 1996. 8x 50MHz CPU, 2Gb of RAM, 16Gb disk array. It was made in'93 and was approx 12U for the unit and array. Things weren't that big in the 90's. 80's yes... [1] http://en.m.wikipedia.org/wiki/Sun4d http://en.m.wikipedia.org/wiki/Sun4d
- w_t_payne 13y ago
  Yeah... it looks like my employer is going down the same route with an EMC Isilon NAS system. We really just want high throughput for linear scans; possibly with a small amount of parallelism, which would argue for a fairly simple solution with as little cacheing as possible (ideally none whatsoever) ... but I cannot even tell (from the website & the literature that I can access) how these things are supposed to operate -- which is a worrying sign in itself. Previously, I got good performance out of a SAN using a handful of NexSan SataBeasts -- which I would have recommended to use again, if anybody had bothered to ask. :-)
mathattack 13y ago
There is principal-agent problems in a lot of companies. CEOs want to run bigger companies, even if it's not best for shareholders. Managers want to run large mini-empires, even if it's not best for the corporation. Engineers like to use the latest tools, even when they're not best for the problems at hand. Solving this requires well respected leadership that knows enough about the details, and which battles are worth fighting.

netcan 13y ago

Tricky catch 22. You want the kind of employees that are interested in what they do enough to want to learn new skills and try new exciting technologies. But, you want them to actually use something boring.

Blahah 13y ago
There's nothing boring about pandas
- dalke 13y ago
  It's even more exciting mixing pandas with the IPython notebook.
  - edraferi 13y ago
    heh. Until you actually have to repeat something! It's quite useful for iterative refinements like tweaking pyplot graphics, but it's all to easy to lose track of what versions of which blocks executed to create the current state. Maybe I just need more discipline..
    - dalke 13y ago
      Walking on a highwire is also exciting. :)
- freyrs3 13y ago
  Until you run out of memory.
  - Blahah 13y ago
    that hardly makes it boring... it's just inappropriate for some kinds of task
  - estebank 13y ago
    That's when the fun starts!

dalke 13y ago

Agreed, but I'm not optimistic that the view will become wide-spread. There was a phrase I learned long ago: "if it fits into cache it's not a supercomputing problem." This essay promotes the related view: "If it fits on a desktop, it's not Big Data." BTW, you might like Greg Wilson's 2008 essay "HPC Considered Harmful". I can't find the slides anymore, but there's a lovely interview with him about this at http://itc.conversationsnetwork.org/shows/detail3682.html http://itc.conversationsnetwork.org/shows/detail3682.html .

jerf 13y ago
Just because it's easy to lose track, I just checked, and "Stereotypical Grandma walks into Best Buy and buys their best desktop because she doesn't know any better" "(and probably got bamboozled by the sales guy)" ends up with Grandma having 16GB of RAM for $900.

harrytuttle 13y ago

Yes I've worked for a couple of outfits that did this over the years. It's usually down to who sold them the solution though. It's worst if the salesperson is external. The funniest has to be the engineering company which I used to work with (not for thank fuck) back in the 90's (when 100Gb was big data!). They managed to bag 2x full 42U racks full of HP N-Class HPUX UNIX kit, PDU systems and separate disk arrays for just over £1,000,000 supplied with one full time UNIX monkey to keep the plates spinning. Turns out they had only 10 CAD/engineering staff and about 8Gb of data in an Oracle DB which a single Sun Ultra desktop (<£10,000) could have handled without a problem at the time.

lukeschlather 13y ago
When in the 90's? I'd expect it would have taken the full 2x 42U racks to keep the whole dataset in memory at one time. Not to say it's required, but that doesn't sound totally crazy for the time period. Also, having an order of magnitude more hardware than you ought to need is totally understandable when hardware capability is increasing by an order of magnitude faster than hardware replacement cycles. It's when you're off by several orders of magnitude that it starts to become suspect.
- harrytuttle 13y ago
  1998. It had 4Gb of RAM in each node which wasn't that scary. Most of it was UPS and disk. I had a Sun 1000E [1] for 3 years as a workstation from 1996. 8x 50MHz CPU, 2Gb of RAM, 16Gb disk array. It was made in'93 and was approx 12U for the unit and array. Things weren't that big in the 90's. 80's yes... [1] http://en.m.wikipedia.org/wiki/Sun4d http://en.m.wikipedia.org/wiki/Sun4d
w_t_payne 13y ago
Yeah... it looks like my employer is going down the same route with an EMC Isilon NAS system. We really just want high throughput for linear scans; possibly with a small amount of parallelism, which would argue for a fairly simple solution with as little cacheing as possible (ideally none whatsoever) ... but I cannot even tell (from the website & the literature that I can access) how these things are supposed to operate -- which is a worrying sign in itself. Previously, I got good performance out of a SAN using a handful of NexSan SataBeasts -- which I would have recommended to use again, if anybody had bothered to ask. :-)

mathattack 13y ago

There is principal-agent problems in a lot of companies. CEOs want to run bigger companies, even if it's not best for shareholders. Managers want to run large mini-empires, even if it's not best for the corporation. Engineers like to use the latest tools, even when they're not best for the problems at hand. Solving this requires well respected leadership that knows enough about the details, and which battles are worth fighting.