3 ms·

I quote my friend who works in "Big Data": "sometimes I think Big Data is just Excel on 128GB of RAM"

by hkmurakami 13y ago

dasil003 13y ago
Not to go all senselessly pedantic, but doesn't Excel have a limit of like 55,000 rows?
- hkmurakami 13y ago
  He's saying his file has way more rows than that, so maybe they upped the limit in the more recent versions of excel? (I think he also wrote a bunch of VBA and hooked into some external systems too)
- objclxt 13y ago
  If you're being pedantic, it's 1,048,576 rows from Excel 2007, 65,536 rows before that.
  - hkmurakami 13y ago
    2^20 rows! :)
- Joeri 13y ago
  16 384 columns and 1 048 576 rows actually
  - n00b101 13y ago
    2^14 columns * 2^20 rows * 8 bytes per cell = 128GB. Bang on.
- dagw 13y ago
  Current limit is 1,000,000 pr worksheet. However there is a tool called PowerPivot which lets you get around that limit and do analysis on larger data sets.
  - chinpokomon 13y ago
    I didn't see this response when I replied to the parent. Power Pivot is pretty great when you can use it.
- deleted 13y ago
  [deleted]
- jjcr 13y ago
  Not since 2007 where it got bumped to over 1 million. http://office.microsoft.com/en-us/excel-help/excel-specifications-and-limits-HA103980614.aspx http://office.microsoft.com/en-us/excel-help/excel-specifica...
- willismichael 13y ago
  The devil is in the details. Big Data is really a massive cluster of VMs running maxed out Excel spreadsheets, and instrumented to restart automatically and restore from redundant backup, a la RAID, when the Excel process crashes one of the Windows VMs.
- chinpokomon 13y ago
  Not when you use it with Power Pivot.
daemonk 13y ago
Big data is a misnomer. Complex data is a better description. Having a terabyte of simple data with 2 columns is really not that difficult to analyze and won't give you much information. Whereas having a few hundred mbs data with complex relationships and many dimensions can yield tons of information and is far more difficult to analyze. Difficulty in "big data" should be about its horizontal breadth (covering many aspects of a system) rather than its vertical depth (covering one aspect of a system in great resolution).

I quote my friend who works in "Big Data": "sometimes I think Big Data is just Excel on 128GB of RAM"

dasil003 13y ago

Not to go all senselessly pedantic, but doesn't Excel have a limit of like 55,000 rows?

hkmurakami 13y ago
He's saying his file has way more rows than that, so maybe they upped the limit in the more recent versions of excel? (I think he also wrote a bunch of VBA and hooked into some external systems too)
objclxt 13y ago
If you're being pedantic, it's 1,048,576 rows from Excel 2007, 65,536 rows before that.
- hkmurakami 13y ago
  2^20 rows! :)
Joeri 13y ago
16 384 columns and 1 048 576 rows actually
- n00b101 13y ago
  2^14 columns * 2^20 rows * 8 bytes per cell = 128GB. Bang on.
dagw 13y ago
Current limit is 1,000,000 pr worksheet. However there is a tool called PowerPivot which lets you get around that limit and do analysis on larger data sets.
- chinpokomon 13y ago
  I didn't see this response when I replied to the parent. Power Pivot is pretty great when you can use it.
deleted 13y ago
[deleted]
jjcr 13y ago
Not since 2007 where it got bumped to over 1 million. http://office.microsoft.com/en-us/excel-help/excel-specifications-and-limits-HA103980614.aspx http://office.microsoft.com/en-us/excel-help/excel-specifica...
willismichael 13y ago
The devil is in the details. Big Data is really a massive cluster of VMs running maxed out Excel spreadsheets, and instrumented to restart automatically and restore from redundant backup, a la RAID, when the Excel process crashes one of the Windows VMs.
chinpokomon 13y ago
Not when you use it with Power Pivot.

daemonk 13y ago

Big data is a misnomer. Complex data is a better description. Having a terabyte of simple data with 2 columns is really not that difficult to analyze and won't give you much information. Whereas having a few hundred mbs data with complex relationships and many dimensions can yield tons of information and is far more difficult to analyze. Difficulty in "big data" should be about its horizontal breadth (covering many aspects of a system) rather than its vertical depth (covering one aspect of a system in great resolution).