2 ms·
I think, as experiments published in last year based on Spark, the inefficiency stems from the assumption that CPU cycles are abundant w.r.t. RAM and bandwidth.
by riyadparvez 10y ago
I think, as experiments published in last year based on Spark, the inefficiency stems from the assumption that CPU cycles are abundant w.r.t. RAM and bandwidth. So, nobody focuses on optimizing program itself as much as to reduce memory or bandwidth usage.
- acjohnson55 10y agoSeeing the talk today, I think the Spark team is very aware that CPU cycles haven't become more abundant, while I/O has. There was an entire slide to this effect. I don't know enough about the space to say more than that, but it seems it's definitely on their mind. One of the things I remember being mentioned is operating on data in a much more efficient binary representation than native Java memory model, allowing better use of cache and such.
- dhoe 10y agoThat's not an assumption, it's a fact.