4 ms·

This guy makes the assumption that all problems are big problems. Most problems are NOT big problems.

by Aloha 13y ago

gaius 13y ago
It's a cargo cult thing - people think "if I do like Google, I will be successful like Google!" Data is getting bigger, sure. Movies in HD, umpteen-megapixel cameras, yadda yadda in the consumer space, but hardware is getting bigger too, for the same price. You can easily buy a terabyte drive now. My first HD was 20M and I thought I'd never need anywhere near that much! But in the corporate space - where is this data actually coming from? If you have a company selling widgets, then by how many orders of magnitude do your sales have to increase before you have to worry about "big data"? Most companies could probably handle a tenfold increase in sales on their existing systems, even if those systems haven't been updated in a few years. Bread and butter software work is not going to change in the foreseeable future, it'll just have a different set of buzzwords.
- Aloha 13y ago
  I see cargo cults all over, its my favorite way to explain the problem of people conflating causation and outcome. In business I see lots of folks who do this: "Well, if I do what XYZ does, I'll be successful just like they are!" So they do what XYZ does without understanding the WHY part of what XYZ does, and invariable either fail, or succeed and making a very poor copy.
jerseyguy 13y ago
I agree. If you can solve the problem in a reasonable amount of time on current high-end hardware, it's not a big problem. If the only way to meet demand is by distributing the solution, then you can start calling it a big problem.
- gaius 13y ago
  Distributed systems have been around for a long time.
  - jerseyguy 13y ago
    Of course. What I meant is that if you have a dataset for which you can give an answer by loading it all into memory on a single machine -- it's not a big problem. That isn't to say that every problem which is better solved using a distributed system is a big problem.
  - stirno 13y ago
    This times 1000. Most developers don't realize that we're just starting to circle around again (this is a known, common pattern) and rediscover the reason for complex distributed systems from our past. Now we'll just do it with shinier tools. As problem spaces expand beyond our current hardwares ability, we start distributing it until a new wave of equipment becomes available and we start labeling our predecessors idiots for having ever built something so complex.
  - Aloha 13y ago
    Yes, and while you can solve many problems with parallelism, that doesn't mean you should.

This guy makes the assumption that all problems are big problems. Most problems are NOT big problems.

by Aloha 13y ago

This guy makes the assumption that all problems are big problems. Most problems are NOT big problems.

gaius 13y ago

It's a cargo cult thing - people think "if I do like Google, I will be successful like Google!" Data is getting bigger, sure. Movies in HD, umpteen-megapixel cameras, yadda yadda in the consumer space, but hardware is getting bigger too, for the same price. You can easily buy a terabyte drive now. My first HD was 20M and I thought I'd never need anywhere near that much! But in the corporate space - where is this data actually coming from? If you have a company selling widgets, then by how many orders of magnitude do your sales have to increase before you have to worry about "big data"? Most companies could probably handle a tenfold increase in sales on their existing systems, even if those systems haven't been updated in a few years. Bread and butter software work is not going to change in the foreseeable future, it'll just have a different set of buzzwords.

Aloha 13y ago
I see cargo cults all over, its my favorite way to explain the problem of people conflating causation and outcome. In business I see lots of folks who do this: "Well, if I do what XYZ does, I'll be successful just like they are!" So they do what XYZ does without understanding the WHY part of what XYZ does, and invariable either fail, or succeed and making a very poor copy.

jerseyguy 13y ago

I agree. If you can solve the problem in a reasonable amount of time on current high-end hardware, it's not a big problem. If the only way to meet demand is by distributing the solution, then you can start calling it a big problem.

gaius 13y ago
Distributed systems have been around for a long time.
- jerseyguy 13y ago
  Of course. What I meant is that if you have a dataset for which you can give an answer by loading it all into memory on a single machine -- it's not a big problem. That isn't to say that every problem which is better solved using a distributed system is a big problem.
- stirno 13y ago
  This times 1000. Most developers don't realize that we're just starting to circle around again (this is a known, common pattern) and rediscover the reason for complex distributed systems from our past. Now we'll just do it with shinier tools. As problem spaces expand beyond our current hardwares ability, we start distributing it until a new wave of equipment becomes available and we start labeling our predecessors idiots for having ever built something so complex.
- Aloha 13y ago
  Yes, and while you can solve many problems with parallelism, that doesn't mean you should.