4 ms·

I think the core problem he was talking about is that people use tools that are bigger than they need to use. If you have 100TB+ of data, you might need a clus

I think the core problem he was talking about is that people use tools that are bigger than they need to use. If you have 100TB+ of data, you might need a cluster. Think about it, Google has to deal with all of the pages on the internet; pretty much no one else has that data. So why are we using their methodologies?

switch33 13y ago
This is exactly what I'm talking about. Even layman people can use their regular computers for more interesting activities like machine learning on decent sized data if it wasn't so focused on just throw your cloud instance up using hadoop and run stuff that way. There are many people out there that can accomplish some of these tasks with better written code. There is also probably a need for a better hadoop. One that is less about clusters/multiple cpus and more about rented out bigger power computers.
- asperous 13y ago
  Hadoop is specifically for clustered computer, it doesn't really make sense otherwise. Like the speaker said, it's really simplified (though you can do a lot of stuff with map/reduce). In some ways the guy is probably blowing some steam, simply grabbing a bigger computer isn't always the best option, that's why clustered computing was even invented. I do think he has a point that people reach for it sooner than they need to. I've made that mistake before, and I just ended up settling for the pipeline approach.
  - switch33 13y ago
    The other point i'm trying to really make is maybe hadoop does serve some purpose but there could be an easier/better approach for parallelism. If we focused harder on reliability and another service "like hadoop" would probably be created but for non-cluster/multicpu. And doing some of the decently sized tasks we would have better, faster, and more reliable tools at hand.

4 ms·

I think the core problem he was talking about is that people use tools that are bigger than they need to use. If you have 100TB+ of data, you might need a clus

by asperous 13y ago

switch33 13y ago
This is exactly what I'm talking about. Even layman people can use their regular computers for more interesting activities like machine learning on decent sized data if it wasn't so focused on just throw your cloud instance up using hadoop and run stuff that way. There are many people out there that can accomplish some of these tasks with better written code. There is also probably a need for a better hadoop. One that is less about clusters/multiple cpus and more about rented out bigger power computers.
- asperous 13y ago
  Hadoop is specifically for clustered computer, it doesn't really make sense otherwise. Like the speaker said, it's really simplified (though you can do a lot of stuff with map/reduce). In some ways the guy is probably blowing some steam, simply grabbing a bigger computer isn't always the best option, that's why clustered computing was even invented. I do think he has a point that people reach for it sooner than they need to. I've made that mistake before, and I just ended up settling for the pipeline approach.
  - switch33 13y ago
    The other point i'm trying to really make is maybe hadoop does serve some purpose but there could be an easier/better approach for parallelism. If we focused harder on reliability and another service "like hadoop" would probably be created but for non-cluster/multicpu. And doing some of the decently sized tasks we would have better, faster, and more reliable tools at hand.