4 ms·

This is interesting. Without fully understanding the design of this system, I wonder how far it scales. Can you process big data with LoCal?

jacobush 7y ago
Is it web scale? :-) Sorry couldn’t resist... https://m.youtube.com/watch?v=b2F-DItXtZs https://m.youtube.com/watch?v=b2F-DItXtZs
- misterman0 7y ago
  I actually thought that by now, in 2019, this forum would have realized that big data is real. It's a thing. Some of us are forced to deal with it. It's a hard problem that is well-suited for having discussions around on forums that are populated by intelligent people, but apparently it's not a suitable topic on this particular forum. I will certainly keep that in mind from hereon out.
  - tempguy9999 7y ago
    Big data is mainly a marketing term to me. I've been involved in a Big Data project that wasn't because someone wanted to massage their CV with fancy tech. With current multicore machines and SSDs or cheap spinny disks you can hold and process incredible amounts of data. So, serious question, how do you define big data? By some size threshold, or "it can't possibly be done on a single box", or something else? And If I may, what is the BD work that you do (feel free to decline that one). TIA
  - kjs3 7y ago
    I actually thought that by now, in 2019, that people doing big data work understood that big data covered a vast number of processing use cases and saying "can it process big data" has about as much semantic meaning as "can it process blue". I will certainly keep that in mind from hereon out.
rrnewton 7y ago
Well, if you have a big document collection or something, the faster you can process each individual document the more you can scale, right? I guess I'm not sure what scalability would mean here other than speeding up that individual, sequential processing job on one node (e.g. the map step in a MapReduce).

jacobush 7y ago

Is it web scale? :-) Sorry couldn’t resist... https://m.youtube.com/watch?v=b2F-DItXtZs https://m.youtube.com/watch?v=b2F-DItXtZs

misterman0 7y ago
I actually thought that by now, in 2019, this forum would have realized that big data is real. It's a thing. Some of us are forced to deal with it. It's a hard problem that is well-suited for having discussions around on forums that are populated by intelligent people, but apparently it's not a suitable topic on this particular forum. I will certainly keep that in mind from hereon out.
- tempguy9999 7y ago
  Big data is mainly a marketing term to me. I've been involved in a Big Data project that wasn't because someone wanted to massage their CV with fancy tech. With current multicore machines and SSDs or cheap spinny disks you can hold and process incredible amounts of data. So, serious question, how do you define big data? By some size threshold, or "it can't possibly be done on a single box", or something else? And If I may, what is the BD work that you do (feel free to decline that one). TIA
- kjs3 7y ago
  I actually thought that by now, in 2019, that people doing big data work understood that big data covered a vast number of processing use cases and saying "can it process big data" has about as much semantic meaning as "can it process blue". I will certainly keep that in mind from hereon out.

rrnewton 7y ago

Well, if you have a big document collection or something, the faster you can process each individual document the more you can scale, right? I guess I'm not sure what scalability would mean here other than speeding up that individual, sequential processing job on one node (e.g. the map step in a MapReduce).