3 ms·
Lots of interesting discussion here about billions of files and what not. I don't really work with big data -- the largest amounts of data I work with are in th
by atombender 7y ago
Lots of interesting discussion here about billions of files and what not. I don't really work with big data -- the largest amounts of data I work with are in the range of tens of millions at most, and my modest "ETL" is about reading from APIs and cleaning up incoming data into consistent schemas, refining data (e.g. geocoding addresses, or correlating against public demographic records/statistics), a process that is super fast and doesn't even require multiple nodes for the most part -- so I never encounter anything I can't solve with some Postgres or Redis. What I'd love to know is what everyone is actually doing with their huge clusters. I can understand the need for big data in hard sciences (CERN, or genetics, or similar), and of course machine learning (e.g. image feature extraction), and then there are real time auction use cases like ad servers that probably have some big data component. What else are people doing out there?
- heyyyouu 7y agoThis is a great question -- I'd love to know this too. I watch this space from the enterprise perspective and I see a lot of big data talk/prep but not so much execution, if that makes sense.