3 ms·
It became ubiquitous. Now it is in many places. Enterprises use it each moment. A laptop hard disk is now capable of holding databases with tens of millions o
by rg111 4y ago
It became ubiquitous.
Now it is in many places. Enterprises use it each moment.
A laptop hard disk is now capable of holding databases with tens of millions of rows.
Traditional "Data Science" and modern Deep Learning rely entirely on it. Millions of datapoints are used to create models everyday.
A sensor on human wrist collects and stores thousands of data points each day.
So do refrigrators, cars, and your washing machine with ubiquity of IoT.
Giant tech cos use billions of rows each day to show users products, or sell their attention as products.
Big Data became ubiquitous. And it became so common that no nody calls it that anymore.
Tools like BigQuery, Dask, and even Pandas and SQL can handle hundreds of thousands to hundreds of millions of rows (or other structure) with normal, regular programming, command, etc.
- zasdffaa 4y ago> A laptop hard disk is now capable of holding databases with tens of millions of rows. > A sensor on human wrist collects and stores thousands of data points each day. I feel we have a very different view of what comprises big data :)
- rg111 4y agoYeah I work with hundreds of GBs of data every day. I have worked on a dataset with 40 million images in the past. I am also aware of OpenAI, Google AI, etc. train on billions of images. Internally, I am aware of Amazon, Google, META handling really large datasets. But that is now. If you are old enough you must remember the early years of big data hype. It was not far from millions or tens of millions of rows. Re: sensors on fitbits, I thought everyone would read between the lines, and consider hundreds of thousands of these devices sold every year (every month?) will definitely amount to "big data". Either these companies are plainly hoarding all of it and running some kind of analysis, or they maybe are doing federated learning. From the cos' standpoint, yes, it is big data.
- bakuninsbart 4y agoFor a blow of the past for younger people I can really recommend Tanenbaum's Computer Architecture. Great book for understanding how a computer works, and the practical example he gives really illuminate well how far we've come!