3 ms·
In my experience the data just seems bigger for organizations when the proper hardware, software, and coding practices aren’t used.
by Guest42 6y ago
In my experience the data just seems bigger for organizations when the proper hardware, software, and coding practices aren’t used.
- p1esk 6y agoThese guys [1] trained a model to generate music. Used 512 GPU cluster for 4 weeks. Dataset is one million songs. Now imagine what would it take to generate video. 10 years from now everyone will want to generate something. [1] https://openai.com/blog/jukebox https://openai.com/blog/jukebox
- Guest42 6y agoGoodness, I worked at a startup that burned through 200k worth of cloud credits pretty fast but that’s amazing
- deleted 6y ago[deleted]
- TeMPOraL 6y agoI think the data should be measured in units of time. Like, how long it takes your software to touch every entry just once. This explains how the same amount of data may be peanuts or Big Data worthy of a Hadoop cluster, depending on how bloated and wasteful your software is.
- gonzo41 6y agoYou should expand your thinking to also consider big data apps such as GIS software and anything that processes multi layer remote sensing data. Many pixels, Many Layers. Sizes start in the GB.
- TeMPOraL 6y agoThis was just a way to have a basic, rough estimate; of course the actual "feel" of data depends on the computations you're doing on it. That said, it's usually the case that the longer you can keep the data on a single machine, the better off you are. Distributing your computing incurs steep performance costs (that you'll have to pay for in infrastructure) and vastly increases complexity of the software.