3 ms·
Most companies probably realized that they don't have Big Data problems because they only have a limited amount of data which you actually can process in an acc
by aeyes 4y ago
Most companies probably realized that they don't have Big Data problems because they only have a limited amount of data which you actually can process in an acceptable amount of time on a single Postgres instance. Distributed data processing has a huge upfront tax and you really only want to be doing it if the data set is enormous.
I guess it is similar to other technologies which most companies or developers would really never need due to their limited scale like distributed databases, NoSQL or microservices: It is interesting technology and engineers would like to get their hands on it because that's what the big boys play with, even if they don't really need it. In the meantime the industry hypes it because the technology is difficult so they know that they can make money doing consulting.
I'm not saying that it is not useful technology, I work at a company where we had the need to go from Postgres to "Big Data" tooling. But for tons of businesses it just doesn't make any sense. And even in our case one of the questions I have most frequently is: What business decision are you taking based on processing this enormous amount of data? Can we not take the same decision based on less data?
- SOLAR_FIELDS 4y agoIt’s been the same experience for me. Until you get into petabyte levels of data a single replicated and vertically scaled pg is probably going to be just fine. Quite a few orgs probably realized that eventually after sinking $MM into some Hadoop/Spark setup in the mid 2010’s. If what you’re doing is 1. Easily parallelizable 2. CPU intensive 3. In 10’s of petabytes or more Then one of these machine gun like setups makes sense in 2022. Otherwise YAGNI (you aren’t gonna need it)
- ryanjkirk 4y ago"For Basecamp, we take a somewhat different path. We are on record about our feelings about sharding. We prefer to use hardware to scale our databases as long as we can, in order to defer the complexity that is involved in partitioning them as long as possible — with any luck, indefinitely." — Mark Imbriaco, 37Signals These quotes are from 2009 and 2010, and yet here most of us are in 2022, having learned the lesson the hard way over the last decade that there is no refuting this simple logic. I'll add my own truism: All else being equal, designing and maintaining simpler systems will always cost less than complex ones. quote references: https://signalvnoise.com/posts/2479-nuts-bolts-database-servers https://signalvnoise.com/posts/2479-nuts-bolts-database-serv... http://37signals.com/svn/posts/1509-mr-moore-gets-to-punt-on-sharding http://37signals.com/svn/posts/1509-mr-moore-gets-to-punt-on...
- eb0la 4y agoDon't forget evolution: Business do change. A simple sistem far easier to evolve than a complex one, too.