4 ms·
It does remind me bit of "Big Data is Dead", which I believe was written by someone who worked on BigQuery (external version of Dremel), and then realized it do
by chubot 2y ago
It does remind me bit of "Big Data is Dead", which I believe was written by someone who worked on BigQuery (external version of Dremel), and then realized it doesn't actually add much value for customers:
Big Data is Dead (2023) - https://news.ycombinator.com/item?id=34694926 https://news.ycombinator.com/item?id=34694926
My opinion is that Google culture enforces scale because it can lead to doing something competitors can't, or if you want to put it less charitably, "monopoly"
For example, there are stories YouTube being a few months from falling over when it was acquired, and being saved by Google scale
At the time, I do think it was true that few other companies could scale to what YouTube users demanded, or at least absorb that cost
---
So the company has a culture of scale, for sure, because it has business value FOR GOOGLE
I remember Marissa Mayer sent out "product guidance", and said that technical advantage is one of the criteria, i.e. being able to do something that a competitor can't do
The market is very hostile -- profits are eroded away by competition. But yeah for most companies, like approximately all Google Cloud customers, scale is not the reason they exist. The product is the reason they exist
By definition, there can only be one company that uses scale as a competitive weapon :)
i.e. there is only one "most scalable" company
Of course that's not strictly true, because there are different dimensions of scale, but it's perhaps a useful way of thinking about why Google culture is obsessed with scale. And conversely, why Google seemingly ignores and sunsets products that don't scale.
- nasretdinov 2y agoYeah, I think people just don't fully realise how powerful hardware has become. You don't need complicated tools if you need to transfer even significantly large amounts of data, since you can have hundreds of gigabits of network capacity on a single server, so unless you're truly large you probably can get away with using vanilla rsync for file transfers for example :).
- waych 2y agoManaged async bulk transfers as-a-service seems relevant to anyone writing lots of data doing multisite including DR. You can easily have this problem and not have yet hit any other scale problems that are addressed by the alluded-to tools people love to hate (k8s, tf, containers etc). What is being managed here with Effingo is the relatively tiny links between sites, not the bandwidth on the fat local links. Bandwidth needs to be managed and prioritized across all the various copies that are happening across different services apps and teams simultaneously across the company, and across the systems themselves as aggregate for the links. Managing it gives you control on cost but also ensures optimal use within the constraints imposed and consistent and reliable handling of the copy use case.