4 ms·

Also, Google's infrastructure is so vertically integrated that it's not obvious how to offer it others. E.g. Borg assigns 1 IP to each machine, which means ever

Also, Google's infrastructure is so vertically integrated that it's not obvious how to offer it others. E.g. Borg assigns 1 IP to each machine, which means every pod-equivalent has to listen on a dynamically allocated port, so Google eschewed DNS in favor of a system that more easily assigned dynamic IP:port pairs without relying on TTLs (BNS/Chubby). Google also decoupled storage from compute and had a bunch of custom databases to run in that paradigm (GFS/CFS/BigTable).

deanCommie 5y ago
The mistake also reveals the constrained imagination: Google could only conceive of offering what they had internally to others, struggled, and continues to struggle. Amazon didn't try - they just knew that they were good at building high scale distributed systems, and thought they could build services for others. That's what S3 and SQS and EC2 were - built from scratch, not trying to figure out how to externalize the internals.
- ec109685 5y ago
  GCP was built from scratch as well.
  - singron 5y ago
    It's more complicated, but I think everything is at least a substantial layer on top of an internal offering. E.g. GCE runs VMs in Borg using the same physical infrastructure used for other services, but GCE has its own API and virtualized networking on top that hides Borg and was not (at the time) used for any internal services.
ra7 5y ago
I’ve heard Google considers (considered?) their infrastructure prowess as a secret sauce. Could it be that they didn’t want anyone else to benefit from it? They had years of headstart on most infrastructure technology that’s ubiquitous now, but didn’t bother to offer it to others. Perhaps they thought only Google should benefit from it, while Amazon saw a business opportunity with AWS.
- oblio 5y ago
  I'm also curious about this. The empirical evidence seems to point that way, a lot of people were noticing that many of the Google stuff became open source or got open source equivalents from Google (Borg/Kubernetes, Blaze/Bazel, etc.) only after a critical mass of former Google employees left Google and since they liked Google tooling started replicating it as open source software outside of Google. In some cases that open source software got traction and Google seemed to be falling behind in some areas. Which basically tells me that in the software world, you can't really have trade secrets. I could be wrong, though.
  - buck4roo 5y ago
    Don't forget Prometheus (clone of borgmon by ex-SRE)
    - jeffbee 5y ago
      There is no way to perceive Prometheus as Google falling behind, though. It's more of a lesson about the perverse willingness of the public to subject themselves to terrible technologies that Google began deprecating years ago.
      - oblio 5y ago
        Why is Prometheus terrible? It's pretty cool.
        jeffbee 5y ago
        At small scale it's fine. Its architecture totally forecloses any possibility of distributed evaluation, so there is a brick wall of scalability. Its human-readable-text-over-HTTP scraping scheme is also hilariously inefficient. It is not, in short, "planet scale". https://www.vldb.org/pvldb/vol13/p3181-adams.pdf https://www.vldb.org/pvldb/vol13/p3181-adams.pdf
        5y ago

deanCommie 5y ago

The mistake also reveals the constrained imagination: Google could only conceive of offering what they had internally to others, struggled, and continues to struggle. Amazon didn't try - they just knew that they were good at building high scale distributed systems, and thought they could build services for others. That's what S3 and SQS and EC2 were - built from scratch, not trying to figure out how to externalize the internals.

ec109685 5y ago
GCP was built from scratch as well.
- singron 5y ago
  It's more complicated, but I think everything is at least a substantial layer on top of an internal offering. E.g. GCE runs VMs in Borg using the same physical infrastructure used for other services, but GCE has its own API and virtualized networking on top that hides Borg and was not (at the time) used for any internal services.

ra7 5y ago

I’ve heard Google considers (considered?) their infrastructure prowess as a secret sauce. Could it be that they didn’t want anyone else to benefit from it? They had years of headstart on most infrastructure technology that’s ubiquitous now, but didn’t bother to offer it to others. Perhaps they thought only Google should benefit from it, while Amazon saw a business opportunity with AWS.

oblio 5y ago
I'm also curious about this. The empirical evidence seems to point that way, a lot of people were noticing that many of the Google stuff became open source or got open source equivalents from Google (Borg/Kubernetes, Blaze/Bazel, etc.) only after a critical mass of former Google employees left Google and since they liked Google tooling started replicating it as open source software outside of Google. In some cases that open source software got traction and Google seemed to be falling behind in some areas. Which basically tells me that in the software world, you can't really have trade secrets. I could be wrong, though.
- buck4roo 5y ago
  Don't forget Prometheus (clone of borgmon by ex-SRE)
  - jeffbee 5y ago
    There is no way to perceive Prometheus as Google falling behind, though. It's more of a lesson about the perverse willingness of the public to subject themselves to terrible technologies that Google began deprecating years ago.
    - oblio 5y ago
      Why is Prometheus terrible? It's pretty cool.
      - jeffbee 5y ago
        At small scale it's fine. Its architecture totally forecloses any possibility of distributed evaluation, so there is a brick wall of scalability. Its human-readable-text-over-HTTP scraping scheme is also hilariously inefficient. It is not, in short, "planet scale". https://www.vldb.org/pvldb/vol13/p3181-adams.pdf https://www.vldb.org/pvldb/vol13/p3181-adams.pdf
        5y ago