4 ms·

This is part of a worrying new trend. Increasingly you can't buy software anymore, only rent. Innovation is being kept from scrutiny hidden behind closed doors

by slackingoff2017 9y ago

This is part of a worrying new trend. Increasingly you can't buy software anymore, only rent. Innovation is being kept from scrutiny hidden behind closed doors. The kind of thing patents were meant to prevent back when the system wasn't broken. Google is one of the better players in this regard, at least telling the world what they're up to. Try to figure out how something like Amazon's systems work and you'll run into a deafening wall of silence. Funny that we're so willing to trust these "clouds" when we know next to nothing about their internal workings. I don't think the honeymoon will last forever. Somebody will eventually abuse their position and within a few years everyone will be "on prem" again.

elvinyung 9y ago
Interestingly, Amazon is also presenting a paper this year at SIGMOD, about Aurora: http://dl.acm.org/authorize?N37778 http://dl.acm.org/authorize?N37778 If my memory serves, it's the second paper they've ever published (the other one being the famous Dynamo paper[1]). [1] http://cloudgroup.neu.edu.cn/papers/cloud%20data%20storage/dynamo-sosp-2007.pdf http://cloudgroup.neu.edu.cn/papers/cloud%20data%20storage/d...
brandur 9y ago
> This is part of a worrying new trend. Increasingly you can't buy software anymore, only rent. More or less agreed, but you'd probably have to concede that the existence of papers like this one and some of Google's other Spanner publications are admirable; they're being more open about the system's design than they have to be. Of course Google knows that the system's secret sauce is not the concept itself, but the cost of its implementation, the infrastructural harness to support it, and the resources to reliably operate it. Even with a rough understanding on how Spanner works, it's still going to be difficult to ever migrate off it for practical reasons alone — who else is going to be able to build and run an alternative? I have huge respect for what the Spanner team is doing, but this is a reason that Citus [1] is also very interesting to me right now. You could conceivably start out with nothing but Postgres and migrate into a Citus cluster when (and if) you need to. If a point comes where you realize that you need out, you could either (1) see if you can scale back down to simple Postgres, (2) host your own cluster on your own infrastructure with the Citus source code, or (3) migrate onto your own Postgres sharding scheme à la Instagram. At no point do you lock yourself into custom GRPC APIs which are going to be ~impossible to get off of. GCP and AWS provide hugely useful foundational IaaS, but they're incentivized to move beyond that layer and provide more custom solutions that (1) provide better margins, and (2) lock you into their services. As people and companies building on top of these clouds, we should be looking for whatever opportunities we can to keep our stacks generic so that AWS <-> GCP <-> Azure migrations are possible, even if a last resort. [1] https://www.citusdata.com/product/cloud https://www.citusdata.com/product/cloud
- slackingoff2017 9y ago
  Does this feel like rent seeking to you? The internet is not young anymore. It feels like cloud hosting is little more than the giants allowing controlled competition as long as you rent their servers. The platforms are just extensions of the massively powerful systems they use internally. I wonder what would happen if the open source community built a viable alternative to the cloud IaaS. Like OpenStack but not a failure :). OpenFlow has shown promise and could form the core for an open IaaS. Network virtualization is the hardest part.
  - speedplane 9y ago
    > I wonder what would happen if the open source community built a viable alternative to the cloud IaaS. ... Network virtualization is the hardest part. I would think that the hardware itself is the hardest part. Companies move to the cloud because it reduces their internal OPs team, and you can scale up and down hardware extremely easily.
    - slackingoff2017 9y ago
      I respectfully disagree. Every company I've been part of moving to the cloud already had plenty of hardware. It was ease of provisioning VM's, backups, deployment, and networking that really convinced them. VMWare comes close to offering the same thing on premesis but Oracle is too obsessed with wringing money from it to let it reach it's full potential.
- speedplane 9y ago
  > GCE and AWS provide hugely useful foundational IaaS, but they're incentivized to move beyond that layer and provide more custom solutions that ... lock you into their services. To Google's credit, they are actually moving to a more open cloud environment. They started with Google App Engine, which definitely had lock-in with it's own custom API. But now they are pushing container services and kubernetes, which are really easy to move to other clouds or run on your own servers.
Joeri 9y ago
Google is not doing anything differently here from what they did before. They've been releasing papers instead of code since 2003 [1]. I think overall this has been a good thing. Had they open-sourced, we'd all be using their stuff, but there wouldn't have been the many competing open source big data projects. The fact that the rest of the world was forced to reimplement what google did created a beehive of open source technologies which is going places google never did. [1] http://blog.mikiobraun.de/2013/02/big-data-beyond-map-reduce-googles-papers.html http://blog.mikiobraun.de/2013/02/big-data-beyond-map-reduce...
rakoo 9y ago
> Try to figure out how something like Amazon's systems work and you'll run into a deafening wall of silence. It looks to me recent history contradicts this point. Amazon is responsible for the Dynamo paper, a mandatory thing to know if you're going to deal with open source NoSQL DBs. All the big players have released how they work internally, giving us Storm and Heron, Luigi, LevelDB, and so many others. Surely you have heard of Kafka ? Coming from a big one as well. I seriously can't understand where you get this from. The amount of internals that is shared is really interesting.
- a-robinson 9y ago
  The Dynamo paper is from 2007... How much has Amazon published about their internal workings since then?

elvinyung 9y ago

Interestingly, Amazon is also presenting a paper this year at SIGMOD, about Aurora: http://dl.acm.org/authorize?N37778 http://dl.acm.org/authorize?N37778 If my memory serves, it's the second paper they've ever published (the other one being the famous Dynamo paper[1]). [1] http://cloudgroup.neu.edu.cn/papers/cloud%20data%20storage/dynamo-sosp-2007.pdf http://cloudgroup.neu.edu.cn/papers/cloud%20data%20storage/d...

brandur 9y ago

> This is part of a worrying new trend. Increasingly you can't buy software anymore, only rent. More or less agreed, but you'd probably have to concede that the existence of papers like this one and some of Google's other Spanner publications are admirable; they're being more open about the system's design than they have to be. Of course Google knows that the system's secret sauce is not the concept itself, but the cost of its implementation, the infrastructural harness to support it, and the resources to reliably operate it. Even with a rough understanding on how Spanner works, it's still going to be difficult to ever migrate off it for practical reasons alone — who else is going to be able to build and run an alternative? I have huge respect for what the Spanner team is doing, but this is a reason that Citus [1] is also very interesting to me right now. You could conceivably start out with nothing but Postgres and migrate into a Citus cluster when (and if) you need to. If a point comes where you realize that you need out, you could either (1) see if you can scale back down to simple Postgres, (2) host your own cluster on your own infrastructure with the Citus source code, or (3) migrate onto your own Postgres sharding scheme à la Instagram. At no point do you lock yourself into custom GRPC APIs which are going to be ~impossible to get off of. GCP and AWS provide hugely useful foundational IaaS, but they're incentivized to move beyond that layer and provide more custom solutions that (1) provide better margins, and (2) lock you into their services. As people and companies building on top of these clouds, we should be looking for whatever opportunities we can to keep our stacks generic so that AWS <-> GCP <-> Azure migrations are possible, even if a last resort. [1] https://www.citusdata.com/product/cloud https://www.citusdata.com/product/cloud

slackingoff2017 9y ago
Does this feel like rent seeking to you? The internet is not young anymore. It feels like cloud hosting is little more than the giants allowing controlled competition as long as you rent their servers. The platforms are just extensions of the massively powerful systems they use internally. I wonder what would happen if the open source community built a viable alternative to the cloud IaaS. Like OpenStack but not a failure :). OpenFlow has shown promise and could form the core for an open IaaS. Network virtualization is the hardest part.
- speedplane 9y ago
  > I wonder what would happen if the open source community built a viable alternative to the cloud IaaS. ... Network virtualization is the hardest part. I would think that the hardware itself is the hardest part. Companies move to the cloud because it reduces their internal OPs team, and you can scale up and down hardware extremely easily.
  - slackingoff2017 9y ago
    I respectfully disagree. Every company I've been part of moving to the cloud already had plenty of hardware. It was ease of provisioning VM's, backups, deployment, and networking that really convinced them. VMWare comes close to offering the same thing on premesis but Oracle is too obsessed with wringing money from it to let it reach it's full potential.
speedplane 9y ago
> GCE and AWS provide hugely useful foundational IaaS, but they're incentivized to move beyond that layer and provide more custom solutions that ... lock you into their services. To Google's credit, they are actually moving to a more open cloud environment. They started with Google App Engine, which definitely had lock-in with it's own custom API. But now they are pushing container services and kubernetes, which are really easy to move to other clouds or run on your own servers.

Joeri 9y ago

Google is not doing anything differently here from what they did before. They've been releasing papers instead of code since 2003 [1]. I think overall this has been a good thing. Had they open-sourced, we'd all be using their stuff, but there wouldn't have been the many competing open source big data projects. The fact that the rest of the world was forced to reimplement what google did created a beehive of open source technologies which is going places google never did. [1] http://blog.mikiobraun.de/2013/02/big-data-beyond-map-reduce-googles-papers.html http://blog.mikiobraun.de/2013/02/big-data-beyond-map-reduce...

rakoo 9y ago

> Try to figure out how something like Amazon's systems work and you'll run into a deafening wall of silence. It looks to me recent history contradicts this point. Amazon is responsible for the Dynamo paper, a mandatory thing to know if you're going to deal with open source NoSQL DBs. All the big players have released how they work internally, giving us Storm and Heron, Luigi, LevelDB, and so many others. Surely you have heard of Kafka ? Coming from a big one as well. I seriously can't understand where you get this from. The amount of internals that is shared is really interesting.

a-robinson 9y ago
The Dynamo paper is from 2007... How much has Amazon published about their internal workings since then?