4 ms·

Why was hybrid not considered? With providers like megaport providing really inexpensive direct connect, this is almost a no brainer. This may seem harsh, but

by keepper 10y ago

Why was hybrid not considered? With providers like megaport providing really inexpensive direct connect, this is almost a no brainer. This may seem harsh, but relying on random commenters shows a huge flaw in how you guys went about this. Physical environments DO work well, but they do require experience to run them. I have used AWS for as long as its been around, but nothing beats physical environments for "known workloads", as long as you run them EXACTLY like you would run a cloud environment. This is why the hybrid approach is such a great thing. Run what you know well in physical and reap great savings, run what you dont know well cloud ( as well as take advantage of the analytics products), and reap the speed ;) - This means thinking of physical servers are individual units. - This means redundancy at every level. - This means architecting for failure ( servers and switches do gie ). - This means no shared storage for performance critical parts ( shared storage as below the OS level ). - This means objects stores/sharding/etc as your storage layer. - This means real engineering - And this means exactly the same whether you're physical or not. I've managed environments of 50 vm's and environments of 50000 physical servers. The methodology is always the same. and yes, this means you can save some monthly cost, and apply it to staff, that can do a lot more than just maintain this infrastructure. PS: for those that think that showing up to a datacenter is required, you're doing it wrong. Pretty much any datacenter has hot-hands service, and with the right redundancy, hardware replacement is something you can do at a slower pace. PPS: im sure someone will nitpick some of my points. The reality is, there is real money savings here. For example, Snapchat spends MORE in cloud infrastructure in 2016/2017 per year, than ALL OF GOOGLE did in 2012... think about that for a second... even netflix runs openconnect to push bits..

dasil003 10y ago
> This may seem harsh, but relying on random commenters shows a huge flaw in how you guys went about this. Your other points and experience are well taken, but this is definitely too harsh. Think about the nature of GitLab as an open company building dev tools and paying below-market salaries. They have a huge amount of developer mindshare relative to their internal head count. They are not very competitive for hiring people with your particular skillset. Given these facts, I think an open decision making process serves them better than trying to make decisions behind closed doors. It may come off a little ham-fisted, but in actuality I think it's far less ham-fisted than a huge number (maybe even majority?) of engineering decisions made every day in companies that just don't have the expertise to make the right decision and often don't even realize they are getting burned by what they don't know they don't know.
- scott_s 10y ago
  It's not just too harsh, it's not true. See the comments from those involved in the decision in this thread.
  - keepper 10y ago
    Why is it not true? Their main competitor, github, runs their own datacenter? Why is that?
    - donarb 10y ago
      What's not true is that Gitlab relied on random commenters to influence its decision.
- extrovert 10y ago
  > They are not very competitive for hiring people with your particular skillset. There is plenty of ops talent willing to work at a high-profile startup. I have experience building and running PB-scale Ceph clusters, private and public clouds, HA infrastructure covering everything from DB failover to BGP, and interviewed at Gitlab when they announced going on-prem. I asked for half the market rate (because I was more interested in solving the scaling problem than money), but did not get it. The HR person at the first (non-technical!) interview asked a bunch of programming related questions from a script which caught me completely off guard. One question went along the lines of "tell me about a difficult problem that you had a particularily elegant solution for". I think they've learned by now that ops problems usually can't be solved by a simple algorithm. In retrospect I'm glad I didn't get it, but they seem to be having issues with more than infrastructure. As a side note, if there are other startups wanting to move to own hardware but struggling to find the right people, I would love to know where to find you. Non-AWS ops jobs are far and few between these days.
  - skuhn 10y ago
    It has definitely become less common to see companies build much outside of cloud environments. And the talent has dried up to an extent as a result. Expertise and interest in building outside of cloudland is important to find, but there's also a challenge in finding the right operational mindset as well. It seems more common with more mature "startups" that have done the cost / benefit analysis and realized that the only way to achieve economies of scale for their business is to exit AWS. Dropbox, GitHub, Uber and so forth. Or businesses that simply must operate their own hardware, like CDNs. There are some smaller places that see value in building a low-cloud service from the beginning. One of those companies is where I work, e-mail in profile.
mrep 10y ago
> Snapchat spends MORE in cloud infrastructure in 2016/2017 per year, than ALL OF GOOGLE did in 2012 Not 2012 but in 2013, Google spent 7.3 Billion dollars on data centers [1]. Snap's has agreed to spend a minimum of 400 Million dollars per year on Google cloud according to their S1 [2]. Where did you get your figures? [1]: http://www.datacenterknowledge.com/archives/2014/02/03/google-spent-7-3-billion-data-centers-2013/ http://www.datacenterknowledge.com/archives/2014/02/03/googl... [2]: https://www.sec.gov/Archives/edgar/data/1564408/000119312517029199/d270216ds1.htm https://www.sec.gov/Archives/edgar/data/1564408/000119312517...

by keepper 10y ago

dasil003 10y ago

> This may seem harsh, but relying on random commenters shows a huge flaw in how you guys went about this. Your other points and experience are well taken, but this is definitely too harsh. Think about the nature of GitLab as an open company building dev tools and paying below-market salaries. They have a huge amount of developer mindshare relative to their internal head count. They are not very competitive for hiring people with your particular skillset. Given these facts, I think an open decision making process serves them better than trying to make decisions behind closed doors. It may come off a little ham-fisted, but in actuality I think it's far less ham-fisted than a huge number (maybe even majority?) of engineering decisions made every day in companies that just don't have the expertise to make the right decision and often don't even realize they are getting burned by what they don't know they don't know.

scott_s 10y ago
It's not just too harsh, it's not true. See the comments from those involved in the decision in this thread.
- keepper 10y ago
  Why is it not true? Their main competitor, github, runs their own datacenter? Why is that?
  - donarb 10y ago
    What's not true is that Gitlab relied on random commenters to influence its decision.
extrovert 10y ago
> They are not very competitive for hiring people with your particular skillset. There is plenty of ops talent willing to work at a high-profile startup. I have experience building and running PB-scale Ceph clusters, private and public clouds, HA infrastructure covering everything from DB failover to BGP, and interviewed at Gitlab when they announced going on-prem. I asked for half the market rate (because I was more interested in solving the scaling problem than money), but did not get it. The HR person at the first (non-technical!) interview asked a bunch of programming related questions from a script which caught me completely off guard. One question went along the lines of "tell me about a difficult problem that you had a particularily elegant solution for". I think they've learned by now that ops problems usually can't be solved by a simple algorithm. In retrospect I'm glad I didn't get it, but they seem to be having issues with more than infrastructure. As a side note, if there are other startups wanting to move to own hardware but struggling to find the right people, I would love to know where to find you. Non-AWS ops jobs are far and few between these days.
- skuhn 10y ago
  It has definitely become less common to see companies build much outside of cloud environments. And the talent has dried up to an extent as a result. Expertise and interest in building outside of cloudland is important to find, but there's also a challenge in finding the right operational mindset as well. It seems more common with more mature "startups" that have done the cost / benefit analysis and realized that the only way to achieve economies of scale for their business is to exit AWS. Dropbox, GitHub, Uber and so forth. Or businesses that simply must operate their own hardware, like CDNs. There are some smaller places that see value in building a low-cloud service from the beginning. One of those companies is where I work, e-mail in profile.

mrep 10y ago

> Snapchat spends MORE in cloud infrastructure in 2016/2017 per year, than ALL OF GOOGLE did in 2012 Not 2012 but in 2013, Google spent 7.3 Billion dollars on data centers [1]. Snap's has agreed to spend a minimum of 400 Million dollars per year on Google cloud according to their S1 [2]. Where did you get your figures? [1]: http://www.datacenterknowledge.com/archives/2014/02/03/google-spent-7-3-billion-data-centers-2013/ http://www.datacenterknowledge.com/archives/2014/02/03/googl... [2]: https://www.sec.gov/Archives/edgar/data/1564408/000119312517029199/d270216ds1.htm https://www.sec.gov/Archives/edgar/data/1564408/000119312517...