20 ms·

How We Knew It Was Time to Leave the Cloud

379 points by sytse 10y ago

codinghorror 10y ago
I don't know about this. We had disastrous experiences with Ceph and Gluster on bare metal. I think this says more about the immaturity (and difficulty) of distributed file systems than the cloud per se.
- BinaryIdiot 10y ago
  Distributed file systems are tough especially if you're putting it together yourself. I'd go for an already built solution every time unless I absolutely could not for whatever the reason.
- connorshea 10y ago
  Was your experience RE: Ceph/Gluster from Stack Overflow? I'd definitely be interested in hearing more about the specifics of that.
  - Rafert 10y ago
    I think he's talking about Discourse.
- pinewurst 10y ago
  I think it says more about the state of open source distributed file systems than the cloud per se. Ceph and Gluster are not the best examples of these, though Lustre is awful too. I'm paid to dig into these at depth and each is like some combination of religion and dumpster fire. Understand that Red Hat (and Intel in the Lustre case) wants support, training and professional services revenue. Outside of their paid domains, it's truly commerce with demons but inside it's not much better. BeeGFS is the only nominally open source one that I'd think about trusting my data too. And no, I don't work for them or am compensated in any way for recommendations.
- sytse 10y ago
  Thanks for posting. What went wrong with Ceph and when was this? We have the idea it improved a lot in the last year or so. But we'd love to learn from your experience.
kmf 10y ago
As always, the neat thing about GitLab is how open they are with their process. I enjoyed this read, and followed the trail down to a corresponding ticket, where the staff is discussing their actual plans for moving to bare metal. Very cool. https://gitlab.com/gitlab-com/infrastructure/issues/727 https://gitlab.com/gitlab-com/infrastructure/issues/727
- lucaspiller 10y ago
  They even list the hardware they've spec'ed out: https://gitlab.com/gitlab-com/infrastructure/issues/727#note_18193242 https://gitlab.com/gitlab-com/infrastructure/issues/727#note...
- connorshea 10y ago
  I'd recommend reading some of the closed issues in that project as well, lots of interesting postmortems on downtime, increased load, etc. :) https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&state=closed https://gitlab.com/gitlab-com/infrastructure/issues?scope=al...
- 20after4 10y ago
  If you like open processes like that then you might like following the work of Wikimedia's Technical Operations team. [1] You won't find any organization that's much more open than Wikimedia. Disclosure: I work for Wikimedia (on the release engineering team) [1] https://phabricator.wikimedia.org/tag/operations/ https://phabricator.wikimedia.org/tag/operations/
wandernotlost 10y ago
It sounds like they didn't design for the cloud and are now experiencing the consequences. The cloud has different tradeoffs and performance characteristics from a datacenter. If you plan for that, it's great. Your software will be antifragile as a result. If you assume the characteristics of a datacenter, you're likely to run into problems.
- jghn 10y ago
  Can you go more into the difference in the tradeoffs and how one should design differently?
  - wandernotlost 10y ago
    A blunt summary would be that everything is unreliable and disposable. You have to design for failure, because most things will almost certainly fail (or just slow down) at some point. I learned a lot when I was first moving to the cloud from Adrian Cockroft. He has a ton of material out there from Netflix's move to the cloud. I recommend googling around for his conference talks. (I haven't watched these, but they're probably relevant: http://perfcap.blogspot.com/2012/03/cloud-architecture-tutorial.html http://perfcap.blogspot.com/2012/03/cloud-architecture-tutor...)
- edejong 10y ago
  I'm trying to understand 'antifragile'. Are you trying to say: 'robust'? If not, what is the difference?
  - Guzba 10y ago
    I don't think antifragile works in the parent comment, robust would be better. As for a definition, antifragile things get stronger from abuse--like harming muslces during a workout so they grow back stronger. If they were just robust, it would be like machinery (no healing / strengthening).
    - wandernotlost 10y ago
strictfp 10y ago
Why do they need one gigantic distributed fs? Seems like a design miss to me.
- btgeekboy 10y ago
  Indeed. If there's one thing I've learned in >10 years of building large, multi-tenant systems, it's that you need the ability to partition as you grow. Partitioning eases growth, reduces blast radius, and limits complexity.
  - Rapzid 10y ago
    I believe GitHub is or was distributing repos instead of files. AWS code commit likely stores pack files in S3 or otherwise takes advantage of existing AWS systems.. Using a single Ceph FS is an odd choice IMO. You need to have or acquire a ton of expertise to run stuff like that, and it can be finicky. I'm not convinced you can't get this running well enough on AWS though.. So I'd be worried the move to bare metal would just cause the team more problems.
- radicalbyte 10y ago
  It might be, but you have to look at what's important for a product like GitLab. It's in a market where those who'll pay want to run their own special version of the system. So it's naturally partitioned. Architecting, or even spending mental cycles on day 1 on distribution isn't going to win you as much as focusing on making an awesome product. This move will probably buy them another year or two, which will give them enough time hopefully implement some form of partitioning.
- jondubois 10y ago
  Agreed, I think the wrong conclusion was drawn here. But I get where they're coming from; container orchestrators like Kubernetes are heavily promoting distributed file systems as being the 'cloud-native approach'. But maybe this issue is more relevant to 'CephFS' specifically than to all distributed file systems in general.
- elygre 10y ago
boulos 10y ago
I appreciate that you didn't sling mud at Azure, but re-reading the commit for the move to Azure [1] there were tell tale signs then that it might be bumpy for the storage layer. For what it's worth, hardware doesn't provide an IOPS/latency SLA either ;). In all seriousness, we (Google, all providers) struggle with deciding what we can strictly promise. Offering you a "guaranteed you can hit 50k IOPS" SLA isn't much comfort if we know that all it takes is a single ToR failure for that to be not true (providers could still offer it, have you ask for a refund if affected, etc. but your experience isn't changed). All that said, I would encourage you to reconsider. I know you're frustrated, but rolling your own infrastructure just means you have to build systems even better than the providers. On the plus side, when it's your fault, it's your fault (or the hardware vendor, or the colo facility). You've been through a lot already, but I'd suggest you'd be better off returning to AWS or coming to us (Google) [Note: Our PD offering historically allowed up to 10 TiB per disk and is now a full 64 TiB, I'm sorry if the docs were confusing]. Again, I'm not saying this to have you come to us or another cloud provider, but because I honestly believe this would be a huge time sink for GitLab. Instead of focusing on your great product, you'd have to play "Let's order more storage" (honestly managing Ceph has a similar annoyance). I'm sorry you had a bad experience with your provider, but it's not all the same. Feel free to reach out to me or others, if you want to chat further. Disclosure: I work on Google Cloud. [1] https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/1097/diffs https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/...
- Dwolb 10y ago
  I think what you're seeing in this post has product implications for Google Cloud. The OP is asking for guaranteed IOPS/latency SLA for which they're willing to pay HUGE money for by ROLLING THEIR OWN. Possibly think about high service pricing tiers for systems that require it. This is a standard operations problem that can co-exist within pooled service models.
  - boulos 10y ago
    See my note about SLAs and ToR failures. We probably could promise something for our Local SSD offering (tail latency < 1ms!), but high-performance, guaranteed networked storage is just tricky. As I said, rolling your own will not give you a guarantee, it will just give you the responsibility for failure. We don't offer the guarantee, because we don't want you to believe it can't fail.
blorgle 10y ago
Coming from OpenStack land where Ceph is used heavily, it's well known that you shouldn't run production Ceph on VMs and that CephFS (which is like an NFS endpoint for Ceph itself) has never been as robust as the underlying Rados Block Device stuff. They probably could have saved themselves a lot of pain by talking to some Ceph experts still working inside RedHat for architectural and other design decisions. I agree with other poster who asked why do they even need a gigantic distributed fs and how that seems like a design miss.
- x0x0 10y ago
  Also -- if you look at the infra updated from the linked article, they mention something about 3M updates/hour to a pg table ([1], slide 9) triggering continuous vacuums. This feels like using a db table as a queue which is not going to be fun at moderate to high loads. [1] https://about.gitlab.com/2016/09/26/infrastructure-update/ https://about.gitlab.com/2016/09/26/infrastructure-update/
  - sytse 10y ago
    For sure there is much we can improve to reduce DB updates, but we do use Redis for queues.
  - pcarranza 10y ago
    It's not updates, it's just querying, updates are not that bad, the main issue there was that the CI runner was keeping a lock in the database while going for the filesystem to get the commit, this generated a lot of contention. Still this is something we need to fix in our CI implementation because, as you say, databases are not good queueing systems.
- camkego 10y ago
  Bingo! Seasoned developers and architects with 15-20+ years of experience would very likely question using software stacks like CephFS with warnings on it's website about production use! You really want no exotic 3rd party stuff in your design, and plain-Jane components like ext3 and Ethernet switches. Choosing a newer exotic distributed filesystem may really come back to bite you in the future.
annerajb 10y ago
BTW HPE provides some storage solutions where you can scale it up to 8 petabyte (in a single rack afaik). I have a really small setup but i would personally look into a dl580 as VM host and having two for redundancy. And a dual path storage system in my case I used a 2u MSA2400 (not sure if that is the latest name) Since it could continue to scale up and provided dual path too. I don't have experience running CEPH so I am not sure what are the hardware requirements for CEPH. (Disclaimer I work at Hewlett Packard Enterprises with servers)
jtwaleson 10y ago
So Gitlab is in a bit of a strange position here. Sticking to a traditional filesystem interface (distributed under the hood) seems stupid at first. Surely there are better technical solutions. However, considering they make money out of private installs of gitlab, it makes sense to keep gitlab.com as an eat-your-own-dog-food-at-scale environment. Necessary for them to keep experience with large installs of gitlab. If one of their customers that run on-prem has performance issues they can't just say: gitlab.com uses a totally different architecture so you're on your own. They need gitlab.com to be as close as possible to the standard product. Pivotal does the same thing with Pivotal Web Services, their public Cloud Foundry solution. All of their money is made in Pivotal Cloud Foundry (private installs). From a business perspective, private installs are a way of distributed computing. Pretty clever, and good way of minimizing risk.
- jtwaleson 10y ago
  A risk is of course that your public environment is so much bigger than your biggest private install, that it doesn't make sense anymore to keep the same technology for both tiers. I think both PWS and gitlab.com suffer from this.
  - cryptarch 10y ago
    Does Gitlab support some kind of federation between instances? If so, they could (in theory) split Gitlab.com into a bunch shards which as a whole match the properties of N% of enterprise users. That'd be a pretty cool way to avoid the different-in-scale problem (although you might still run into novel problems as you're now the Gitlab instance with the most shards..).
    - woot01 10y ago
      this is what Salesforce does - they have publicly visible shards in the url (na11, na14, eu23, etc) and login.salesforce.com redirects you to your shard when it figures out who you are.
      - 10y ago
eliben 10y ago
To me it makes total sense for something like Gitlab to do their own HW, since it's really their core business. Sure, there's no point for say, target.com to use their own servers - computers is not really what they do, and cloud helps them keep expensive programmers and sysadmins to a minimum. But Gitlab is a whole different story.
mSparks 10y ago
Fwiw, this is pretty much exactly the situation we had from the very start, since the early days before "big data" was even a thing people talked about, and why I went "bare metal" from the beginning. At the time (about 2005 iirc) I was read/writing between 500GB and 2TB a day, with heavy calculations across two high performance desktops. Then as I came to scale up AWS was announced, but when I priced it, just what was running already on those desktops came to more for the first month than buying 6 heavy duty dual CPU Dell servers with tons of storage. So that's exactly how we started. The only "shock" I got was how much of a b!tch keeping all that cool is, and that took a year or two to get that "just right".
ChuckMcM 10y ago
There is a threshold of performance on the cloud and if you need more, you will have to pay a lot more, be punished with latencies, or leave the cloud. I've seen this a lot, and for a given workload I can tell when leaving the cloud will be the right choice. But the unspoken part is "can we change our application given the limitations we see in the cloud?" Probably pretty difficult in a DVCS but not impossible. Sadly, storage isn't a first class service in most clouds (it should be) and so you end up with machines doing storage inefficiently and that costs time, power, and complexity.
- boulos 10y ago
  > Sadly, storage isn't a first class service in most clouds What would constitute a "first-class service" for you? (Or what does "storage" mean to you) I think both of Persistent Disk for block and GCS for Object storage as fairly reasonable. I agree that the world of "Give me multiple writer networked block devices, preferably speaking NFS" is still pretty bad. Usual Disclosure: I work on Google Cloud (and for a long time, specifically Compute Engine).
  - ChuckMcM 10y ago
    Storage as a first class service means that storage is provided to the fabric directly. Just as routing is provided to the fabric directly by switches and routers so 'networking' is a first class service. In some networks time is provided by hardware dedicated to that so time is a "first class" service. "Storage" as a definition is an addressable persistent store of data. But that isn't as useful as one might hope for discussions, so I tend to think of it in terms of the collection of storage components and compute resources that enable those components to be accessed across the "network." So at Google a GFS cluster provides "storage" but if the compute is also running compute jobs, web server back ends, etc. It isn't the "only" task of the infrastructure and that is the definition of "second" or "not first class". Back in the day Urs would argue that storage takes so few compute cycles that it made no sense to dedicate an index to the serving up of blocks of disk. But that also constrained how much storage per index you could service. And that is why from a TCO perspective "storage as a service" is cheap when you need the CPUs for other things anyway, but it's very expensive when you just want storage. I wrote a white paper for Bart comparing GFS cost per gigabyte to the NetApp cost per gigabyte, and NetApp was way cheaper because it wasn't replicated 9 times on mission critical data, and one index (aka one filer head) could talk to 1,000 drives. That same sort of effect happens in cloud services where if you want 100TB of storage you end up having to pay for 10 high storage instances, even if your data movement needs could be addressed by a single server instance with say 32GB of memory. The startup DriveScale is targeting this imbalance for things like Hadoop clusters.
zzzcpan 10y ago
> when you get into the consistency, accessibility, and partition tolerance (CAP) of CephFS, it will just give away availability in exchange for consistency. Not their fault, POSIX API cannot fit into eventual consistency model to guarantee availability (i.e. latency). Moving to your own hardware doesn't actually solve the problem, just gives some room to scale vertically for some time. After that the only way to keep global consistency but minimize impact of unavailability is to shard everything, at least this way unavailability will be contained in its own shard without any impact on every other shard. It's better to avoid POSIX API in the first place.
- chetanahuja 10y ago
  This is the correct answer to run github, the SAAS product at scale. Probably not practical for github the on-prem installable software. They may have to bite the bullet and separate out the two systems at some point anyway though. edit: it's gitlab not github. The point still stands though.
- deleted 10y ago
  [deleted]
atsaloli 10y ago
Not to second guess the infrastructure architects at GitLab, but just to bring up a data point they might possibly not be aware of : Joyent virtual hosts have significatly better I/O performance profiles -- see https://www.joyent.com/blog/docker-bake-off-aws-vs-joyent https://www.joyent.com/blog/docker-bake-off-aws-vs-joyent for detail. Don't necessarily write off the cloud -- there are different clouds out there and if I was running something I/O intensive, I'd want to try it on Joyent. That said, nothing beats complete control, if you have the resources to handle that level of responsibility. =)
diziet 10y ago
I priced out the hardware that they specced out to be around $1.4 million: https://gitlab.com/gitlab-com/infrastructure/issues/727#note_18193242 https://gitlab.com/gitlab-com/infrastructure/issues/727#note... The R730xd are probably around $50k after dell discounts - depends a bit on what they ended up configuring with regards to support, exact network configuration, etc The R830s are about $50k as configured - 1.5gb ram is expensive as the R830 only has 48 DIMMs and they need the relatively expensive 32GB RDIMMs The R630s should be about 15k each: The switches say they are 48x 40G QSFP+ which are very expensive (I'd put them at 30k each from Dell) 50k * 20 + 50k * 4 + 15k * 10k + 2 * 30k 1m + 200k + 150k + 60k ~= $1.4m invested updated with better R830s pricing From my perspective consumer Samsung 850 EVO drives which can be under-provisioned to match the 1.8tb (and get better performance characteristics) would give Gitlab cheaper and more reliable storage in terms of IOPS/Latency when compared to 10k 1.8tb drives.
- connorshea 10y ago
  Unfortunately I don't believe we'll be able to comment on the accuracy of this since the quotes from companies are private information, but I do love math like this :D (Community Advocate at GitLab)
- redstripe 10y ago
  What kind of dell discounts does dell typically give larger customers? We get 0 (from HP) at my workplace and I've always wondered. I'm sure it depends on how much you buy. Does it vary by components? They seem to charge a lot for drives so I'm guessing those can be heavily discounted.
  - icelancer 10y ago
    I helped design a DC for a big data mining / science company and we got just under 20% off the higher margin packages. I felt very good about the number and Dell was great to work with on the purchase side.
  - late2part 10y ago
    Please stop and read this and show it to your boss. If you are really getting 0% discount you need to private message me and send me $10k so I can tell you this one weird trick. You simply ask for them. Just do that. Everyone gets discounts.
user5994461 10y ago
No information in this article. How much data? We don't even know if it's GB/TB/PB/EB? How many files/objects? How many read IOPS are needed? How many write IOPS? What's the current setup on AWS? What's the current cost? What are they hosting? Can it scale horizontally? How do they shard jobs/users? What's running on Postgre? What's running on Ceph? What's running on NFS? How much disk bandwidth is used? How much network bandwidth is used? How are we supposed to review their architecture if they don't explain anything... I bet that there is a valid narrative where PostgreSQL and NFS was their doom, but I'd need data to explain that ^^
- connorshea 10y ago
  Unfortunately I don't have deep knowledge of our infrastructure, so I can't answer all of these questions myself. That said, a decent chunk of this info can be found in the discussions on our Infrastructure issue tracker[1]. The last infrastructure update[2] includes some slide decks that contain more data (albeit it's now a little under 2 months old). Looking at our internal Grafana instance, it looks like we're using about 1.25 TiB combined on NFS and just under 16 TiB on Ceph. We're working on migrating the data currently hosted on Ceph back to NFS soon[3]. I'll get someone from the infrastructure team to respond with more info. [1]: https://gitlab.com/gitlab-com/infrastructure/issues https://gitlab.com/gitlab-com/infrastructure/issues [2]: https://about.gitlab.com/2016/09/26/infrastructure-update/ https://about.gitlab.com/2016/09/26/infrastructure-update/ [3]: https://gitlab.com/gitlab-com/infrastructure/issues/711 https://gitlab.com/gitlab-com/infrastructure/issues/711
  - flik 10y ago
    >>If one of the hosts delays writing to the journal, then the rest of the fleet is waiting for that operation alone, and the whole file system is blocked. When this happens, all of the hosts halt, and you have a locked file system; no one can read or write anything and that basically takes everything down. I have seen similar issues where a GC pause on one server, freeze the entire cluster. Is this one single monolithic file system? On the service side, can the code be asynchronous with request queues for each shard? This can help free up threads from getting blocked and serve requests for other shards.
random3 10y ago
if latency spikes affect the overall performance, it seems more that CephFS may have a design problem (global FS journal) rather than this being a cloud problem. However perhaps they shouldn't try to run Ceph in the first place. Azure has a rather powerful blob storage (e.g. block, pages and append-only blobs) that allows high performance applications. You could use that directly and it will likely be cheaper and work better than Ceph on bare metal. Like other commenters suggest, in order to take advantage of cloud infrastructure you need to design with those constraints in mind, rather than trying to shoehorn the familiar technologies. Bare metal can be better and cheaper, etc. but it requires even more skills and experience and a relatively large scale.
- cryptarch 10y ago
  Wouldn't that make Gitlab dependent on Azure-specific services? That's quite a risk to take when you're already not sure if you want to stay with MS hosting services.
setheron 10y ago
We just launched BMC at Oracle - bare metal compute. Sounds like a good solution to those who want an IaaS offering but on bare met.
- nixgeek 10y ago
  This actually works fairly well given it was just launched. Definitely not at feature parity with other providers yet. Only a limited selection of instance shapes available and most are "extremely large" (vs. what GitLab is running now). There exists public documentation over here including API references and it does not require login - https://docs.us-az-phoenix-1.oracleiaas.com https://docs.us-az-phoenix-1.oracleiaas.com Disclosure: Working on Oracle Data Cloud and have been partnering internally with BMC for our own workloads.
- elygre 10y ago
  The video on https://blogs.oracle.com/cloud/entry/oracle_bare_metal_cloud_services https://blogs.oracle.com/cloud/entry/oracle_bare_metal_cloud... is interesting. They rent the entire machine, with all cloud management software running outside the server.
brongondwana 10y ago
Having never really gone to the cloud, thankfully, FastMail can strongly recommend New York Internet if you're looking for a datacentre provider. They've been amazing. https://blog.fastmail.com/2014/12/10/security-availability/ https://blog.fastmail.com/2014/12/10/security-availability/ Actually, we don't appear to blog quite enough about how awesome NYI are. They're a major part of our good uptime. And some stuff about our hardware. I'd strongly recommend hot stuff on RAID1 SSDs and colder stuff on hard disks. The performance difference between rust and SSD is just massive. We're looking at PCI SSDs for our next hardware upgrades: http://www.intel.com.au/content/www/au/en/solid-state-drives/intel-ssd-dc-family-for-pcie.html http://www.intel.com.au/content/www/au/en/solid-state-drives... (there are two types of SSD on the market. Ones that lose your data, and SSDs from Intel) - we're currently running mostly DC3700 SATA/SAS SSDs. We customise the layout of our machines very much to match the heat patterns of our data: https://blog.fastmail.com/2015/12/06/getting-the-most-out-of-hardware/ https://blog.fastmail.com/2015/12/06/getting-the-most-out-of... https://blog.fastmail.com/2014/12/15/dec-15-putting-the-fast-in-fastmail-loading-your-mailbox-quickly/ https://blog.fastmail.com/2014/12/15/dec-15-putting-the-fast... https://blog.fastmail.com/2014/12/04/standalone-mail-servers/ https://blog.fastmail.com/2014/12/04/standalone-mail-servers...
user5994461 10y ago
I feel like I have to do it and ask the stupidest yet most important question in this thread... Why didn't you use S3?
- jakewins 10y ago
  S3 does not allow random access reads/writes to the files. "Editing" a file in S3 just means deleting the original blob and creating a new one. Meaning, you can't run git on it, you'd need to make a full copy of the repo each time someone makes a commit.
  - alblue 10y ago
    That isn't true; a git repository isn't just a single file. It is one (or many) pack and object files, and it's possible to have a repository by using each push (which may be multiple commits) to store a new pack file rather than deleting the old one. The only time you would need to delete/replace is during a GC operation which can be scheduled periodically to compact the representation. The JGit driver supports pushing to S3 in this way, and I have implemented a custom backend to hosting git repositories in an S3 like service (which in turn is based on how Google hosts git repositories at scale)
- boulos 10y ago
  I assume for three reasons: - small file performance (if they're not storing the git objects as packed) - POSIX compliance - Their on-prem installations for customers For posix compliance, you can use s3fuse/gcsfuse or put Avere in front, but you're either paying more for it or you're not getting the best performance. For small file performance though, no object storage system is particularly tuned for that (often the granularity is more like MBs, and at least with GCS you're talking ~100ms per write). Finally, they need to be testing and supporting customers who want to run the whole thing "at home" wherever that is. So they would still need to run Ceph! Disclosure: I work on Google Cloud.
  - jhartmann 10y ago
    I wouldn't say no distributed object storage system is particularly tuned for that. Facebook's haystack and systems patterned after it can work very well for tiny files.
mianos 10y ago
At least you know your own boxes will be still there in a few years too. AWS pretty sure, Azure, who knows, GCE, anyones guess. Sure they are paying people to post in hackernews today but further out such support is likely to go the way of any other Google service that has proven successful and scaled. Complete and utter silence. At least you can reach out to AWS. Ever had a bug in appengine or Android? You are on your own.
- nixgeek 10y ago
  I think this is a pessimistic view of Google Cloud given the revenues they're now making from it. A more interesting question would be about Rackspace and SoftLayer, the former of which has been taken private and the latter was acquired by IBM, both are in the "We will give you anything you want, this is how much it'll cost" side of the hosting business which is AFAICS most likely to be under pressure in the next 10 years as workloads get modernised.
  - mianos 10y ago
    It is my understanding they are making a fair profit in the Pixel phone and Android from ads too. Sure it's pessimistic but, after my experience, I still love to hate AWS and will choose it first. While the privatisation of Rackspace is a worry, damn their support is so good, albeit, they only offer a very limited number of services compared to GCE, AWS and Azure.
- vgt 10y ago
  Perhaps you can talk to my boss and get boulos and I paid for posting on hn! Reality is, we want to be engaged, and we are allowed. This is a luxury, not a burden in exchange for salary. We all have Very busy day jobs. And very often we learn from the community and pass the feedback on. We aren't in marketing, pr, devrel, or even sales. (Work at Google cloud)
dontbeapussy 10y ago
What I see here is an inability to deal with the ish. Pull yourself up by the balls dude.
dontbeapussy 10y ago
What I see here is an inability to deal with the ish. Pull yourself up by the balls dude.
holri 10y ago
There is no cloud, only other people computers.
raspasov 10y ago
How much total storage do you need? Something to consider: A few years ago I used Rackspace's OnMetal servers https://www.rackspace.com/en-us/cloud/servers/onmetal https://www.rackspace.com/en-us/cloud/servers/onmetal for a dedicated MySQL 128GB RAM server that would handle 100s of thousands of very active hardcore mobile game users. We were doing thousands of HTTP requests per seconds and 10s of thousands of queries (and a lot of those were writes) per second, all on one server. The DB server would not skip a beat, CPU/IO was always <20% and all of our queries would run in 1-5ms range. I'm not affiliated with Rackspace in any capacity, but my experience with them in the past has been top-notch, esp. when it comes to "dedicated-like" cloud hardware, which is what OnMetal is - your are 100% on one machine, no neighbors. Their prices can be high but the reliability is top-notch, and the description of the hardware is very accurate, much more detailed than AWS for example, and without "fluffy" cloud terms :). For example: Boot device: 2x 240 GB hot-swappable SSDs configured in a RAID 1 mirror Storage: 2x 1.6 TB PCIe storage devices (Seagate Nytro XP6302)
- sytse 10y ago
  We need about 70TB now and are planning for 256TB.
- jayofdoom 10y ago
  Thanks :). Glad to hear you liked the product.
steeve 10y ago
I may be wrong but it seems to me that immutable blob (git) are a perfect match for any blob storage such as s3, gcs or azure blob storage. I even think there are some libgit2 backends which do exactly this. It may then become a problem of latency per blob, in which case you could coalesce blobs into bigger ones as needed (histories don't change much) and optimize for that.
londons_explore 10y ago
The advice in this post is, Imo, misguided. In large systems design, you should always design for a large variation in individual systems performance. You should be able to meet customers expectations if any machine drops to 1% performance at anytime. Here they are blaming the cloud for the variation, but at big enough sizes they'll see the same on real hardware. Real hardware gets thermally throttled when heatsinks go bad, has io failures which cause dramatic performance drops, CPU's failing leaving only one core out of 32 operational, or ECC memory controllers that have to correct and reread every byte of memory. In a large enough system, at any time there will always be a system with a fault like this. Sure you only see it occasionally in your 200 node cluster, but in a 20k machine cluster it'll happen every day. You'll write code to detect the common cases and exclude the machines, but you'll never find all the cases. The conclusion is that instead you shouldn't try. Your application should handle performance variation, and to make sure it does, you would be advised to deliberately give it variable performance on all the nodes. Run at low CPU or io priority on a shared machine for example. In the example of a distributed filesystem, all data is stored in many places for redundancy. Overcome the variable performance by selecting the node to read from based on reported load. In a system with a "master" (which is a bad design pattern anyway IMO), instead have 5 masters and use a 3/5 vote. Now your system performance depends on the median performance of those 5.
dade_ 10y ago
Or move from a pure public cloud provider. I don't know what their spend is, but there are hybrid offers that provide managed premise and public cloud delivered on a consumption model that allow an organization to run a single administration and management interface, and can control what runs together on the premise gear.
gerbilly 10y ago
If only there was a way to run git in a decentralized manner. :-) Then users could host their own repositories themselves and manage their storage. This kind of setup would scale a lot better.
Steeeve 10y ago
> The problem with CephFS is that in order to work, it needs to have a really performant underlaying infrastructure because it needs to read and write a lot of things really fast. If one of the hosts delays writing to the journal, then the rest of the fleet is waiting for that operation alone, and the whole file system is blocked. This could easily have been titled "Why we couldn't use SephFS"
djb_hackernews 10y ago
Is provisioned iops not an option on azure? We use it on aws ebs and it's really helped us address some of the issues in this blog post.
Shorel 10y ago
The hardware part seems to be completely solved with blackblaze designs: https://www.backblaze.com/blog/open-source-data-storage-server/ https://www.backblaze.com/blog/open-source-data-storage-serv... Therefore the only question left is what to use as the software for this.
limeyx 10y ago
I think when people say "Add more machines to scale" they more often mean more worker-type machines all doing independent tasks or smaller pieces of a large Not continuing to link ever-increasing sets of machines to each other in a big glob ... thats much harder to make work in the cloud

379 points by sytse 10y ago

I don't know about this. We had disastrous experiences with Ceph and Gluster on bare metal. I think this says more about the immaturity (and difficulty) of distributed file systems than the cloud per se.

BinaryIdiot 10y ago
Distributed file systems are tough especially if you're putting it together yourself. I'd go for an already built solution every time unless I absolutely could not for whatever the reason.
connorshea 10y ago
Was your experience RE: Ceph/Gluster from Stack Overflow? I'd definitely be interested in hearing more about the specifics of that.
- Rafert 10y ago
  I think he's talking about Discourse.
pinewurst 10y ago
I think it says more about the state of open source distributed file systems than the cloud per se. Ceph and Gluster are not the best examples of these, though Lustre is awful too. I'm paid to dig into these at depth and each is like some combination of religion and dumpster fire. Understand that Red Hat (and Intel in the Lustre case) wants support, training and professional services revenue. Outside of their paid domains, it's truly commerce with demons but inside it's not much better. BeeGFS is the only nominally open source one that I'd think about trusting my data too. And no, I don't work for them or am compensated in any way for recommendations.
sytse 10y ago
Thanks for posting. What went wrong with Ceph and when was this? We have the idea it improved a lot in the last year or so. But we'd love to learn from your experience.

kmf 10y ago

As always, the neat thing about GitLab is how open they are with their process. I enjoyed this read, and followed the trail down to a corresponding ticket, where the staff is discussing their actual plans for moving to bare metal. Very cool. https://gitlab.com/gitlab-com/infrastructure/issues/727 https://gitlab.com/gitlab-com/infrastructure/issues/727

lucaspiller 10y ago
They even list the hardware they've spec'ed out: https://gitlab.com/gitlab-com/infrastructure/issues/727#note_18193242 https://gitlab.com/gitlab-com/infrastructure/issues/727#note...
connorshea 10y ago
I'd recommend reading some of the closed issues in that project as well, lots of interesting postmortems on downtime, increased load, etc. :) https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&state=closed https://gitlab.com/gitlab-com/infrastructure/issues?scope=al...
20after4 10y ago
If you like open processes like that then you might like following the work of Wikimedia's Technical Operations team. [1] You won't find any organization that's much more open than Wikimedia. Disclosure: I work for Wikimedia (on the release engineering team) [1] https://phabricator.wikimedia.org/tag/operations/ https://phabricator.wikimedia.org/tag/operations/

wandernotlost 10y ago

It sounds like they didn't design for the cloud and are now experiencing the consequences. The cloud has different tradeoffs and performance characteristics from a datacenter. If you plan for that, it's great. Your software will be antifragile as a result. If you assume the characteristics of a datacenter, you're likely to run into problems.

jghn 10y ago
Can you go more into the difference in the tradeoffs and how one should design differently?
- wandernotlost 10y ago
  A blunt summary would be that everything is unreliable and disposable. You have to design for failure, because most things will almost certainly fail (or just slow down) at some point. I learned a lot when I was first moving to the cloud from Adrian Cockroft. He has a ton of material out there from Netflix's move to the cloud. I recommend googling around for his conference talks. (I haven't watched these, but they're probably relevant: http://perfcap.blogspot.com/2012/03/cloud-architecture-tutorial.html http://perfcap.blogspot.com/2012/03/cloud-architecture-tutor...)
edejong 10y ago
I'm trying to understand 'antifragile'. Are you trying to say: 'robust'? If not, what is the difference?
- Guzba 10y ago
  I don't think antifragile works in the parent comment, robust would be better. As for a definition, antifragile things get stronger from abuse--like harming muslces during a workout so they grow back stronger. If they were just robust, it would be like machinery (no healing / strengthening).
  - wandernotlost 10y ago

strictfp 10y ago

Why do they need one gigantic distributed fs? Seems like a design miss to me.

btgeekboy 10y ago
Indeed. If there's one thing I've learned in >10 years of building large, multi-tenant systems, it's that you need the ability to partition as you grow. Partitioning eases growth, reduces blast radius, and limits complexity.
- Rapzid 10y ago
  I believe GitHub is or was distributing repos instead of files. AWS code commit likely stores pack files in S3 or otherwise takes advantage of existing AWS systems.. Using a single Ceph FS is an odd choice IMO. You need to have or acquire a ton of expertise to run stuff like that, and it can be finicky. I'm not convinced you can't get this running well enough on AWS though.. So I'd be worried the move to bare metal would just cause the team more problems.
radicalbyte 10y ago
It might be, but you have to look at what's important for a product like GitLab. It's in a market where those who'll pay want to run their own special version of the system. So it's naturally partitioned. Architecting, or even spending mental cycles on day 1 on distribution isn't going to win you as much as focusing on making an awesome product. This move will probably buy them another year or two, which will give them enough time hopefully implement some form of partitioning.
jondubois 10y ago
Agreed, I think the wrong conclusion was drawn here. But I get where they're coming from; container orchestrators like Kubernetes are heavily promoting distributed file systems as being the 'cloud-native approach'. But maybe this issue is more relevant to 'CephFS' specifically than to all distributed file systems in general.
elygre 10y ago

boulos 10y ago

I appreciate that you didn't sling mud at Azure, but re-reading the commit for the move to Azure [1] there were tell tale signs then that it might be bumpy for the storage layer. For what it's worth, hardware doesn't provide an IOPS/latency SLA either ;). In all seriousness, we (Google, all providers) struggle with deciding what we can strictly promise. Offering you a "guaranteed you can hit 50k IOPS" SLA isn't much comfort if we know that all it takes is a single ToR failure for that to be not true (providers could still offer it, have you ask for a refund if affected, etc. but your experience isn't changed). All that said, I would encourage you to reconsider. I know you're frustrated, but rolling your own infrastructure just means you have to build systems even better than the providers. On the plus side, when it's your fault, it's your fault (or the hardware vendor, or the colo facility). You've been through a lot already, but I'd suggest you'd be better off returning to AWS or coming to us (Google) [Note: Our PD offering historically allowed up to 10 TiB per disk and is now a full 64 TiB, I'm sorry if the docs were confusing]. Again, I'm not saying this to have you come to us or another cloud provider, but because I honestly believe this would be a huge time sink for GitLab. Instead of focusing on your great product, you'd have to play "Let's order more storage" (honestly managing Ceph has a similar annoyance). I'm sorry you had a bad experience with your provider, but it's not all the same. Feel free to reach out to me or others, if you want to chat further. Disclosure: I work on Google Cloud. [1] https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/1097/diffs https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/...

Dwolb 10y ago
I think what you're seeing in this post has product implications for Google Cloud. The OP is asking for guaranteed IOPS/latency SLA for which they're willing to pay HUGE money for by ROLLING THEIR OWN. Possibly think about high service pricing tiers for systems that require it. This is a standard operations problem that can co-exist within pooled service models.
- boulos 10y ago
  See my note about SLAs and ToR failures. We probably could promise something for our Local SSD offering (tail latency < 1ms!), but high-performance, guaranteed networked storage is just tricky. As I said, rolling your own will not give you a guarantee, it will just give you the responsibility for failure. We don't offer the guarantee, because we don't want you to believe it can't fail.

blorgle 10y ago

Coming from OpenStack land where Ceph is used heavily, it's well known that you shouldn't run production Ceph on VMs and that CephFS (which is like an NFS endpoint for Ceph itself) has never been as robust as the underlying Rados Block Device stuff. They probably could have saved themselves a lot of pain by talking to some Ceph experts still working inside RedHat for architectural and other design decisions. I agree with other poster who asked why do they even need a gigantic distributed fs and how that seems like a design miss.

x0x0 10y ago
Also -- if you look at the infra updated from the linked article, they mention something about 3M updates/hour to a pg table ([1], slide 9) triggering continuous vacuums. This feels like using a db table as a queue which is not going to be fun at moderate to high loads. [1] https://about.gitlab.com/2016/09/26/infrastructure-update/ https://about.gitlab.com/2016/09/26/infrastructure-update/
- sytse 10y ago
  For sure there is much we can improve to reduce DB updates, but we do use Redis for queues.
- pcarranza 10y ago
  It's not updates, it's just querying, updates are not that bad, the main issue there was that the CI runner was keeping a lock in the database while going for the filesystem to get the commit, this generated a lot of contention. Still this is something we need to fix in our CI implementation because, as you say, databases are not good queueing systems.
camkego 10y ago
Bingo! Seasoned developers and architects with 15-20+ years of experience would very likely question using software stacks like CephFS with warnings on it's website about production use! You really want no exotic 3rd party stuff in your design, and plain-Jane components like ext3 and Ethernet switches. Choosing a newer exotic distributed filesystem may really come back to bite you in the future.

annerajb 10y ago

BTW HPE provides some storage solutions where you can scale it up to 8 petabyte (in a single rack afaik). I have a really small setup but i would personally look into a dl580 as VM host and having two for redundancy. And a dual path storage system in my case I used a 2u MSA2400 (not sure if that is the latest name) Since it could continue to scale up and provided dual path too. I don't have experience running CEPH so I am not sure what are the hardware requirements for CEPH. (Disclaimer I work at Hewlett Packard Enterprises with servers)

jtwaleson 10y ago

So Gitlab is in a bit of a strange position here. Sticking to a traditional filesystem interface (distributed under the hood) seems stupid at first. Surely there are better technical solutions. However, considering they make money out of private installs of gitlab, it makes sense to keep gitlab.com as an eat-your-own-dog-food-at-scale environment. Necessary for them to keep experience with large installs of gitlab. If one of their customers that run on-prem has performance issues they can't just say: gitlab.com uses a totally different architecture so you're on your own. They need gitlab.com to be as close as possible to the standard product. Pivotal does the same thing with Pivotal Web Services, their public Cloud Foundry solution. All of their money is made in Pivotal Cloud Foundry (private installs). From a business perspective, private installs are a way of distributed computing. Pretty clever, and good way of minimizing risk.

jtwaleson 10y ago
A risk is of course that your public environment is so much bigger than your biggest private install, that it doesn't make sense anymore to keep the same technology for both tiers. I think both PWS and gitlab.com suffer from this.
- cryptarch 10y ago
  Does Gitlab support some kind of federation between instances? If so, they could (in theory) split Gitlab.com into a bunch shards which as a whole match the properties of N% of enterprise users. That'd be a pretty cool way to avoid the different-in-scale problem (although you might still run into novel problems as you're now the Gitlab instance with the most shards..).
  - woot01 10y ago
    this is what Salesforce does - they have publicly visible shards in the url (na11, na14, eu23, etc) and login.salesforce.com redirects you to your shard when it figures out who you are.
    - 10y ago

eliben 10y ago

To me it makes total sense for something like Gitlab to do their own HW, since it's really their core business. Sure, there's no point for say, target.com to use their own servers - computers is not really what they do, and cloud helps them keep expensive programmers and sysadmins to a minimum. But Gitlab is a whole different story.

mSparks 10y ago

Fwiw, this is pretty much exactly the situation we had from the very start, since the early days before "big data" was even a thing people talked about, and why I went "bare metal" from the beginning. At the time (about 2005 iirc) I was read/writing between 500GB and 2TB a day, with heavy calculations across two high performance desktops. Then as I came to scale up AWS was announced, but when I priced it, just what was running already on those desktops came to more for the first month than buying 6 heavy duty dual CPU Dell servers with tons of storage. So that's exactly how we started. The only "shock" I got was how much of a b!tch keeping all that cool is, and that took a year or two to get that "just right".

ChuckMcM 10y ago

There is a threshold of performance on the cloud and if you need more, you will have to pay a lot more, be punished with latencies, or leave the cloud. I've seen this a lot, and for a given workload I can tell when leaving the cloud will be the right choice. But the unspoken part is "can we change our application given the limitations we see in the cloud?" Probably pretty difficult in a DVCS but not impossible. Sadly, storage isn't a first class service in most clouds (it should be) and so you end up with machines doing storage inefficiently and that costs time, power, and complexity.

boulos 10y ago
> Sadly, storage isn't a first class service in most clouds What would constitute a "first-class service" for you? (Or what does "storage" mean to you) I think both of Persistent Disk for block and GCS for Object storage as fairly reasonable. I agree that the world of "Give me multiple writer networked block devices, preferably speaking NFS" is still pretty bad. Usual Disclosure: I work on Google Cloud (and for a long time, specifically Compute Engine).
- ChuckMcM 10y ago
  Storage as a first class service means that storage is provided to the fabric directly. Just as routing is provided to the fabric directly by switches and routers so 'networking' is a first class service. In some networks time is provided by hardware dedicated to that so time is a "first class" service. "Storage" as a definition is an addressable persistent store of data. But that isn't as useful as one might hope for discussions, so I tend to think of it in terms of the collection of storage components and compute resources that enable those components to be accessed across the "network." So at Google a GFS cluster provides "storage" but if the compute is also running compute jobs, web server back ends, etc. It isn't the "only" task of the infrastructure and that is the definition of "second" or "not first class". Back in the day Urs would argue that storage takes so few compute cycles that it made no sense to dedicate an index to the serving up of blocks of disk. But that also constrained how much storage per index you could service. And that is why from a TCO perspective "storage as a service" is cheap when you need the CPUs for other things anyway, but it's very expensive when you just want storage. I wrote a white paper for Bart comparing GFS cost per gigabyte to the NetApp cost per gigabyte, and NetApp was way cheaper because it wasn't replicated 9 times on mission critical data, and one index (aka one filer head) could talk to 1,000 drives. That same sort of effect happens in cloud services where if you want 100TB of storage you end up having to pay for 10 high storage instances, even if your data movement needs could be addressed by a single server instance with say 32GB of memory. The startup DriveScale is targeting this imbalance for things like Hadoop clusters.

zzzcpan 10y ago

> when you get into the consistency, accessibility, and partition tolerance (CAP) of CephFS, it will just give away availability in exchange for consistency. Not their fault, POSIX API cannot fit into eventual consistency model to guarantee availability (i.e. latency). Moving to your own hardware doesn't actually solve the problem, just gives some room to scale vertically for some time. After that the only way to keep global consistency but minimize impact of unavailability is to shard everything, at least this way unavailability will be contained in its own shard without any impact on every other shard. It's better to avoid POSIX API in the first place.

chetanahuja 10y ago
This is the correct answer to run github, the SAAS product at scale. Probably not practical for github the on-prem installable software. They may have to bite the bullet and separate out the two systems at some point anyway though. edit: it's gitlab not github. The point still stands though.
deleted 10y ago
[deleted]

atsaloli 10y ago

Not to second guess the infrastructure architects at GitLab, but just to bring up a data point they might possibly not be aware of : Joyent virtual hosts have significatly better I/O performance profiles -- see https://www.joyent.com/blog/docker-bake-off-aws-vs-joyent https://www.joyent.com/blog/docker-bake-off-aws-vs-joyent for detail. Don't necessarily write off the cloud -- there are different clouds out there and if I was running something I/O intensive, I'd want to try it on Joyent. That said, nothing beats complete control, if you have the resources to handle that level of responsibility. =)

diziet 10y ago

I priced out the hardware that they specced out to be around $1.4 million: https://gitlab.com/gitlab-com/infrastructure/issues/727#note_18193242 https://gitlab.com/gitlab-com/infrastructure/issues/727#note... The R730xd are probably around $50k after dell discounts - depends a bit on what they ended up configuring with regards to support, exact network configuration, etc The R830s are about $50k as configured - 1.5gb ram is expensive as the R830 only has 48 DIMMs and they need the relatively expensive 32GB RDIMMs The R630s should be about 15k each: The switches say they are 48x 40G QSFP+ which are very expensive (I'd put them at 30k each from Dell) 50k * 20 + 50k * 4 + 15k * 10k + 2 * 30k 1m + 200k + 150k + 60k ~= $1.4m invested updated with better R830s pricing From my perspective consumer Samsung 850 EVO drives which can be under-provisioned to match the 1.8tb (and get better performance characteristics) would give Gitlab cheaper and more reliable storage in terms of IOPS/Latency when compared to 10k 1.8tb drives.

connorshea 10y ago
Unfortunately I don't believe we'll be able to comment on the accuracy of this since the quotes from companies are private information, but I do love math like this :D (Community Advocate at GitLab)
redstripe 10y ago
What kind of dell discounts does dell typically give larger customers? We get 0 (from HP) at my workplace and I've always wondered. I'm sure it depends on how much you buy. Does it vary by components? They seem to charge a lot for drives so I'm guessing those can be heavily discounted.
- icelancer 10y ago
  I helped design a DC for a big data mining / science company and we got just under 20% off the higher margin packages. I felt very good about the number and Dell was great to work with on the purchase side.
- late2part 10y ago
  Please stop and read this and show it to your boss. If you are really getting 0% discount you need to private message me and send me $10k so I can tell you this one weird trick. You simply ask for them. Just do that. Everyone gets discounts.

user5994461 10y ago

No information in this article. How much data? We don't even know if it's GB/TB/PB/EB? How many files/objects? How many read IOPS are needed? How many write IOPS? What's the current setup on AWS? What's the current cost? What are they hosting? Can it scale horizontally? How do they shard jobs/users? What's running on Postgre? What's running on Ceph? What's running on NFS? How much disk bandwidth is used? How much network bandwidth is used? How are we supposed to review their architecture if they don't explain anything... I bet that there is a valid narrative where PostgreSQL and NFS was their doom, but I'd need data to explain that ^^

connorshea 10y ago
Unfortunately I don't have deep knowledge of our infrastructure, so I can't answer all of these questions myself. That said, a decent chunk of this info can be found in the discussions on our Infrastructure issue tracker[1]. The last infrastructure update[2] includes some slide decks that contain more data (albeit it's now a little under 2 months old). Looking at our internal Grafana instance, it looks like we're using about 1.25 TiB combined on NFS and just under 16 TiB on Ceph. We're working on migrating the data currently hosted on Ceph back to NFS soon[3]. I'll get someone from the infrastructure team to respond with more info. [1]: https://gitlab.com/gitlab-com/infrastructure/issues https://gitlab.com/gitlab-com/infrastructure/issues [2]: https://about.gitlab.com/2016/09/26/infrastructure-update/ https://about.gitlab.com/2016/09/26/infrastructure-update/ [3]: https://gitlab.com/gitlab-com/infrastructure/issues/711 https://gitlab.com/gitlab-com/infrastructure/issues/711
- flik 10y ago
  >>If one of the hosts delays writing to the journal, then the rest of the fleet is waiting for that operation alone, and the whole file system is blocked. When this happens, all of the hosts halt, and you have a locked file system; no one can read or write anything and that basically takes everything down. I have seen similar issues where a GC pause on one server, freeze the entire cluster. Is this one single monolithic file system? On the service side, can the code be asynchronous with request queues for each shard? This can help free up threads from getting blocked and serve requests for other shards.

random3 10y ago

if latency spikes affect the overall performance, it seems more that CephFS may have a design problem (global FS journal) rather than this being a cloud problem. However perhaps they shouldn't try to run Ceph in the first place. Azure has a rather powerful blob storage (e.g. block, pages and append-only blobs) that allows high performance applications. You could use that directly and it will likely be cheaper and work better than Ceph on bare metal. Like other commenters suggest, in order to take advantage of cloud infrastructure you need to design with those constraints in mind, rather than trying to shoehorn the familiar technologies. Bare metal can be better and cheaper, etc. but it requires even more skills and experience and a relatively large scale.

cryptarch 10y ago
Wouldn't that make Gitlab dependent on Azure-specific services? That's quite a risk to take when you're already not sure if you want to stay with MS hosting services.

setheron 10y ago

We just launched BMC at Oracle - bare metal compute. Sounds like a good solution to those who want an IaaS offering but on bare met.

nixgeek 10y ago
This actually works fairly well given it was just launched. Definitely not at feature parity with other providers yet. Only a limited selection of instance shapes available and most are "extremely large" (vs. what GitLab is running now). There exists public documentation over here including API references and it does not require login - https://docs.us-az-phoenix-1.oracleiaas.com https://docs.us-az-phoenix-1.oracleiaas.com Disclosure: Working on Oracle Data Cloud and have been partnering internally with BMC for our own workloads.
elygre 10y ago
The video on https://blogs.oracle.com/cloud/entry/oracle_bare_metal_cloud_services https://blogs.oracle.com/cloud/entry/oracle_bare_metal_cloud... is interesting. They rent the entire machine, with all cloud management software running outside the server.

brongondwana 10y ago

Having never really gone to the cloud, thankfully, FastMail can strongly recommend New York Internet if you're looking for a datacentre provider. They've been amazing. https://blog.fastmail.com/2014/12/10/security-availability/ https://blog.fastmail.com/2014/12/10/security-availability/ Actually, we don't appear to blog quite enough about how awesome NYI are. They're a major part of our good uptime. And some stuff about our hardware. I'd strongly recommend hot stuff on RAID1 SSDs and colder stuff on hard disks. The performance difference between rust and SSD is just massive. We're looking at PCI SSDs for our next hardware upgrades: http://www.intel.com.au/content/www/au/en/solid-state-drives/intel-ssd-dc-family-for-pcie.html http://www.intel.com.au/content/www/au/en/solid-state-drives... (there are two types of SSD on the market. Ones that lose your data, and SSDs from Intel) - we're currently running mostly DC3700 SATA/SAS SSDs. We customise the layout of our machines very much to match the heat patterns of our data: https://blog.fastmail.com/2015/12/06/getting-the-most-out-of-hardware/ https://blog.fastmail.com/2015/12/06/getting-the-most-out-of... https://blog.fastmail.com/2014/12/15/dec-15-putting-the-fast-in-fastmail-loading-your-mailbox-quickly/ https://blog.fastmail.com/2014/12/15/dec-15-putting-the-fast... https://blog.fastmail.com/2014/12/04/standalone-mail-servers/ https://blog.fastmail.com/2014/12/04/standalone-mail-servers...

user5994461 10y ago

I feel like I have to do it and ask the stupidest yet most important question in this thread... Why didn't you use S3?

jakewins 10y ago
S3 does not allow random access reads/writes to the files. "Editing" a file in S3 just means deleting the original blob and creating a new one. Meaning, you can't run git on it, you'd need to make a full copy of the repo each time someone makes a commit.
- alblue 10y ago
  That isn't true; a git repository isn't just a single file. It is one (or many) pack and object files, and it's possible to have a repository by using each push (which may be multiple commits) to store a new pack file rather than deleting the old one. The only time you would need to delete/replace is during a GC operation which can be scheduled periodically to compact the representation. The JGit driver supports pushing to S3 in this way, and I have implemented a custom backend to hosting git repositories in an S3 like service (which in turn is based on how Google hosts git repositories at scale)
boulos 10y ago
I assume for three reasons: - small file performance (if they're not storing the git objects as packed) - POSIX compliance - Their on-prem installations for customers For posix compliance, you can use s3fuse/gcsfuse or put Avere in front, but you're either paying more for it or you're not getting the best performance. For small file performance though, no object storage system is particularly tuned for that (often the granularity is more like MBs, and at least with GCS you're talking ~100ms per write). Finally, they need to be testing and supporting customers who want to run the whole thing "at home" wherever that is. So they would still need to run Ceph! Disclosure: I work on Google Cloud.
- jhartmann 10y ago
  I wouldn't say no distributed object storage system is particularly tuned for that. Facebook's haystack and systems patterned after it can work very well for tiny files.

mianos 10y ago

At least you know your own boxes will be still there in a few years too. AWS pretty sure, Azure, who knows, GCE, anyones guess. Sure they are paying people to post in hackernews today but further out such support is likely to go the way of any other Google service that has proven successful and scaled. Complete and utter silence. At least you can reach out to AWS. Ever had a bug in appengine or Android? You are on your own.

nixgeek 10y ago
I think this is a pessimistic view of Google Cloud given the revenues they're now making from it. A more interesting question would be about Rackspace and SoftLayer, the former of which has been taken private and the latter was acquired by IBM, both are in the "We will give you anything you want, this is how much it'll cost" side of the hosting business which is AFAICS most likely to be under pressure in the next 10 years as workloads get modernised.
- mianos 10y ago
  It is my understanding they are making a fair profit in the Pixel phone and Android from ads too. Sure it's pessimistic but, after my experience, I still love to hate AWS and will choose it first. While the privatisation of Rackspace is a worry, damn their support is so good, albeit, they only offer a very limited number of services compared to GCE, AWS and Azure.
vgt 10y ago
Perhaps you can talk to my boss and get boulos and I paid for posting on hn! Reality is, we want to be engaged, and we are allowed. This is a luxury, not a burden in exchange for salary. We all have Very busy day jobs. And very often we learn from the community and pass the feedback on. We aren't in marketing, pr, devrel, or even sales. (Work at Google cloud)

dontbeapussy 10y ago

What I see here is an inability to deal with the ish. Pull yourself up by the balls dude.

dontbeapussy 10y ago

What I see here is an inability to deal with the ish. Pull yourself up by the balls dude.

holri 10y ago

There is no cloud, only other people computers.

raspasov 10y ago

How much total storage do you need? Something to consider: A few years ago I used Rackspace's OnMetal servers https://www.rackspace.com/en-us/cloud/servers/onmetal https://www.rackspace.com/en-us/cloud/servers/onmetal for a dedicated MySQL 128GB RAM server that would handle 100s of thousands of very active hardcore mobile game users. We were doing thousands of HTTP requests per seconds and 10s of thousands of queries (and a lot of those were writes) per second, all on one server. The DB server would not skip a beat, CPU/IO was always <20% and all of our queries would run in 1-5ms range. I'm not affiliated with Rackspace in any capacity, but my experience with them in the past has been top-notch, esp. when it comes to "dedicated-like" cloud hardware, which is what OnMetal is - your are 100% on one machine, no neighbors. Their prices can be high but the reliability is top-notch, and the description of the hardware is very accurate, much more detailed than AWS for example, and without "fluffy" cloud terms :). For example: Boot device: 2x 240 GB hot-swappable SSDs configured in a RAID 1 mirror Storage: 2x 1.6 TB PCIe storage devices (Seagate Nytro XP6302)

sytse 10y ago
We need about 70TB now and are planning for 256TB.
jayofdoom 10y ago
Thanks :). Glad to hear you liked the product.

steeve 10y ago

I may be wrong but it seems to me that immutable blob (git) are a perfect match for any blob storage such as s3, gcs or azure blob storage. I even think there are some libgit2 backends which do exactly this. It may then become a problem of latency per blob, in which case you could coalesce blobs into bigger ones as needed (histories don't change much) and optimize for that.

londons_explore 10y ago

The advice in this post is, Imo, misguided. In large systems design, you should always design for a large variation in individual systems performance. You should be able to meet customers expectations if any machine drops to 1% performance at anytime. Here they are blaming the cloud for the variation, but at big enough sizes they'll see the same on real hardware. Real hardware gets thermally throttled when heatsinks go bad, has io failures which cause dramatic performance drops, CPU's failing leaving only one core out of 32 operational, or ECC memory controllers that have to correct and reread every byte of memory. In a large enough system, at any time there will always be a system with a fault like this. Sure you only see it occasionally in your 200 node cluster, but in a 20k machine cluster it'll happen every day. You'll write code to detect the common cases and exclude the machines, but you'll never find all the cases. The conclusion is that instead you shouldn't try. Your application should handle performance variation, and to make sure it does, you would be advised to deliberately give it variable performance on all the nodes. Run at low CPU or io priority on a shared machine for example. In the example of a distributed filesystem, all data is stored in many places for redundancy. Overcome the variable performance by selecting the node to read from based on reported load. In a system with a "master" (which is a bad design pattern anyway IMO), instead have 5 masters and use a 3/5 vote. Now your system performance depends on the median performance of those 5.

dade_ 10y ago

Or move from a pure public cloud provider. I don't know what their spend is, but there are hybrid offers that provide managed premise and public cloud delivered on a consumption model that allow an organization to run a single administration and management interface, and can control what runs together on the premise gear.

gerbilly 10y ago

If only there was a way to run git in a decentralized manner. :-) Then users could host their own repositories themselves and manage their storage. This kind of setup would scale a lot better.

Steeeve 10y ago

> The problem with CephFS is that in order to work, it needs to have a really performant underlaying infrastructure because it needs to read and write a lot of things really fast. If one of the hosts delays writing to the journal, then the rest of the fleet is waiting for that operation alone, and the whole file system is blocked. This could easily have been titled "Why we couldn't use SephFS"

djb_hackernews 10y ago

Is provisioned iops not an option on azure? We use it on aws ebs and it's really helped us address some of the issues in this blog post.

Shorel 10y ago

The hardware part seems to be completely solved with blackblaze designs: https://www.backblaze.com/blog/open-source-data-storage-server/ https://www.backblaze.com/blog/open-source-data-storage-serv... Therefore the only question left is what to use as the software for this.

limeyx 10y ago

I think when people say "Add more machines to scale" they more often mean more worker-type machines all doing independent tasks or smaller pieces of a large Not continuing to link ever-increasing sets of machines to each other in a big glob ... thats much harder to make work in the cloud