6 ms·

It sounds like they didn't design for the cloud and are now experiencing the consequences. The cloud has different tradeoffs and performance characteristics fro

It sounds like they didn't design for the cloud and are now experiencing the consequences. The cloud has different tradeoffs and performance characteristics from a datacenter. If you plan for that, it's great. Your software will be antifragile as a result. If you assume the characteristics of a datacenter, you're likely to run into problems.

jghn 10y ago
Can you go more into the difference in the tradeoffs and how one should design differently?
- wandernotlost 10y ago
  A blunt summary would be that everything is unreliable and disposable. You have to design for failure, because most things will almost certainly fail (or just slow down) at some point. I learned a lot when I was first moving to the cloud from Adrian Cockroft. He has a ton of material out there from Netflix's move to the cloud. I recommend googling around for his conference talks. (I haven't watched these, but they're probably relevant: http://perfcap.blogspot.com/2012/03/cloud-architecture-tutorial.html http://perfcap.blogspot.com/2012/03/cloud-architecture-tutor...)
edejong 10y ago
I'm trying to understand 'antifragile'. Are you trying to say: 'robust'? If not, what is the difference?
- Guzba 10y ago
  I don't think antifragile works in the parent comment, robust would be better. As for a definition, antifragile things get stronger from abuse--like harming muslces during a workout so they grow back stronger. If they were just robust, it would be like machinery (no healing / strengthening).
  - wandernotlost 10y ago
    I suppose I had Netflix in mind...as part of their move to the cloud, they developed chaos monkeys to actively attack their own software to make sure it is resilient to the failures that will inevitably, but less frequently, happen as a result of operating in the cloud.
    - solipsism 10y ago
      Stop using the term, you don't understand what it means. Chaos monkey is a form of testing. Testing is not antifragility.
      - wandernotlost 10y ago
        Chaos monkey attacks their production environment. They must make their software stronger/more resilient/less fragile in response. It would be more helpful if you clarified how you think that's different from antifragility.
        solipsism 10y ago
        Antifragility describes a system that becomes more resilient automatically, because it is built such that it must, in response to attack or damage. Such a system doesn't require management. It doesn't require people to actively test, to think about how to make the system better, to implement the fixes.
        wandernotlost 10y ago
        I think that's an overly narrow definition of the term antifragility and interpretation of the system in this case. While I can imagine software that is in itself antifragile, I think it's entirely reasonable to include the totality of the people and process that make up an operational system, in which case even your narrow definition applies here.
        10y ago
- nicky0 10y ago
  There's a whole book[0] about it. But to summarise (poorly): 'robust' - resilient against shocks, volatility and stressors 'antifragile' - thrives on shocks, volatility and stressors (ie. gets better in response) Antifragile is a step beyond robust. Examples of antifragility are evolutionary systems and true free market economies (as opposed to our too-big-to fail version of propped-up, overly interconnected capitalism). 0: https://en.wikipedia.org/wiki/Antifragile https://en.wikipedia.org/wiki/Antifragile
  - smallnamespace 10y ago
    Isn't that bad because now you're depending on having stressors and shocks to have the best performance? If you can't control the amount of stressors and shocks, you want a system that is neither antifragile or fragile, but strictly indifferent to the level of shocks.
    - nicky0 10y ago
      Good questions. It is a very interesting subject. If you haven't read the book, I recommend it. I think you will find it interesting if you read it with an open mind.
  - dsr_ 10y ago
    Can you provide any examples of working, real-world antifragile systems which were designed and built by humans and accomplish their purposes? Preferably in terms of software and hardware? So far you've named "evolutionary systems", which are quite fragile in the real world, and an imaginary thing called a "true free market economy".
    - wandernotlost 10y ago
      Netflix with their chaos monkeys is a great and relevant example.
      - hueving 10y ago
        But their system doesn't thrive on chaos monkey. It's just resilient to it.
Rapzid 10y ago
This got me curious again about the pluggable storage backends in Git(I assume AWS code commit is using something like this). I've looked at Azures blob storage API in the past and found it incredibly flexible.. Here is an article from a few years ago: http://blog.deveo.com/your-git-repository-in-a-database-pluggable-backends-in-libgit2/ http://blog.deveo.com/your-git-repository-in-a-database-plug... In any case, GitLab is amazing and I can see how it's tempting to believe that GitLab the omnibus package is the core product. However, HOSTED GitLab's core product is GitLab as a SERVICE. That might require designs tailored a bit more for the cloud than simply operating a yoooge fs and calling it a day.
- solatic 10y ago
  Gitlab is limited by the fact that they need their hosted product to run the same code as their on-prem product in order to avoid forking the codebase. If on-prem customers can't get AWS/GCE/Azure SuperFastCloudStorage™, then it can't be part of their codebase.
  - sytse 10y ago
    Exactly, we want our users to be able to scale with us using open source technologies. Some are at 20k+ users so they are close to needing something like Ceph themselves.
    - Rapzid 10y ago
      From looking at the comments here, thinking about what I would do/want, and what others have done to scale git GitLab is in the minority of wanting to solve this issue with a Ceph cluster. > Some are at 20k+ users so they are close to needing something like Ceph Or they will scale it themselves ala Alibaba: http://www.slideshare.net/MinqiPan/how-we-scaled-git-lab-for-a-30k-employee-company http://www.slideshare.net/MinqiPan/how-we-scaled-git-lab-for... . They appear to have written a libgit2 backend for their object store(among other things). I don't see a good reason why solutions using different storage backends could not make it into the OSS project. Many companies run their own Swift cluster, which is OSS. If you're using CephFS and everyone else wants to be using other Cloud storage solutions, that would actually put you at a disconnect with your users and leave room for a competitor with the tools and experience to scale out on Cloud storage to come in offering support. I would at least consider all the opinions in this thread and maybe reach out to that Minqi Pan fellow from Alibaba with questions.. I actually really like GitLab and wish we could be using it at my company; this is why I'm spending so much effort on this topic(and scaling git is interesting). Hopefully my opinions are not out of place.
solipsism 10y ago
Planning for the cloud doesn't make your software antifragile. antifragile != robust
user5994461 10y ago
They designed for a small operations with limited resources. That might be fair given their budget/funding which we don't know. At the moment, the discussion in the GitHub issues looks like people who are buying servers to put and run in their garage ^^

jghn 10y ago

Can you go more into the difference in the tradeoffs and how one should design differently?

wandernotlost 10y ago
A blunt summary would be that everything is unreliable and disposable. You have to design for failure, because most things will almost certainly fail (or just slow down) at some point. I learned a lot when I was first moving to the cloud from Adrian Cockroft. He has a ton of material out there from Netflix's move to the cloud. I recommend googling around for his conference talks. (I haven't watched these, but they're probably relevant: http://perfcap.blogspot.com/2012/03/cloud-architecture-tutorial.html http://perfcap.blogspot.com/2012/03/cloud-architecture-tutor...)

edejong 10y ago

I'm trying to understand 'antifragile'. Are you trying to say: 'robust'? If not, what is the difference?

Guzba 10y ago
I don't think antifragile works in the parent comment, robust would be better. As for a definition, antifragile things get stronger from abuse--like harming muslces during a workout so they grow back stronger. If they were just robust, it would be like machinery (no healing / strengthening).
- wandernotlost 10y ago
  I suppose I had Netflix in mind...as part of their move to the cloud, they developed chaos monkeys to actively attack their own software to make sure it is resilient to the failures that will inevitably, but less frequently, happen as a result of operating in the cloud.
  - solipsism 10y ago
    Stop using the term, you don't understand what it means. Chaos monkey is a form of testing. Testing is not antifragility.
    - wandernotlost 10y ago
      Chaos monkey attacks their production environment. They must make their software stronger/more resilient/less fragile in response. It would be more helpful if you clarified how you think that's different from antifragility.
      - solipsism 10y ago
        Antifragility describes a system that becomes more resilient automatically, because it is built such that it must, in response to attack or damage. Such a system doesn't require management. It doesn't require people to actively test, to think about how to make the system better, to implement the fixes.
        wandernotlost 10y ago
        I think that's an overly narrow definition of the term antifragility and interpretation of the system in this case. While I can imagine software that is in itself antifragile, I think it's entirely reasonable to include the totality of the people and process that make up an operational system, in which case even your narrow definition applies here.
        10y ago
nicky0 10y ago
There's a whole book[0] about it. But to summarise (poorly): 'robust' - resilient against shocks, volatility and stressors 'antifragile' - thrives on shocks, volatility and stressors (ie. gets better in response) Antifragile is a step beyond robust. Examples of antifragility are evolutionary systems and true free market economies (as opposed to our too-big-to fail version of propped-up, overly interconnected capitalism). 0: https://en.wikipedia.org/wiki/Antifragile https://en.wikipedia.org/wiki/Antifragile
- smallnamespace 10y ago
  Isn't that bad because now you're depending on having stressors and shocks to have the best performance? If you can't control the amount of stressors and shocks, you want a system that is neither antifragile or fragile, but strictly indifferent to the level of shocks.
  - nicky0 10y ago
    Good questions. It is a very interesting subject. If you haven't read the book, I recommend it. I think you will find it interesting if you read it with an open mind.
- dsr_ 10y ago
  Can you provide any examples of working, real-world antifragile systems which were designed and built by humans and accomplish their purposes? Preferably in terms of software and hardware? So far you've named "evolutionary systems", which are quite fragile in the real world, and an imaginary thing called a "true free market economy".
  - wandernotlost 10y ago
    Netflix with their chaos monkeys is a great and relevant example.
    - hueving 10y ago
      But their system doesn't thrive on chaos monkey. It's just resilient to it.

Rapzid 10y ago

This got me curious again about the pluggable storage backends in Git(I assume AWS code commit is using something like this). I've looked at Azures blob storage API in the past and found it incredibly flexible.. Here is an article from a few years ago: http://blog.deveo.com/your-git-repository-in-a-database-pluggable-backends-in-libgit2/ http://blog.deveo.com/your-git-repository-in-a-database-plug... In any case, GitLab is amazing and I can see how it's tempting to believe that GitLab the omnibus package is the core product. However, HOSTED GitLab's core product is GitLab as a SERVICE. That might require designs tailored a bit more for the cloud than simply operating a yoooge fs and calling it a day.

solatic 10y ago
Gitlab is limited by the fact that they need their hosted product to run the same code as their on-prem product in order to avoid forking the codebase. If on-prem customers can't get AWS/GCE/Azure SuperFastCloudStorage™, then it can't be part of their codebase.
- sytse 10y ago
  Exactly, we want our users to be able to scale with us using open source technologies. Some are at 20k+ users so they are close to needing something like Ceph themselves.
  - Rapzid 10y ago
    From looking at the comments here, thinking about what I would do/want, and what others have done to scale git GitLab is in the minority of wanting to solve this issue with a Ceph cluster. > Some are at 20k+ users so they are close to needing something like Ceph Or they will scale it themselves ala Alibaba: http://www.slideshare.net/MinqiPan/how-we-scaled-git-lab-for-a-30k-employee-company http://www.slideshare.net/MinqiPan/how-we-scaled-git-lab-for... . They appear to have written a libgit2 backend for their object store(among other things). I don't see a good reason why solutions using different storage backends could not make it into the OSS project. Many companies run their own Swift cluster, which is OSS. If you're using CephFS and everyone else wants to be using other Cloud storage solutions, that would actually put you at a disconnect with your users and leave room for a competitor with the tools and experience to scale out on Cloud storage to come in offering support. I would at least consider all the opinions in this thread and maybe reach out to that Minqi Pan fellow from Alibaba with questions.. I actually really like GitLab and wish we could be using it at my company; this is why I'm spending so much effort on this topic(and scaling git is interesting). Hopefully my opinions are not out of place.

solipsism 10y ago

Planning for the cloud doesn't make your software antifragile. antifragile != robust

user5994461 10y ago

They designed for a small operations with limited resources. That might be fair given their budget/funding which we don't know. At the moment, the discussion in the GitHub issues looks like people who are buying servers to put and run in their garage ^^