4 ms·

How could such a tiny change have such an outsized impact on the site? “That same story happened so many different times,” my old boss David told me. “Someone l

How could such a tiny change have such an outsized impact on the site? “That same story happened so many different times,” my old boss David told me. “Someone launched a small, relatively innocuous change that did one of the millions of unexpected things it could have done, which then happened to break some part of the site, and then bring it all down—sometimes bring it down to the point where we couldn’t recover it for hours.” Preventing this is one of the basic motivations of functional programming. If a function is "referentially transparent" or "pure" it can't do anything unexpected. It seems to me this is the only reasonable way we can reason about software at scale.

jolmg 8y ago
While I generally agree that functional programming does help immensely to make secure software, in this case, the function wasn't pure. It was meant to return data that didn't exist from a database, but the error was silenced. Also, the reason why the servers were "burning" was because this somehow resulted in a memory leak that consumed all the memory available to the system, which is why they also had trouble pushing fixes. Even in a pure function in Haskell, memory leaks are possible and the same scenario could've resulted.
mreome 8y ago
While I can understand some of the benefits of functional programming within a single sub-system, or within academic research, I struggle to understand how pure functional programming can address the need for the large amounts of state information and abstraction layers required for a large scale system. I work with systems where (large/numerous) sub components require very specialized domain knowledge to understand internally, so multiple levels of black-box abstraction are required to integrate/interact with the sub components. Additionally, both the client systems and the sub components may have huge amounts of persistent state information. Could you suggest any resources that might help me understand how this can be accomplished in a pure functional paradigm? Something that assumes only a casual understanding of functional programming would be especially helpful.
- TheDong 8y ago
  > Do you have anything that explains this complex topic? > Something that assumes I don't know anything about the pre-requisite topic would be helpful Unfortunately the only way to gain a deep understanding of the powerful tools functional languages do give you is to gain a deep understanding of functional paradigms. Me saying "Monads can help with state abstractions" doesn't help you. The fact that large non-trivial applications have been written in haskell should be enough evidence that this is possible. The fact that the majority of people involved in such endeavors have claimed that it made their code safer and easier to refactor and had relatively few bugs should be enough evidence that it's a good idea. The fact that those who decry it more often than not do not know the subject should give their criticisms no weight. I don't see why you need any more evidence that you should simply learn functional programming so that you may first-hand answer your own question.
  - kbenson 8y ago
    > Unfortunately the only way to gain a deep understanding of the powerful tools functional languages do give you is to gain a deep understanding of functional paradigms. > I don't see why you need any more evidence that you should simply learn functional programming so that you may first-hand answer your own question. It is possible to interpret the GP comment as a request for good resources to learn FP. You seem like you could probably point one or two out. Since they've already expressed a desire to understand, that might be more beneficial for them than the equivalent of "just do it".
    - mreome 8y ago
      Thank you. Basically I was looking for the functional-programming equivalent to the OOP factory-equipment analogy, or asking if such an analogy does/can exist (if it can't, the why might also be a valuable explanation). That analogy can be used to explain the main ideas behind encapsulation, interfaces, and internal state, and why they are valuable, before having any understanding of how those are implemented in practice, and even without any programming knowledge at all. So far everything I've found on functional programming starts with implementation details and provides no initial framework for understanding the broad-strokes of organisation and value proposition.
adrianN 8y ago
A slight change in performance characteristics can also lead to cascading failures. Pure functions don't help with that. That aside, even functional programs need to have side effects (other than heating up the CPU). Referential transparency doesn't help when the state of your monad is a little bit weird and that causes some corner of your code to kill the DB.
freekh 8y ago
Anecdoctally this happened to me the other day in my Haskell backend: had a recursive function (in the IO monad) that every ten second would take a connection of my connection pool (and since it was recursive) would not give it back to the pool. Of course it was a stupid bug, but it was hard to figure out because it would take a while before it got to that point and once it was there every call to the backend would block then timeout. I like functional programming, and yes: it was in the IO monad so not pure, but I think most applications ends up in some stateful monad even if your stack is built for functional stuff... This was 'not at scale' but it as a Haskell project it is fairly big. Point being: yes, pure apps will not havr this predicament, but even with a stack built for pureness I can see it happen quite often. Running software is messy...
- Jeff_Brown 8y ago
  This makes another argument for Haskell, because the language allows you to isolate the non-pure code into a thin top layer. That makes debugging much easier than it would be if the complexity of IO extended all the way down through your app's subfunctions.
dxhdr 8y ago
> If a function is "referentially transparent" or "pure" it can't do anything unexpected. Sure it can; at a minimum all functions consume compute resources. A tiny change in a "pure" JavaScript function could lead to cascading deoptimizations that destroys performance and compromises service availability. For example, changing 0.0 to 0 could do it.
bunderbunder 8y ago
By definition, pure functions can't read from or write to disk, can't read from or write to the database, and can't read from or write to a socket. If you aren't able to do any of those things, there is no "at scale" - you're trapped in a single process on a single computer, spinning away but not able to communicate with anyone else. Pure functional programming helps within an individual component by letting you quarantine the statefulness at defined areas (e.g., in Haskell, at the program's entry point), but it doesn't let you make it disappear. Sooner or later, you're gonna have to deal with it. At scale, you've got tens or hundreds or thousands of different processes, and each of them has its own statefulness that it ultimately has to deal with. When you're looking at the big picture like that, I'm not sure it makes sense to worry about whether it's been quarantined, Haskell-style, or not. It's still there, and, in the big picture, it's still distributed more-or-less homogeneously throughout the system. Even a process that doesn't use disk or talk to the DB or anything like that is still not referentially transparent, because it's still subject to side effects by virtue of the network being unreliable, the OOM killer being unpredictable, and all that fun stuff.
- Jeff_Brown 8y ago
  > I'm not sure it makes sense to worry about whether > it's been quarantined, Haskell-style, or not. > It's still there True, but the quarantine makes the bug much easier to find.

jolmg 8y ago

While I generally agree that functional programming does help immensely to make secure software, in this case, the function wasn't pure. It was meant to return data that didn't exist from a database, but the error was silenced. Also, the reason why the servers were "burning" was because this somehow resulted in a memory leak that consumed all the memory available to the system, which is why they also had trouble pushing fixes. Even in a pure function in Haskell, memory leaks are possible and the same scenario could've resulted.

mreome 8y ago

While I can understand some of the benefits of functional programming within a single sub-system, or within academic research, I struggle to understand how pure functional programming can address the need for the large amounts of state information and abstraction layers required for a large scale system. I work with systems where (large/numerous) sub components require very specialized domain knowledge to understand internally, so multiple levels of black-box abstraction are required to integrate/interact with the sub components. Additionally, both the client systems and the sub components may have huge amounts of persistent state information. Could you suggest any resources that might help me understand how this can be accomplished in a pure functional paradigm? Something that assumes only a casual understanding of functional programming would be especially helpful.

TheDong 8y ago
> Do you have anything that explains this complex topic? > Something that assumes I don't know anything about the pre-requisite topic would be helpful Unfortunately the only way to gain a deep understanding of the powerful tools functional languages do give you is to gain a deep understanding of functional paradigms. Me saying "Monads can help with state abstractions" doesn't help you. The fact that large non-trivial applications have been written in haskell should be enough evidence that this is possible. The fact that the majority of people involved in such endeavors have claimed that it made their code safer and easier to refactor and had relatively few bugs should be enough evidence that it's a good idea. The fact that those who decry it more often than not do not know the subject should give their criticisms no weight. I don't see why you need any more evidence that you should simply learn functional programming so that you may first-hand answer your own question.
- kbenson 8y ago
  > Unfortunately the only way to gain a deep understanding of the powerful tools functional languages do give you is to gain a deep understanding of functional paradigms. > I don't see why you need any more evidence that you should simply learn functional programming so that you may first-hand answer your own question. It is possible to interpret the GP comment as a request for good resources to learn FP. You seem like you could probably point one or two out. Since they've already expressed a desire to understand, that might be more beneficial for them than the equivalent of "just do it".
  - mreome 8y ago
    Thank you. Basically I was looking for the functional-programming equivalent to the OOP factory-equipment analogy, or asking if such an analogy does/can exist (if it can't, the why might also be a valuable explanation). That analogy can be used to explain the main ideas behind encapsulation, interfaces, and internal state, and why they are valuable, before having any understanding of how those are implemented in practice, and even without any programming knowledge at all. So far everything I've found on functional programming starts with implementation details and provides no initial framework for understanding the broad-strokes of organisation and value proposition.

adrianN 8y ago

A slight change in performance characteristics can also lead to cascading failures. Pure functions don't help with that. That aside, even functional programs need to have side effects (other than heating up the CPU). Referential transparency doesn't help when the state of your monad is a little bit weird and that causes some corner of your code to kill the DB.

freekh 8y ago

Anecdoctally this happened to me the other day in my Haskell backend: had a recursive function (in the IO monad) that every ten second would take a connection of my connection pool (and since it was recursive) would not give it back to the pool. Of course it was a stupid bug, but it was hard to figure out because it would take a while before it got to that point and once it was there every call to the backend would block then timeout. I like functional programming, and yes: it was in the IO monad so not pure, but I think most applications ends up in some stateful monad even if your stack is built for functional stuff... This was 'not at scale' but it as a Haskell project it is fairly big. Point being: yes, pure apps will not havr this predicament, but even with a stack built for pureness I can see it happen quite often. Running software is messy...

Jeff_Brown 8y ago
This makes another argument for Haskell, because the language allows you to isolate the non-pure code into a thin top layer. That makes debugging much easier than it would be if the complexity of IO extended all the way down through your app's subfunctions.

dxhdr 8y ago

> If a function is "referentially transparent" or "pure" it can't do anything unexpected. Sure it can; at a minimum all functions consume compute resources. A tiny change in a "pure" JavaScript function could lead to cascading deoptimizations that destroys performance and compromises service availability. For example, changing 0.0 to 0 could do it.

bunderbunder 8y ago

By definition, pure functions can't read from or write to disk, can't read from or write to the database, and can't read from or write to a socket. If you aren't able to do any of those things, there is no "at scale" - you're trapped in a single process on a single computer, spinning away but not able to communicate with anyone else. Pure functional programming helps within an individual component by letting you quarantine the statefulness at defined areas (e.g., in Haskell, at the program's entry point), but it doesn't let you make it disappear. Sooner or later, you're gonna have to deal with it. At scale, you've got tens or hundreds or thousands of different processes, and each of them has its own statefulness that it ultimately has to deal with. When you're looking at the big picture like that, I'm not sure it makes sense to worry about whether it's been quarantined, Haskell-style, or not. It's still there, and, in the big picture, it's still distributed more-or-less homogeneously throughout the system. Even a process that doesn't use disk or talk to the DB or anything like that is still not referentially transparent, because it's still subject to side effects by virtue of the network being unreliable, the OOM killer being unpredictable, and all that fun stuff.

Jeff_Brown 8y ago
> I'm not sure it makes sense to worry about whether > it's been quarantined, Haskell-style, or not. > It's still there True, but the quarantine makes the bug much easier to find.