3 ms·

Of the bugs that I've experienced in large-scale, Rails production systems, typing is a small subset. Manually reconstructing logistical errors based on a comb

by aantix 3y ago

Of the bugs that I've experienced in large-scale, Rails production systems, typing is a small subset. Manually reconstructing logistical errors based on a combination of user input and system data, are the most time-consuming issues to diagnose. When your codebase is 500,000+ lines of code, which code paths are relevant for a given endpoint? What methods were called and under what context? How do we begin to reconstruct this bug? These are the scenarios for which Call Stacking gives instant visibility to.

IshKebab 3y ago
Depends on the type system. I would say for Java / Python level static types they catch a small but significant fraction of bugs (10-20% according to the only objective measurement I've seen, which is easily worth it). However some languages like Rust, Haskell and OCaml let you express much more in the type system. Subjectively it feels like that catches more like 30-60% of bugs. So this thing is still useful but Berkes is right that you need it a lot less if you use better static types. > which code paths are relevant for a given endpoint? This is exactly the question that static types can answer... statically. You don't need a runtime log to find out. You do need a runtime log to see the actual values though. So it's not like a debugger is completely useless in Rust. But I definitely reach for it much less than in other languages.
- theonething 3y ago
  > This is exactly the question that static types can answer... statically. You don't need a runtime log to find out. This tool seems to be able to display the relevant code paths. That sounds super convenient and useful. Do statically typed languages have tools to do similar?
  - IshKebab 3y ago
    Yeah "find all references" will show you everything that can call a particular function. As I said it won't give you the actual values so this still seems useful.
  - kikimora 3y ago
    I don’t know if a similar tool exists but in principle Java and .NET has profiling APIs that let you look into method calls. They can be used as a basis for a similar tool.
- eddtries 3y ago
  I worked on a large financial transaction system for up to $7B per day in Haskell that used formal methods like Agda, TLA+, etc, with a lot of logic in the type level (i.e. Liquid Haskell), as a test engineer. The entire system was covered in property based tests, proofs, and then normal tests from unit -> system. We literally had two bugs categorised as P2/P1/P0 on release, both of which were design related, and both were fixed before users saw them. It was crazy effective (but took years).
  - tome 3y ago
    Nice! Can you say what company you were working for?
    - eddtries 3y ago
      I don’t want to totally dox myself but there’s a couple crypto companies doing that sort of stuff like Algorand, Tezos, Cardano, etc :)
philwelch 3y ago
> Of the bugs that I've experienced in large-scale, Rails production systems, typing is a small subset. Really? Are you counting errors where a value turned out to be nil when it wasn’t expected to be? Because that’s a type error. It’s not a type error that many statically typed languages fix (Java is notorious for null pointer exceptions) but it’s a type error that can, in principle, be fixed with static typing.
- dlisboa 3y ago
  In the app I work on a huge percentage (60% in the timeframe I analyzed) of the errors are errors a type system would catch. Nil references, long method chains where the middle call returns a different object than expected, bad refactorings changing the return type of methods… It’s staggering really. These could be solved with better programming practices, they’re not errors usually experienced programmers make, but they exist nonetheless. An exception/error system, with checked exceptions or handling of errors via result monad/errors like in Go, would solve the vast majority of other errors. Type systems also encourage other things like contracts on API endpoints and typed messages, serving as a early warning system when writing those bugs. Purely business logic bugs are actually not common. I guess we haven’t had the opportunity to create those “higher order” bugs while wading through the others.
  - sethherr 3y ago
    Do you have tests? I feel like 60% of the errors I encounter is high for type errors. I also feel like with testing, fixing those errors occupies less than 5% of my debugging time.
- nicolas_t 3y ago
  I'm always surprised by that, I've worked with 10-15 people team on relatively large rails codebase and, yes, the bugs we usually see are not type related bugs (including nil when it's not expected). I keep reading people saying that type systems eliminate a huge class of bugs but, it's not been my experience with languages that use poor type systems (java, rust... ). Languages that have much better types like OCaml are different and I've had great luck with OCaml in particular (my favorite language to use when I can) but, there's also the fact that devs who use ocaml tend to also be much better than average (in the same way that in my experience devs working with php or nodejs exclusively tend to be much worse than average). Note before I'm downvoted for my last comment, there are exceptions but looking at the average candidate applying for a php, nodejs position compared to one for rails and one for ocaml.

by aantix 3y ago

IshKebab 3y ago

Depends on the type system. I would say for Java / Python level static types they catch a small but significant fraction of bugs (10-20% according to the only objective measurement I've seen, which is easily worth it). However some languages like Rust, Haskell and OCaml let you express much more in the type system. Subjectively it feels like that catches more like 30-60% of bugs. So this thing is still useful but Berkes is right that you need it a lot less if you use better static types. > which code paths are relevant for a given endpoint? This is exactly the question that static types can answer... statically. You don't need a runtime log to find out. You do need a runtime log to see the actual values though. So it's not like a debugger is completely useless in Rust. But I definitely reach for it much less than in other languages.

theonething 3y ago
> This is exactly the question that static types can answer... statically. You don't need a runtime log to find out. This tool seems to be able to display the relevant code paths. That sounds super convenient and useful. Do statically typed languages have tools to do similar?
- IshKebab 3y ago
  Yeah "find all references" will show you everything that can call a particular function. As I said it won't give you the actual values so this still seems useful.
- kikimora 3y ago
  I don’t know if a similar tool exists but in principle Java and .NET has profiling APIs that let you look into method calls. They can be used as a basis for a similar tool.
eddtries 3y ago
I worked on a large financial transaction system for up to $7B per day in Haskell that used formal methods like Agda, TLA+, etc, with a lot of logic in the type level (i.e. Liquid Haskell), as a test engineer. The entire system was covered in property based tests, proofs, and then normal tests from unit -> system. We literally had two bugs categorised as P2/P1/P0 on release, both of which were design related, and both were fixed before users saw them. It was crazy effective (but took years).
- tome 3y ago
  Nice! Can you say what company you were working for?
  - eddtries 3y ago
    I don’t want to totally dox myself but there’s a couple crypto companies doing that sort of stuff like Algorand, Tezos, Cardano, etc :)

philwelch 3y ago

> Of the bugs that I've experienced in large-scale, Rails production systems, typing is a small subset. Really? Are you counting errors where a value turned out to be nil when it wasn’t expected to be? Because that’s a type error. It’s not a type error that many statically typed languages fix (Java is notorious for null pointer exceptions) but it’s a type error that can, in principle, be fixed with static typing.

dlisboa 3y ago
In the app I work on a huge percentage (60% in the timeframe I analyzed) of the errors are errors a type system would catch. Nil references, long method chains where the middle call returns a different object than expected, bad refactorings changing the return type of methods… It’s staggering really. These could be solved with better programming practices, they’re not errors usually experienced programmers make, but they exist nonetheless. An exception/error system, with checked exceptions or handling of errors via result monad/errors like in Go, would solve the vast majority of other errors. Type systems also encourage other things like contracts on API endpoints and typed messages, serving as a early warning system when writing those bugs. Purely business logic bugs are actually not common. I guess we haven’t had the opportunity to create those “higher order” bugs while wading through the others.
- sethherr 3y ago
  Do you have tests? I feel like 60% of the errors I encounter is high for type errors. I also feel like with testing, fixing those errors occupies less than 5% of my debugging time.
nicolas_t 3y ago
I'm always surprised by that, I've worked with 10-15 people team on relatively large rails codebase and, yes, the bugs we usually see are not type related bugs (including nil when it's not expected). I keep reading people saying that type systems eliminate a huge class of bugs but, it's not been my experience with languages that use poor type systems (java, rust... ). Languages that have much better types like OCaml are different and I've had great luck with OCaml in particular (my favorite language to use when I can) but, there's also the fact that devs who use ocaml tend to also be much better than average (in the same way that in my experience devs working with php or nodejs exclusively tend to be much worse than average). Note before I'm downvoted for my last comment, there are exceptions but looking at the average candidate applying for a php, nodejs position compared to one for rails and one for ocaml.