5 ms·

My company has used Julia for about 5 years now. 90% of the time in application code you only need single dispatch, same as OOP. One case where I actually rely

My company has used Julia for about 5 years now. 90% of the time in application code you only need single dispatch, same as OOP. One case where I actually rely on multiple dispatch is conversion to more or less structured data. Inspired by Clojure, I have a function `conform(OutputDataType, input_data::SomeType)` and specific implementations depend on both types involved. Multiple dispatch is also really helpful when two libraries don't quite do the same thing. In python pandas, numpy, and pytorch all have slightly different implementations of x.std() (standard deviation) with slightly different APIs. This means you can't write code that's generic across a numpy array or a pytorch tensor. In Python this could only easily be fixed if library authors coordinate, but with multiple dispatch you can just fix this in your own code.

paddy_m 1y ago
That makes sense for writing your own generic libraries. Do you frequently have to convert from pytorch to numpy? I would think that a for a given execution unit (pipeline step, single program, model running on a server) that the data will stay as the same type after some initial conversion.
- SatvikBeri 1y ago
  I'd say we have about 60% library code and 40% application code internally, so making it easier to write libraries is really nice. E.g. you don't want to have to write different statistical calculations for every type you use. In particular, the fact that these types would silently give you different answers if you called `.std()` was a big headache It was very common for us to want to be generic over pandas series and numpy arrays. A bit less so with pytorch tensors, but that was because we just aggressively converted them to numpy arrays. Fundamentally these three are all very similar types so it's frustrating to have to treat them differently.
  - paddy_m 1y ago
    which implementation of `std()` did you go with? I was writing unit tests once against histograms. That code is super finnicky, and I couldn't get pandas and polars numbers to tie out. I wasn't super concerned with the exact output for my application, just that number of buckets was the same and they were roughly the same size. Just bumping to a new version of numpy would result in test breakages because of floating point rounding errors. I'm sure there are much more robust numerical testing things I could have done
    - SatvikBeri 1y ago
      Yeah this kind of stuff drove us crazy. Numpy uses things like SIMD everywhere with no option to turn it off, which is good for performance but makes life really hard when you want consistently reproducible results across different machines. Before switching to Julia we eventually standardized on numpy.std with ddof=1, and with some error tolerances in our tests.

paddy_m 1y ago

That makes sense for writing your own generic libraries. Do you frequently have to convert from pytorch to numpy? I would think that a for a given execution unit (pipeline step, single program, model running on a server) that the data will stay as the same type after some initial conversion.

SatvikBeri 1y ago
I'd say we have about 60% library code and 40% application code internally, so making it easier to write libraries is really nice. E.g. you don't want to have to write different statistical calculations for every type you use. In particular, the fact that these types would silently give you different answers if you called `.std()` was a big headache It was very common for us to want to be generic over pandas series and numpy arrays. A bit less so with pytorch tensors, but that was because we just aggressively converted them to numpy arrays. Fundamentally these three are all very similar types so it's frustrating to have to treat them differently.
- paddy_m 1y ago
  which implementation of `std()` did you go with? I was writing unit tests once against histograms. That code is super finnicky, and I couldn't get pandas and polars numbers to tie out. I wasn't super concerned with the exact output for my application, just that number of buckets was the same and they were roughly the same size. Just bumping to a new version of numpy would result in test breakages because of floating point rounding errors. I'm sure there are much more robust numerical testing things I could have done
  - SatvikBeri 1y ago
    Yeah this kind of stuff drove us crazy. Numpy uses things like SIMD everywhere with no option to turn it off, which is good for performance but makes life really hard when you want consistently reproducible results across different machines. Before switching to Julia we eventually standardized on numpy.std with ddof=1, and with some error tolerances in our tests.