5 ms·

Could you provide a few links to popular open source projects, which switched to generics and got better code because of this?

cpuguy83 4y ago
Can you point to one that had the opposite effect?
- valyala 4y ago
  No. I suppose, maintainers of popular open source projects couldn't justify introducing generics into their code. This is exactly my story - I was actively seeking places in VictoriaMetrics codebase, which could benefit from switching to generics. Unfortunately, not a single place has been found yet. There was a promising place for generics [1], but it has been appeared that it is better from readability and maintainability PoV to use plain old interfaces instead. [1] https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/lib/blockcache/blockcache.go https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast...
  - cpuguy83 4y ago
    Tacking on generics on a working code base is difficult. I effectively used it in this little experimental CLI library: https://github.com/cpuguy83/go-cli/blob/main/command.go https://github.com/cpuguy83/go-cli/blob/main/command.go It's pretty simple, but the nice thing is it can use any flag library you want (stdlib flag package, pflag, whatever).
    - valyala 4y ago
      Try substituting F generic type with the FlagSet interface - and the code will become more clear without the need to use generics.
      - cpuguy83 4y ago
        That totally changes it as it requires type assertions to use. With generics `Flags()` returns a concrete type and not an interface.
        valyala 4y ago
        The question is whether the Flags() should return the concrete type instead of FlagSet interface? Another question is whether the additional complexity related to generics worth the ability to get the concrete type from Flags()? In fact, the generic-free implementation based on the FlagSet interface is more flexible, since it allows storing multiple different FlagSet implementations in the same Cmd.
        cpuguy83 4y ago
        I mean the generic type def there could be changed such that different Cmd's can have different flag implementations. The main point is that generics come in handy when building libraries so that you aren't forcing callers of your library into specific types or loosing some type safety.
jppittma 4y ago
Suppose you're the maintainer of a popular open source project, and you have a million users running a working, stable version using code generation. What priority would you assign a bug that migrates the existing, working code that other people are doing god knows what with downstream? What would the theoretical benefits "better code" be to your downstream, and how do they weigh against the cost of "I ran go mod tidy, and our build broke?" I can think of a use where it would lead to better code, and that's in k8s custom resources, where each resource type also has an associated list type that people create with code generation. It'd be much neater for k8s lists to be type List[T runtime.Object] struct { ... Items []T } Than the way it is now: https://www.google.com/search?q=site%3Agithub.com+zz_generated.deepcopy.go https://www.google.com/search?q=site%3Agithub.com+zz_generat...
jen20 4y ago
Here’s one: https://pkg.go.dev/golang.org/x/exp/maps https://pkg.go.dev/golang.org/x/exp/maps
- valyala 4y ago
  I doubt this library is useful. Some of the provided functions such as Clone(), Clear(), Copy() and Equal() are substituted by simple and readable three-line loops. Others are plain harmful. For example, Keys() and Values() may lead to huge memory allocations when working over big maps, while DeleteFunc() worsens code readability and maintainability comparing to a simple 3-line loop. Additionally, it may slow down the resulting code.
  - oauea 4y ago
    Useful enough to be included in the standard library. https://github.com/golang/go/issues/57436#issuecomment-1412566002 https://github.com/golang/go/issues/57436#issuecomment-14125...
    - valyala 4y ago
      That would be a mistake :(
      - oauea 4y ago
        FUD
  - jen20 4y ago
    > I doubt this library is useful It is. > Additionally, it may slow down the resulting code. FUD.
    - valyala 4y ago
      Agreed, it's my fault that I didn't explain why maps.DeleteFunc() can be slower than a simple loop with "if" statement inside. The reasons for the slowdown are: - Go compiler may fail to inline the callback passed to the maps.DeleteFunc(). This will result into an additional overhead for callback calls per each item in the map. - Go compiler may move some variables inside the callback from stack to a heap. This will result in an additional memory allocations comparing to a simple loop, leading to an additional load on garbage collector.
bboreham 4y ago
Well there’s this: https://github.com/prometheus/prometheus/pull/11054 https://github.com/prometheus/prometheus/pull/11054
- valyala 4y ago
  That's a good example of using generics-based sorting! I think it would be more beneficial to work on compiler optimizations, which could optimize the standard interface-based sort.Slice and sort.Sort to the same level. This shouldn't be too hard to do for the most widespread case when the compiler knows the actual type hidden behind the sort.Interface, so it could inline methods of this interface. This optimization then could speed up other similar interface-based algorithms.

cpuguy83 4y ago

Can you point to one that had the opposite effect?

valyala 4y ago
No. I suppose, maintainers of popular open source projects couldn't justify introducing generics into their code. This is exactly my story - I was actively seeking places in VictoriaMetrics codebase, which could benefit from switching to generics. Unfortunately, not a single place has been found yet. There was a promising place for generics [1], but it has been appeared that it is better from readability and maintainability PoV to use plain old interfaces instead. [1] https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/lib/blockcache/blockcache.go https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast...
- cpuguy83 4y ago
  Tacking on generics on a working code base is difficult. I effectively used it in this little experimental CLI library: https://github.com/cpuguy83/go-cli/blob/main/command.go https://github.com/cpuguy83/go-cli/blob/main/command.go It's pretty simple, but the nice thing is it can use any flag library you want (stdlib flag package, pflag, whatever).
  - valyala 4y ago
    Try substituting F generic type with the FlagSet interface - and the code will become more clear without the need to use generics.
    - cpuguy83 4y ago
      That totally changes it as it requires type assertions to use. With generics `Flags()` returns a concrete type and not an interface.
      - valyala 4y ago
        The question is whether the Flags() should return the concrete type instead of FlagSet interface? Another question is whether the additional complexity related to generics worth the ability to get the concrete type from Flags()? In fact, the generic-free implementation based on the FlagSet interface is more flexible, since it allows storing multiple different FlagSet implementations in the same Cmd.
        cpuguy83 4y ago
        I mean the generic type def there could be changed such that different Cmd's can have different flag implementations. The main point is that generics come in handy when building libraries so that you aren't forcing callers of your library into specific types or loosing some type safety.

jppittma 4y ago

Suppose you're the maintainer of a popular open source project, and you have a million users running a working, stable version using code generation. What priority would you assign a bug that migrates the existing, working code that other people are doing god knows what with downstream? What would the theoretical benefits "better code" be to your downstream, and how do they weigh against the cost of "I ran go mod tidy, and our build broke?" I can think of a use where it would lead to better code, and that's in k8s custom resources, where each resource type also has an associated list type that people create with code generation. It'd be much neater for k8s lists to be type List[T runtime.Object] struct { ... Items []T } Than the way it is now: https://www.google.com/search?q=site%3Agithub.com+zz_generated.deepcopy.go https://www.google.com/search?q=site%3Agithub.com+zz_generat...

jen20 4y ago

Here’s one: https://pkg.go.dev/golang.org/x/exp/maps https://pkg.go.dev/golang.org/x/exp/maps

valyala 4y ago
I doubt this library is useful. Some of the provided functions such as Clone(), Clear(), Copy() and Equal() are substituted by simple and readable three-line loops. Others are plain harmful. For example, Keys() and Values() may lead to huge memory allocations when working over big maps, while DeleteFunc() worsens code readability and maintainability comparing to a simple 3-line loop. Additionally, it may slow down the resulting code.
- oauea 4y ago
  Useful enough to be included in the standard library. https://github.com/golang/go/issues/57436#issuecomment-1412566002 https://github.com/golang/go/issues/57436#issuecomment-14125...
  - valyala 4y ago
    That would be a mistake :(
    - oauea 4y ago
      FUD
- jen20 4y ago
  > I doubt this library is useful It is. > Additionally, it may slow down the resulting code. FUD.
  - valyala 4y ago
    Agreed, it's my fault that I didn't explain why maps.DeleteFunc() can be slower than a simple loop with "if" statement inside. The reasons for the slowdown are: - Go compiler may fail to inline the callback passed to the maps.DeleteFunc(). This will result into an additional overhead for callback calls per each item in the map. - Go compiler may move some variables inside the callback from stack to a heap. This will result in an additional memory allocations comparing to a simple loop, leading to an additional load on garbage collector.

bboreham 4y ago

Well there’s this: https://github.com/prometheus/prometheus/pull/11054 https://github.com/prometheus/prometheus/pull/11054

valyala 4y ago
That's a good example of using generics-based sorting! I think it would be more beneficial to work on compiler optimizations, which could optimize the standard interface-based sort.Slice and sort.Sort to the same level. This shouldn't be too hard to do for the most widespread case when the compiler knows the actual type hidden behind the sort.Interface, so it could inline methods of this interface. This optimization then could speed up other similar interface-based algorithms.