4 ms·

Layered Design in Go

141 points by misonic 1y ago

__s 1y ago
Kind of reminds me of the concept of spheres in randomizers
breadchris 1y ago
great blog post! also this website has a lot of incredible posts, if you like learning about functional programming, you should check out https://jerf.org/iri/blogbooks/functional-programming-lessons-in-imperative-code/ https://jerf.org/iri/blogbooks/functional-programming-lesson...
rapidlua 1y ago
> Packages may not circularly reference each other. Actually possible with go:linkname.
- jerf 1y ago
  Fair, but I think we can classify that under "unsafe" and ignore it under normal circumstances. I can also say things like "Go doesn't have pointer arithmetic" with a straight face, even though unsafe permits pointer arithmetic just fine. If you're programming with that routinely, you're out of the bounds of my advice for architecture anyhow. Whether for good or bad reasons would be left as an exercise for the architect in question.
  - ncruces 1y ago
    Looking at the diagram for the SQLite VFS page I didn't think I was going overboard with designing my driver around 3 packages: https://sqlite.org/vfs.html https://sqlite.org/vfs.html One layer wraps SQLite C API, below it lives a pure Go VFS, and above all that database/sql driver. Even this coarse split (with a collection internal packages, the bigger one of them “utils”) is enough to need various band-aids to accommodate the impossibility of circular dependencies. I honestly don't think it helps much. At the module level, there are obvious benefits from the impossibility. Just like all the pain around v2 modules can be justified, even if I find it annoying. When packages are also the only layer at which you can enforce visibility, it becomes worse.
    - jerf 1y ago
      See the last case of how to handle circular dependencies. Ports are special cases. Always are.
pjmlp 1y ago
Looks like I am reading a book about Yourdon structured method.
- jerf 1y ago
  Since I meant this just as a "how I do it" post I suppose I forgot the disclaimer that I'm not particularly claiming to have invented anything or to be the first. Indeed to a large degree I consider myself just to be following the grain of Go and hardly doing anything myself. That said, after some quick googling around, I don't think I feel bad not knowing what Yourdon design is, as it seems to be somewhat proprietary and behind paywalls, so it's hard for me to tell if there's much similarity. Certainly it has a lot of stuff I tend to eschew; lots of references to diagrams and state charts and such. I tend to prefer a more "agile but wait before you panic I mean 'original' agile not 'scrum' or whatever other abomination it was turned into", my formal method is more based around exploring the design space with extensive unit tests and code rather than that sort of up-front design.
  - pjmlp 1y ago
    It was a common way to structure enterprise C code during the 1990's, and the snarky remark is how the anti-enterprise culture from Go ends up adopting the same big corporation principles, given enough wind behind its sails. Yourdon is the big wave of enterprise methodologies immediately predating the OOP wave with Booch, UML, GoF and friends. I can gladly bet there are some Go pattern books around the corner as well. As for the book paywall, it is certainly available in many libraries, given its age.
    - jerf 1y ago
      Nobody has accused my code bases of being "too enterprise" yet. Edit: I should probably elaborate on that before my edit window closes. Other than pervasive use of dependency injection, done directly with no framework simply by passing values around, there are effectively no "Enterprise" structures in sight in my code base. That's what I mean by "this design is sufficient for me". The only thing that resembles a "factory" is in the precise place I need to construct values from a type specified by an input string. No patterns put in place "just in case". No top-level frameworks used for 3% of their functionality. I use a process monitor but the interface that requires is "Serve(context.Context)", which is just the minimum you need to be able to monitor a service. There are what I'd call "patterns", but they're there to do their job to the full, not guesses about what maybe I'll need later. I've actually got a half-written pattern book for Go I've been trying to figure out what to do with, but the introduction is basically "why pattern books shouldn't just be a recitation of the original GoF patterns", because that is, well, stupid. Even the original book bit off too much trying to straddle Smalltalk and C++ in one shot. They require different patterns. My pattern book is how to solve Go problems in Go, not to give people words to slap in their code in case someday their code might grow enough to need it.
kubb 1y ago
Cool description of how jerf thinks about packages and how he deals with circular dependencies!
nickcw 1y ago
In my opinion, not allowing circular dependencies is a great design choice for building large programs. It forces you to separate your concerns properly. If you get a circular dependency something is wrong with your design and the article does a good job on how to fix them. I sometimes use function pointers which other packages override to fix circular dependencies which I don't think was mentioned in the article. My only wish is that the go compiler gave more helpful output when you make a circular dependency. Currently it gives a list of all the packages involved in the loop which can be quite long, though generally it is the last thing you changed which caused the problem.
- ternaryoperator 1y ago
  In the abstract, I think I agree with you. But in reality, what I see is that go projects use far fewer packages than, say, programs in Java. Many go projects use one or two omnibus packages--principally, I expect, to avoid having to worry about circularity issues. By forcing this design pattern on developers (something no other language does), I think the result has been overall worse code rather than better. Perhaps a warning, rather than stop-the-compiler error would have been a better choice. Not sure. Either way though, I wholly agree that the compiler gives too little information, which is curious because it knows the needed data and should easily be able to present it in a useful way.
  - jen20 1y ago
    I see the opposite in most Java programs: poor organization with packages based on a type of thing (eg models, controllers) rather than related behaviors. Go doesn’t have warnings, which is great - if something is worth warning about, it is also worth erroring about. It never ceases to amaze me when a brand new JavaScript project spits out dozens of errors after pulling in a common library, and everyone thinks that is ok.
    - ncruces 1y ago
      Of course Go has warnings. It has go vet, which is not a linter, and according to the authors doesn't need comments to ignore checks, because the checks are always correct about you having written shitty code. Except where it might warn you about something completely outside your control.
shizcakes 1y ago
One bonus technique related to the “move to a third package” advice: generating many of your model structures (SQL, Protobuf, graphql, etc) allows you to set up obvious directionality between generated layers and to provide all generated code as “base packages” to your application code, which then composes everything together. Prior to this technique we often had “models importing models circularly” as an issue but that’s entirely disappeared due to the introduction of the structural additional layer.
darioush 1y ago
A funny quirk about golang is you cannot have circular dependencies at the package level, but you can have circular dependencies in go.mod The tl;dr is don't do that either.
- 0xjnml 1y ago
  go.mod files have no dependencies, so neither they can have circular dependencies. go.mod files just list the dependencies of their respective package.

141 points by misonic 1y ago

__s 1y ago

Kind of reminds me of the concept of spheres in randomizers

great blog post! also this website has a lot of incredible posts, if you like learning about functional programming, you should check out https://jerf.org/iri/blogbooks/functional-programming-lessons-in-imperative-code/ https://jerf.org/iri/blogbooks/functional-programming-lesson...

rapidlua 1y ago

> Packages may not circularly reference each other. Actually possible with go:linkname.

jerf 1y ago
Fair, but I think we can classify that under "unsafe" and ignore it under normal circumstances. I can also say things like "Go doesn't have pointer arithmetic" with a straight face, even though unsafe permits pointer arithmetic just fine. If you're programming with that routinely, you're out of the bounds of my advice for architecture anyhow. Whether for good or bad reasons would be left as an exercise for the architect in question.
- ncruces 1y ago
  Looking at the diagram for the SQLite VFS page I didn't think I was going overboard with designing my driver around 3 packages: https://sqlite.org/vfs.html https://sqlite.org/vfs.html One layer wraps SQLite C API, below it lives a pure Go VFS, and above all that database/sql driver. Even this coarse split (with a collection internal packages, the bigger one of them “utils”) is enough to need various band-aids to accommodate the impossibility of circular dependencies. I honestly don't think it helps much. At the module level, there are obvious benefits from the impossibility. Just like all the pain around v2 modules can be justified, even if I find it annoying. When packages are also the only layer at which you can enforce visibility, it becomes worse.
  - jerf 1y ago
    See the last case of how to handle circular dependencies. Ports are special cases. Always are.

pjmlp 1y ago

Looks like I am reading a book about Yourdon structured method.

jerf 1y ago
Since I meant this just as a "how I do it" post I suppose I forgot the disclaimer that I'm not particularly claiming to have invented anything or to be the first. Indeed to a large degree I consider myself just to be following the grain of Go and hardly doing anything myself. That said, after some quick googling around, I don't think I feel bad not knowing what Yourdon design is, as it seems to be somewhat proprietary and behind paywalls, so it's hard for me to tell if there's much similarity. Certainly it has a lot of stuff I tend to eschew; lots of references to diagrams and state charts and such. I tend to prefer a more "agile but wait before you panic I mean 'original' agile not 'scrum' or whatever other abomination it was turned into", my formal method is more based around exploring the design space with extensive unit tests and code rather than that sort of up-front design.
- pjmlp 1y ago
  It was a common way to structure enterprise C code during the 1990's, and the snarky remark is how the anti-enterprise culture from Go ends up adopting the same big corporation principles, given enough wind behind its sails. Yourdon is the big wave of enterprise methodologies immediately predating the OOP wave with Booch, UML, GoF and friends. I can gladly bet there are some Go pattern books around the corner as well. As for the book paywall, it is certainly available in many libraries, given its age.
  - jerf 1y ago
    Nobody has accused my code bases of being "too enterprise" yet. Edit: I should probably elaborate on that before my edit window closes. Other than pervasive use of dependency injection, done directly with no framework simply by passing values around, there are effectively no "Enterprise" structures in sight in my code base. That's what I mean by "this design is sufficient for me". The only thing that resembles a "factory" is in the precise place I need to construct values from a type specified by an input string. No patterns put in place "just in case". No top-level frameworks used for 3% of their functionality. I use a process monitor but the interface that requires is "Serve(context.Context)", which is just the minimum you need to be able to monitor a service. There are what I'd call "patterns", but they're there to do their job to the full, not guesses about what maybe I'll need later. I've actually got a half-written pattern book for Go I've been trying to figure out what to do with, but the introduction is basically "why pattern books shouldn't just be a recitation of the original GoF patterns", because that is, well, stupid. Even the original book bit off too much trying to straddle Smalltalk and C++ in one shot. They require different patterns. My pattern book is how to solve Go problems in Go, not to give people words to slap in their code in case someday their code might grow enough to need it.

kubb 1y ago

Cool description of how jerf thinks about packages and how he deals with circular dependencies!

nickcw 1y ago

In my opinion, not allowing circular dependencies is a great design choice for building large programs. It forces you to separate your concerns properly. If you get a circular dependency something is wrong with your design and the article does a good job on how to fix them. I sometimes use function pointers which other packages override to fix circular dependencies which I don't think was mentioned in the article. My only wish is that the go compiler gave more helpful output when you make a circular dependency. Currently it gives a list of all the packages involved in the loop which can be quite long, though generally it is the last thing you changed which caused the problem.

ternaryoperator 1y ago
In the abstract, I think I agree with you. But in reality, what I see is that go projects use far fewer packages than, say, programs in Java. Many go projects use one or two omnibus packages--principally, I expect, to avoid having to worry about circularity issues. By forcing this design pattern on developers (something no other language does), I think the result has been overall worse code rather than better. Perhaps a warning, rather than stop-the-compiler error would have been a better choice. Not sure. Either way though, I wholly agree that the compiler gives too little information, which is curious because it knows the needed data and should easily be able to present it in a useful way.
- jen20 1y ago
  I see the opposite in most Java programs: poor organization with packages based on a type of thing (eg models, controllers) rather than related behaviors. Go doesn’t have warnings, which is great - if something is worth warning about, it is also worth erroring about. It never ceases to amaze me when a brand new JavaScript project spits out dozens of errors after pulling in a common library, and everyone thinks that is ok.
  - ncruces 1y ago
    Of course Go has warnings. It has go vet, which is not a linter, and according to the authors doesn't need comments to ignore checks, because the checks are always correct about you having written shitty code. Except where it might warn you about something completely outside your control.

shizcakes 1y ago

One bonus technique related to the “move to a third package” advice: generating many of your model structures (SQL, Protobuf, graphql, etc) allows you to set up obvious directionality between generated layers and to provide all generated code as “base packages” to your application code, which then composes everything together. Prior to this technique we often had “models importing models circularly” as an issue but that’s entirely disappeared due to the introduction of the structural additional layer.

darioush 1y ago

A funny quirk about golang is you cannot have circular dependencies at the package level, but you can have circular dependencies in go.mod The tl;dr is don't do that either.

0xjnml 1y ago
go.mod files have no dependencies, so neither they can have circular dependencies. go.mod files just list the dependencies of their respective package.