8 ms·

Maintain a clean architecture in Python with dependency rules

139 points by rekahrv 4y ago

Thaxll 4y ago
Your DB / api layer should never touch the same models.
- rekahrv 4y ago
  Thanks, that's a good point and perhaps a good topic for a future post :-) How to ensure that the API and the db use different models even if those models are in the same package?
  - dangets 4y ago
    I struggle with this also, I assume the answer is to not have them in the same package. You can also break the application into separate 'domain', 'infra', 'application' modules as documented in [0] with rules on what dependencies are allowed in each module (e.g. domain should not have db or serialization implementation). The problem is that this does create several adapter layers which adds to the mental complexity. [0] https://learn.microsoft.com/en-us/dotnet/architecture/microservices/microservice-ddd-cqrs-patterns/ddd-oriented-microservice https://learn.microsoft.com/en-us/dotnet/architecture/micros...
- inwit 4y ago
  It's these kind of rules that mean I'm here wading through 5 layers of exquisitely decoupled nonsense that could be done in a few lines
  - Thaxll 4y ago
    Convert function between API and DB model does not sound complicated. Storing the API model in your DB is really a bad idea.
  - camgunz 4y ago
    I just listened to the DHH/Kent Beck/Martin Fowler discussion about TDD "damage" and both sides still seemed unconvinced by the end of it, but this exact example came up. It seems like SOA (whether it's DDD or Hexagonal or Clean or w/e) and TDD really push you towards this kind of layer bloat for one reason or another. I'm (maybe obviously) on the SOA-skeptic side, my arguments generally are: - Most apps aren't that big and don't need multiple layers of abstraction (i.e. the ORM and its models are totally fine). If the app starts getting too big for its britches, probably the best thing to do is make it 2 apps (too big: 2 apps is a good slogan here). - Dependency injection and mocks are pretty bad ideas that are only occasionally useful (DHH uses the example of a payments gateway), but mostly push IoC through your whole app and make control flow confusingly backwards. Mocks are always in disrepair, and almost never accurately reflect what they're trying to mock, and thus ironically are big vectors for bugs that make it through testing. - Having tons of unit tests tends to slow eng velocity to a crawl, because they test the parts of the application that aren't the requirements (were these functions called, what's the call signature of this function, was this class instantiated, etc.). Unit tests create a super-fine-grained shadow spec about the lowest level details of your application, and mostly I think they shouldn't ever be committed to a repo. They help during individual development, but then the whole team is stuck maintaining them forever whenever they make changes. They also tend to slow down CI because they're slow and always flaky. - You almost certainly will never need to switch databases, let alone abstract across a database, a message queue, and a web api. It's not worth doing a "repo" abstraction and encapsulating those details. - There are (now) really good libraries for almost anything you want to do. ORMs literally map database entities to domain entities--they just abstract the persistence for you. Sounds like a repo to me! We also have good validation, logging, monitoring, auth/auth etc. built into frameworks and 3rd party services. A lot of the things you might put into other layers or even other services are now neatly packaged into libraries/frameworks you can just use and SaaS things you can just buy, leaving you free to just implement your business logic.
    - hbrn 4y ago
      I mostly agree with you and DHH on that topic, however in my experience reasonably applied SOA/DDD actually shields me from this layering nonsense. When your apps live as a service on the network or as a nicely isolated module in your repo, you no longer have a reason to over-engineer them. You don't need a grandiose architecture that solves every problem, instead you can make local decisions that are good enough in the specific context. Though, admittedly, I found it hard to sell such "inconsistencies" to other tech leaders, most folks aspire to those grandiosities. > If the app starts getting too big for its britches, probably the best thing to do is make it 2 apps That's the argument in favor of SOA, isn't it?
- aobdev 4y ago
  I hate that these are called models (probably because they extend pydantic’s BaseModel), but if they were called Schema or Serializers would this still be true? Typically what you see in a FastAPI project is a class that parsers the request body, and the same or slightly modified class that serializes the response back out after touching the DB. And this isn’t a new idea, because Flask+Marshmallow and DRF do the exact same thing.
  - rekahrv 4y ago
    I've used multiple names for similar packages incl. `models` and `schemas`. :-) Yes, for this example, I picked `models` to follow Pydantic's terminology. IMO, the FastAPI approach you described makes a lot of sense: The "schema" stored in the db and the "schema" returned by the API aren't the same, but they are quite similar. They have many common properties => They can often have a common base class.
    - deleted 4y ago
      [deleted]
- marginalia_nu 4y ago
  Why not?
w_t_payne 4y ago
I do exactly this in my side project. I have a set of rules which put restrictions on which packages and modules can be included from other packages and modules. For example, a high maturity package is not allowed to depend upon a low maturity package. Similarly, a core library package is not allowed to rely on a package that is specific to a particular product or a particular piece of bespoke development. In this way, much of the potential for circular dependencies is eliminated, and the purpose and internet is clearly communicated. (I don't do this using sourcery though ... I have my own set of rules)
- rekahrv 4y ago
  That's very cool. Can you tell a bit more about this set of rules? "For example, a high maturity package is not allowed to depend upon a low maturity package. Similarly, a core library package is not allowed to rely on a package that is specific to a particular product or a particular piece of bespoke development." I really like these.
  - w_t_payne 4y ago
    I have a general system for representing metadata in source files. (I use YAML documents embedded in block comments). Some of this metadata gives traceability information for requirements, tests etc.. while other metadata enables me to associate a maturity level with each file. My build system understands this metadata and uses it to inform e.g. the minimum test coverage that it expects on a file-by-file basis. The same metadata is used to ensure that all of the other components that a file references are at the same level of maturity or higher. I also have metadata for each file (partly derived from location in the repository) that gives each file a number which defines it's position in a hierarchy of design elements. The position in the hierarchy helps to indicate what the purpose of the file is. I use this to make a distinction between those core, foundational, stable design elements upon which other design elements may build, and those more peripheral, ephemeral and 'agile' design elements which can be quickly tailored to meet the needs of a client or partner. This means that a (hopefully stable) core API component can be prevented from relying upon a (perhaps less stable) bespoke customer-specific component. It also means that there's more freedom in changing and adapting peripheral designs as you can have confidence that it's stability is not something that is going to be relied upon.
hakanderyal 4y ago
After dealing with that problem and enduring the pain of it for years, I finally switched to C#/.NET. It has the necessary tooling to achieve this and more. Rewriting a lot of things was time well spent rather than trying to tame the dynamic nature of Python and my tendency to overuse it. And I can't believe I'm writing this after all these years evangelizing Python and dynamically typed languages.
- whiskey14 4y ago
  Can you please give a short summary of why I should give C#/.NET a go for my backend services? I've been fighting battles in Python backend services to get a nicely decoupled API, logic and DB layers for a while...but sqlalchelmy, alembic and flask/django/fastAPI are my safety blankets
  - hakanderyal 4y ago
    I was in the same boat. I wanted to switch a few years ago actually, but EF core was missing core features I took for granted in Sqlalchemy. As for the reasons: - Static typing and C# projects makes code organization and refactoring dead easy. - Modern C# doesn’t require that much boilerplate, and has features that allows a developer to speed up development, like Python. - EF Core covers everything I need from SqlAlchemy/Alembic. - LINQ is an awesome way to work with collections. Type safe DB queries comes handy both when developing and refactoring. - ASP.NET covers everything I need from flask/Fastapi. - More speed and lower resource usage is nice. - Being able to use an IDE with it’s full power is nice. My main reason to switch was static typing, and my only requirement was a good ORM comparable to SqlAlchemy. The rest is just bonus.
    - danuker 4y ago
      > - More speed and lower resource usage is nice. > - Being able to use an IDE with it’s full power is nice. In my experience, Visual Studio is much slower when developing. My Python workflow affords a 200ms red-green-refactor loop, while VS is on the order of several seconds. This might not seem like much, but it has a great impact on my engagement, flow, and satisfaction.
yuppiepuppie 4y ago
Title is misleading, it should be "Maintain a clean architecture in FastAPI with dependency rules"
- lyu07282 4y ago
  It has nothing to do with FastAPI, pretty sure sourcery would work with anything.
- rekahrv 4y ago
  Thanks, that's a good point. We thought that a small FastAPI project shows the general concept as well. Do you have suggestions which other examples would be useful?
mikeholler 4y ago
Is there anything similar to this for Java/Kotlin/Gradle?
- Sankozi 4y ago
  There is ArchUnit - https://www.archunit.org/ https://www.archunit.org/
  - mikeholler 4y ago
    Thanks, that looks like exactly what I was looking for.
- naizarak 4y ago
  JPMS allows for exactly this, but that entire project was so poorly implemented that no one uses it.
andrew_eu 4y ago
Before clicking on this, I expected to see import-linter [0] which achieves something very similar but with, in my opinion, a bit less magic. Another solution in a similar spirit is Pants [1], though this is actually a build system which allows you to constrain dependencies between different artifacts (e.g. which modules are allowed to depend on which modules). To Sourcery's credit, their product looks much more in the realm of "developer experience" -- closer to Copilot (or what I understand of it) than to import-linter. Props to them for at least having a page about security [2] and building a solution which doesn't inherently require all of your source code to be shared with a vendor's server. [0] https://github.com/seddonym/import-linter https://github.com/seddonym/import-linter [1] https://www.pantsbuild.org/ https://www.pantsbuild.org/ [2] https://docs.sourcery.ai/Product/Permissions-and-Security/ https://docs.sourcery.ai/Product/Permissions-and-Security/
- memco 4y ago
  Thanks for the additional tools to tackle this problem. We usually don’t have problems with this at work, but I just so happened to discover one today and was dreading the work it will take to sort out how to fix it.
lyu07282 4y ago
Is there something like this for react/jsx? I always wished I could constrain component dependencies across the atom > molecule > organism layers.
neves 4y ago
I've already seem tools like this for other languages, but never seen someone effectively using them. Does anyone here has good or bad experiences with these architecture rule systems?
hbrn 4y ago
This (plus Law of Demeter) is the right way to handle medium-big size projects, though I'm not completely sold on the tooling. I mostly do it manually (yes, it is still doable with dozens of modules since dependency hierarchy doesn't change often). One recommendation I have is to present the hierarchy as DAG. Existing image (https://sourcery.ai/static/05300f06cb847360719e2aa31dc5a31b/8c857/dependencies.png https://sourcery.ai/static/05300f06cb847360719e2aa31dc5a31b/...) doesn't make it very obvious that api is a highest-level module, even though it is clearly stated in the rules.
- rekahrv 4y ago
  Yes, Law of Demeter is exactly what these rules are trying to achieve. :-) Thanks, a DAG is a great recommendation.
- AlphaSite 4y ago
  Import Linter is probably a better choice for Python since it’s free https://pypi.org/project/import-linter/ https://pypi.org/project/import-linter/
  - hbrn 4y ago
    I thought about using it at a small scale, but frankly I find more value in a visual representation, and once I have that I don't want to explicitly blacklist imports: those rules can already be derived from the graph (i.e. any import that introduces a cycle is a violation). Doing it manually allows me the following: 1. I get to define what are the namespaces (domains) that matter, irrespective of the package structure. E.g. import from stripe.api_resources is still a dependency on stripe, not on stripe.api_resources. 2. Work around a bunch of dependency caveats (frameworks like Django do runtime imports and mix high and low level concepts in settings, db foreign keys might inverse your dependencies, etc) 3. Violations are very easy to see: they are cycles in the graph, i.e. arrows pointing upwards. Those are typically design flaws. Though I still allow certain violations because practicality beats purity. 4. Since some violations are allowed, I get to decide how to arrange the graph so that it is more clear what is the flaw and how to address it. I haven't found a good tool that allows me to get all of these. One day I'll have to build it myself.
thundergolfer 4y ago
This is supported in Bazel with package visibility rules. Once you've got that feature as a way to tame a larger and expanding codebase, you'll wonder why it isn't a feature in more systems. https://bazel.build/concepts/visibility https://bazel.build/concepts/visibility
- rekahrv 4y ago
  Thanks a lot for sharing this link. I haven't used Bazel, but this concept of target and load visibility sounds cool. "Once you've got that feature as a way to tame a larger and expanding codebase, you'll wonder why it isn't a feature in more systems." :-)
  - shankr 4y ago
    This has also been recently integrated in pants. https://github.com/pantsbuild/pants/issues/13393 https://github.com/pantsbuild/pants/issues/13393
revskill 4y ago
In Typescript, i normally just allow interface to be dependencies between layers. (API, command line programs,..) -> (Services) -> (Database).
vasili111 4y ago
Am I only person that prefers to use raw SQL over of SQLAlchemy? I do not see any real advantage of using SQLAlchemy over raw SQL if I do not plan (which I do not plan) in future to switch DB engine for the application. Do you see any real advantage of using SQLALchemy over raw SQL queries if you do not plan to switch DB engine for your application in future?
- willseth 4y ago
  It's nice to have all of your db operations in Python and automatically integrated with existing Python tooling. It also makes it easier to refactor, organize, etc. SQLAlchemy comes OOTB with a lot of nice convenience tools and functions, and there's an ecosystem built around it, e.g. Alembic for schema migration. There are some cases like really complex queries where it can get in the way, but overall I find the tradeoffs are easily worth it for the convenience
- dontlaugh 4y ago
  Alchemy makes it quite easy to compose queries, which isn’t possible with SQL. That’s about it.
- gghhzzgghhzz 4y ago
  I use it mostly for reverse engineering a model on top of a legacy database when working on projects to clean and migrate that data. I seen some very legacy database 'designs' and have never failed to model them with a combination of sqlalchemy join mapping, datatype mapping and some object properties in python for cases that are simpler to just express as list comprehensions. You end up with some data quality rules / Transformation logic you can reasonably share with business users. On the Load end I normally do that via sql bulk inserts as using an ORM just adds too much overhead and not enough control.
- piafraus 4y ago
  No, I feel raw SQL is much cleaner and you can see immediately what will be executed, which indexes will be used, etc. No black-box magic.
falcor84 4y ago
>While Python doesn't allow circular dependencies between modules, it won't stop you from introducing circular dependencies between packages. Just to nitpick, while it is a very potent foot-gun, Python absolutely does allow circular dependencies between regular modules; here's a good write-up about this: https://stackabuse.com/python-circular-imports/ https://stackabuse.com/python-circular-imports/
rekahrv 4y ago
A follow-up post: https://sourcery.ai/blog/dependency-architecture/ https://sourcery.ai/blog/dependency-architecture/

139 points by rekahrv 4y ago

Thaxll 4y ago

Your DB / api layer should never touch the same models.

rekahrv 4y ago
Thanks, that's a good point and perhaps a good topic for a future post :-) How to ensure that the API and the db use different models even if those models are in the same package?
- dangets 4y ago
  I struggle with this also, I assume the answer is to not have them in the same package. You can also break the application into separate 'domain', 'infra', 'application' modules as documented in [0] with rules on what dependencies are allowed in each module (e.g. domain should not have db or serialization implementation). The problem is that this does create several adapter layers which adds to the mental complexity. [0] https://learn.microsoft.com/en-us/dotnet/architecture/microservices/microservice-ddd-cqrs-patterns/ddd-oriented-microservice https://learn.microsoft.com/en-us/dotnet/architecture/micros...
inwit 4y ago
It's these kind of rules that mean I'm here wading through 5 layers of exquisitely decoupled nonsense that could be done in a few lines
- Thaxll 4y ago
  Convert function between API and DB model does not sound complicated. Storing the API model in your DB is really a bad idea.
- camgunz 4y ago
  I just listened to the DHH/Kent Beck/Martin Fowler discussion about TDD "damage" and both sides still seemed unconvinced by the end of it, but this exact example came up. It seems like SOA (whether it's DDD or Hexagonal or Clean or w/e) and TDD really push you towards this kind of layer bloat for one reason or another. I'm (maybe obviously) on the SOA-skeptic side, my arguments generally are: - Most apps aren't that big and don't need multiple layers of abstraction (i.e. the ORM and its models are totally fine). If the app starts getting too big for its britches, probably the best thing to do is make it 2 apps (too big: 2 apps is a good slogan here). - Dependency injection and mocks are pretty bad ideas that are only occasionally useful (DHH uses the example of a payments gateway), but mostly push IoC through your whole app and make control flow confusingly backwards. Mocks are always in disrepair, and almost never accurately reflect what they're trying to mock, and thus ironically are big vectors for bugs that make it through testing. - Having tons of unit tests tends to slow eng velocity to a crawl, because they test the parts of the application that aren't the requirements (were these functions called, what's the call signature of this function, was this class instantiated, etc.). Unit tests create a super-fine-grained shadow spec about the lowest level details of your application, and mostly I think they shouldn't ever be committed to a repo. They help during individual development, but then the whole team is stuck maintaining them forever whenever they make changes. They also tend to slow down CI because they're slow and always flaky. - You almost certainly will never need to switch databases, let alone abstract across a database, a message queue, and a web api. It's not worth doing a "repo" abstraction and encapsulating those details. - There are (now) really good libraries for almost anything you want to do. ORMs literally map database entities to domain entities--they just abstract the persistence for you. Sounds like a repo to me! We also have good validation, logging, monitoring, auth/auth etc. built into frameworks and 3rd party services. A lot of the things you might put into other layers or even other services are now neatly packaged into libraries/frameworks you can just use and SaaS things you can just buy, leaving you free to just implement your business logic.
  - hbrn 4y ago
    I mostly agree with you and DHH on that topic, however in my experience reasonably applied SOA/DDD actually shields me from this layering nonsense. When your apps live as a service on the network or as a nicely isolated module in your repo, you no longer have a reason to over-engineer them. You don't need a grandiose architecture that solves every problem, instead you can make local decisions that are good enough in the specific context. Though, admittedly, I found it hard to sell such "inconsistencies" to other tech leaders, most folks aspire to those grandiosities. > If the app starts getting too big for its britches, probably the best thing to do is make it 2 apps That's the argument in favor of SOA, isn't it?
aobdev 4y ago
I hate that these are called models (probably because they extend pydantic’s BaseModel), but if they were called Schema or Serializers would this still be true? Typically what you see in a FastAPI project is a class that parsers the request body, and the same or slightly modified class that serializes the response back out after touching the DB. And this isn’t a new idea, because Flask+Marshmallow and DRF do the exact same thing.
- rekahrv 4y ago
  I've used multiple names for similar packages incl. `models` and `schemas`. :-) Yes, for this example, I picked `models` to follow Pydantic's terminology. IMO, the FastAPI approach you described makes a lot of sense: The "schema" stored in the db and the "schema" returned by the API aren't the same, but they are quite similar. They have many common properties => They can often have a common base class.
  - deleted 4y ago
    [deleted]
marginalia_nu 4y ago
Why not?

w_t_payne 4y ago

I do exactly this in my side project. I have a set of rules which put restrictions on which packages and modules can be included from other packages and modules. For example, a high maturity package is not allowed to depend upon a low maturity package. Similarly, a core library package is not allowed to rely on a package that is specific to a particular product or a particular piece of bespoke development. In this way, much of the potential for circular dependencies is eliminated, and the purpose and internet is clearly communicated. (I don't do this using sourcery though ... I have my own set of rules)

rekahrv 4y ago
That's very cool. Can you tell a bit more about this set of rules? "For example, a high maturity package is not allowed to depend upon a low maturity package. Similarly, a core library package is not allowed to rely on a package that is specific to a particular product or a particular piece of bespoke development." I really like these.
- w_t_payne 4y ago
  I have a general system for representing metadata in source files. (I use YAML documents embedded in block comments). Some of this metadata gives traceability information for requirements, tests etc.. while other metadata enables me to associate a maturity level with each file. My build system understands this metadata and uses it to inform e.g. the minimum test coverage that it expects on a file-by-file basis. The same metadata is used to ensure that all of the other components that a file references are at the same level of maturity or higher. I also have metadata for each file (partly derived from location in the repository) that gives each file a number which defines it's position in a hierarchy of design elements. The position in the hierarchy helps to indicate what the purpose of the file is. I use this to make a distinction between those core, foundational, stable design elements upon which other design elements may build, and those more peripheral, ephemeral and 'agile' design elements which can be quickly tailored to meet the needs of a client or partner. This means that a (hopefully stable) core API component can be prevented from relying upon a (perhaps less stable) bespoke customer-specific component. It also means that there's more freedom in changing and adapting peripheral designs as you can have confidence that it's stability is not something that is going to be relied upon.

hakanderyal 4y ago

After dealing with that problem and enduring the pain of it for years, I finally switched to C#/.NET. It has the necessary tooling to achieve this and more. Rewriting a lot of things was time well spent rather than trying to tame the dynamic nature of Python and my tendency to overuse it. And I can't believe I'm writing this after all these years evangelizing Python and dynamically typed languages.

whiskey14 4y ago
Can you please give a short summary of why I should give C#/.NET a go for my backend services? I've been fighting battles in Python backend services to get a nicely decoupled API, logic and DB layers for a while...but sqlalchelmy, alembic and flask/django/fastAPI are my safety blankets
- hakanderyal 4y ago
  I was in the same boat. I wanted to switch a few years ago actually, but EF core was missing core features I took for granted in Sqlalchemy. As for the reasons: - Static typing and C# projects makes code organization and refactoring dead easy. - Modern C# doesn’t require that much boilerplate, and has features that allows a developer to speed up development, like Python. - EF Core covers everything I need from SqlAlchemy/Alembic. - LINQ is an awesome way to work with collections. Type safe DB queries comes handy both when developing and refactoring. - ASP.NET covers everything I need from flask/Fastapi. - More speed and lower resource usage is nice. - Being able to use an IDE with it’s full power is nice. My main reason to switch was static typing, and my only requirement was a good ORM comparable to SqlAlchemy. The rest is just bonus.
  - danuker 4y ago
    > - More speed and lower resource usage is nice. > - Being able to use an IDE with it’s full power is nice. In my experience, Visual Studio is much slower when developing. My Python workflow affords a 200ms red-green-refactor loop, while VS is on the order of several seconds. This might not seem like much, but it has a great impact on my engagement, flow, and satisfaction.

yuppiepuppie 4y ago

Title is misleading, it should be "Maintain a clean architecture in FastAPI with dependency rules"

lyu07282 4y ago
It has nothing to do with FastAPI, pretty sure sourcery would work with anything.
rekahrv 4y ago
Thanks, that's a good point. We thought that a small FastAPI project shows the general concept as well. Do you have suggestions which other examples would be useful?

mikeholler 4y ago

Is there anything similar to this for Java/Kotlin/Gradle?

Sankozi 4y ago
There is ArchUnit - https://www.archunit.org/ https://www.archunit.org/
- mikeholler 4y ago
  Thanks, that looks like exactly what I was looking for.
naizarak 4y ago
JPMS allows for exactly this, but that entire project was so poorly implemented that no one uses it.

andrew_eu 4y ago

Before clicking on this, I expected to see import-linter [0] which achieves something very similar but with, in my opinion, a bit less magic. Another solution in a similar spirit is Pants [1], though this is actually a build system which allows you to constrain dependencies between different artifacts (e.g. which modules are allowed to depend on which modules). To Sourcery's credit, their product looks much more in the realm of "developer experience" -- closer to Copilot (or what I understand of it) than to import-linter. Props to them for at least having a page about security [2] and building a solution which doesn't inherently require all of your source code to be shared with a vendor's server. [0] https://github.com/seddonym/import-linter https://github.com/seddonym/import-linter [1] https://www.pantsbuild.org/ https://www.pantsbuild.org/ [2] https://docs.sourcery.ai/Product/Permissions-and-Security/ https://docs.sourcery.ai/Product/Permissions-and-Security/

memco 4y ago
Thanks for the additional tools to tackle this problem. We usually don’t have problems with this at work, but I just so happened to discover one today and was dreading the work it will take to sort out how to fix it.

lyu07282 4y ago

Is there something like this for react/jsx? I always wished I could constrain component dependencies across the atom > molecule > organism layers.

neves 4y ago

I've already seem tools like this for other languages, but never seen someone effectively using them. Does anyone here has good or bad experiences with these architecture rule systems?

hbrn 4y ago

This (plus Law of Demeter) is the right way to handle medium-big size projects, though I'm not completely sold on the tooling. I mostly do it manually (yes, it is still doable with dozens of modules since dependency hierarchy doesn't change often). One recommendation I have is to present the hierarchy as DAG. Existing image (https://sourcery.ai/static/05300f06cb847360719e2aa31dc5a31b/8c857/dependencies.png https://sourcery.ai/static/05300f06cb847360719e2aa31dc5a31b/...) doesn't make it very obvious that api is a highest-level module, even though it is clearly stated in the rules.

rekahrv 4y ago
Yes, Law of Demeter is exactly what these rules are trying to achieve. :-) Thanks, a DAG is a great recommendation.
AlphaSite 4y ago
Import Linter is probably a better choice for Python since it’s free https://pypi.org/project/import-linter/ https://pypi.org/project/import-linter/
- hbrn 4y ago
  I thought about using it at a small scale, but frankly I find more value in a visual representation, and once I have that I don't want to explicitly blacklist imports: those rules can already be derived from the graph (i.e. any import that introduces a cycle is a violation). Doing it manually allows me the following: 1. I get to define what are the namespaces (domains) that matter, irrespective of the package structure. E.g. import from stripe.api_resources is still a dependency on stripe, not on stripe.api_resources. 2. Work around a bunch of dependency caveats (frameworks like Django do runtime imports and mix high and low level concepts in settings, db foreign keys might inverse your dependencies, etc) 3. Violations are very easy to see: they are cycles in the graph, i.e. arrows pointing upwards. Those are typically design flaws. Though I still allow certain violations because practicality beats purity. 4. Since some violations are allowed, I get to decide how to arrange the graph so that it is more clear what is the flaw and how to address it. I haven't found a good tool that allows me to get all of these. One day I'll have to build it myself.

thundergolfer 4y ago

This is supported in Bazel with package visibility rules. Once you've got that feature as a way to tame a larger and expanding codebase, you'll wonder why it isn't a feature in more systems. https://bazel.build/concepts/visibility https://bazel.build/concepts/visibility

rekahrv 4y ago
Thanks a lot for sharing this link. I haven't used Bazel, but this concept of target and load visibility sounds cool. "Once you've got that feature as a way to tame a larger and expanding codebase, you'll wonder why it isn't a feature in more systems." :-)
- shankr 4y ago
  This has also been recently integrated in pants. https://github.com/pantsbuild/pants/issues/13393 https://github.com/pantsbuild/pants/issues/13393

revskill 4y ago

In Typescript, i normally just allow interface to be dependencies between layers. (API, command line programs,..) -> (Services) -> (Database).

vasili111 4y ago

Am I only person that prefers to use raw SQL over of SQLAlchemy? I do not see any real advantage of using SQLAlchemy over raw SQL if I do not plan (which I do not plan) in future to switch DB engine for the application. Do you see any real advantage of using SQLALchemy over raw SQL queries if you do not plan to switch DB engine for your application in future?

willseth 4y ago
It's nice to have all of your db operations in Python and automatically integrated with existing Python tooling. It also makes it easier to refactor, organize, etc. SQLAlchemy comes OOTB with a lot of nice convenience tools and functions, and there's an ecosystem built around it, e.g. Alembic for schema migration. There are some cases like really complex queries where it can get in the way, but overall I find the tradeoffs are easily worth it for the convenience
dontlaugh 4y ago
Alchemy makes it quite easy to compose queries, which isn’t possible with SQL. That’s about it.
gghhzzgghhzz 4y ago
I use it mostly for reverse engineering a model on top of a legacy database when working on projects to clean and migrate that data. I seen some very legacy database 'designs' and have never failed to model them with a combination of sqlalchemy join mapping, datatype mapping and some object properties in python for cases that are simpler to just express as list comprehensions. You end up with some data quality rules / Transformation logic you can reasonably share with business users. On the Load end I normally do that via sql bulk inserts as using an ORM just adds too much overhead and not enough control.
piafraus 4y ago
No, I feel raw SQL is much cleaner and you can see immediately what will be executed, which indexes will be used, etc. No black-box magic.

falcor84 4y ago

>While Python doesn't allow circular dependencies between modules, it won't stop you from introducing circular dependencies between packages. Just to nitpick, while it is a very potent foot-gun, Python absolutely does allow circular dependencies between regular modules; here's a good write-up about this: https://stackabuse.com/python-circular-imports/ https://stackabuse.com/python-circular-imports/

rekahrv 4y ago

A follow-up post: https://sourcery.ai/blog/dependency-architecture/ https://sourcery.ai/blog/dependency-architecture/