5 ms·

Some parts of this book are extremely useful, especially when it's talking about concepts that are more general than Python or any other specific language -- su

Some parts of this book are extremely useful, especially when it's talking about concepts that are more general than Python or any other specific language -- such as event-driven architecture, commands, CQRS etc. That being said, I have a number issues with other parts of it, and I have seen how dangerous it can be when inexperienced developers take it as a gospel and try to implement everything at once (which is a common problem with any collection of design patterns like this. For example, repository is a helpful pattern in general; but in many cases, including the examples in the book itself, it is a huge overkill that adds complexity with very little benefit. Even more so as they're using SQLAlchemy, which is a "repository" in its own right (or, more precisely, a relational database abstraction layer with an ORM added on top). Similarly, service layers and unit of work are useful when you have complex applications that cover multiple complex use cases; but in a system consisting of small services with narrow responsibilities they quickly become overly bloated using this pattern. And don't even get me started with dependency injection in Python. The essential thing about design patterns is that they're tools like any other, and the developers should understand when to use them, and even more importantly when not to use them. This book has some advice in that direction, but in my opinion it should be more prominent and placed upfront rather at the end of each chapter.

kelafoja 2y ago
Could you explain how repository pattern is a "huge overkill that adds complexity with very little benefit"? I find it a very light-weight pattern and would recommend to always use it when database access is needed, to clearly separate concerns. In the end, it's just making sure that all database access for a specific entity all goes through one point (the repository for that entity). Inside the repository, you can do whatever you want (run queries yourself, use ORM, etc). A lot of the stuff written in the article under the section Repository pattern has very little to do with the pattern, and much more to do with all sorts of Python, Django, and SQLAlchemy details.
- unculture 2y ago
  Repository pattern is useful if you really feel like you're going to need to switch out your database layer for something else at some point in the future, but I've literally never seen this happen in my career ever. Otherwise, it's just duplicate code you have to write.
  - kelafoja 2y ago
    What is the alternative that you use, how do you provide data access in a clean, separated, maintainable way? I have seen it a lot in my career, and have used it a lot. I've never used it in any situation to switch out a database layer for something else. It seems like we have very different careers. I also don't really see how it duplicates code. At the basic level, it's practically nothing more than putting database access code in one place rather than all over the place.
    - BerislavLopac 2y ago
      OK, let's first define some things. What we are talking about is a "transformation" or "mapper" layer isolating your domain entities from the persistence. If this is what we call "Repository" then yes, I absolutely agree with you -- this is the right approach to this problem. But if the "Repository pattern" means a complex structure of abstract and concrete classes and inheritance trees -- as I have usually seen it implemented -- then it is usually an overkill and rarely a good idea.
      - kelafoja 2y ago
        Thanks. In my mind, anything about complex structures of (abstract) classes and/or inheritance trees has nothing to do with a Repository pattern. As I understand it, Repository pattern is basically a generalization of the Data Access Object (DAO) pattern, and sometimes treated synonymously. The way I mean it and implement it, is basically for each entity have a separate class to provide the database access. E.g. you have a Person (not complex at all, simply a value object) and a PersonRepository to get, update, and delete Person objects. Then based on the complexity and scope of the project, Person either 1-to-1 maps to a e.g. a database table or stored object/document, or it is a somewhat more complex object in the business domain and the repository could be doing a little bit more work to fetch and construct it (e.g. perhaps some joins or more than 1 query for some data).
- ryan-duve 2y ago
  TL;DR, YAGNI I had a former boss who strongly pushed my team to use the repository pattern for a microservice. The team wanted to try it out since it was new to us and, like the other commenters are saying, it worked but we never actually needed it. So it just sat there as another layer of abstraction, more code, more tests, and nothing benefited from it. Anecdotally, the project was stopped after nine months because it took too long. The decision to use the repository pattern wasn't the straw that broke the camel's back, but I think using patterns that were more complicated than the usecase required was at the heart of it.
  - kelafoja 2y ago
    Could you give me some insights what the possible alternative was that you would have rather seen? I am either now learning that the Repository pattern is something different than what I understand it to be, or there is misunderstanding here. I cannot understand how (basically) tucking away database access code in a repository can lead to complicated code, long development times, and the entire project failing.
    - esafak 2y ago
      It doesn't. Use it; it's easy.
    - ddejohn 2y ago
      Your understanding of the repository pattern is correct. It's the other people in this thread that seem to have misunderstood it and/or implemented it incorrectly. I use the repository pattern in virtually every service (when appropriate) and it's incredibly simple, easy to test and document, and easy to teach to coworkers. Because most of our services use the repository pattern, we can jump into any project we're not familiar with and immediately have the lay of the land, knowing where to go to find business logic or make modifications. One thing to note -- you stated in another comment that the repository pattern is just for database access, but this isn't really true. You can use the repository pattern for any type of service that requires fetching data from some other location or multiple locations -- whether that's a database, another HTTP API, a plain old file system, a gRPC server, an ftp server, a message queue, an email service... whatever. This has been hugely helpful for me as one of the things my company does is aggregate data from a lot of other APIs (whois records, stuff of that nature). Multiple times we've had to switch providers due to contract issues or because we found something better/cheaper. Being able to swap out implementations was incredibly helpful because the business logic layer and its unit tests didn't need to be touched at all. Before I started my current role, we had been using kafka for message queues. There was a huge initiative to switch over to rabbit and it was extremely painful ripping out all the kafka stuff and replacing it with rabbit stuff and it took forever and we still have issues with how the switch was executed to this day, years later. If we'd been using the repository pattern, the switch would've been a piece of cake.
- mosburger 2y ago
  I find repository pattern useful for testing ... it's a lot easier to mock a repository than to mock stuff that SQLalchemy might need to do.
  - williamdclt 2y ago
    I rarely mock a repository. Mocking the database is nice for unit-testing, it's also a lot faster than using a real DB, but the DB and DB-application interface are some of the hottest spots for bugs: using a real DB (same engine as prod) gives me a whole lot more confidence that my code actually works. It's probably the thing I'm least likely to mock out, despite making tests more difficult to write and quite a bit slowerq
- bulatb 2y ago
  In theory it's a nice abstraction, and the benefit is clear. In practice, your repository likely ends up forwarding its arguments one-for-one to SQLAlchemy's select() or session.query(). That's aside from their particular example of SQLAlchemy sessions, which is extra weird because a Session is already a repository, more or less. I mean, sure, there's a difference between your repository for your things and types you might consider foreign, in theory, but how theoretical are we going to get? For what actual gain? How big of an app are we talking? You could alias Repository = Session, or define a simple protocol with stubs for some of Session's methods, just for typing, and you'd get the same amount of theoretical decoupling with no extra layer. If you want to test without a database, don't bind your models to a session. If you want to use a session anyway but still not touch the database, replace your Session's scopefunc and your tested code will never know the difference. It's not a convincing example. Building your repository layer over theirs, admittedly you stop the Query type from leaking out. But then you implement essentially the Query interface in little bits for use in different layers, just probably worse, and lacking twenty years of testing.
  - kelafoja 2y ago
    Thanks, that makes a lot of sense. I don't have a whole bunch of experience with SQLAlchemy itself. In general, I prefer not to use ORMs but just write queries and map the results into value objects. That work I would put into a Repository. Also in my opinion it's important to decouple the database structure from the domain model in the code. One might have a Person type which is constructed by getting data from 3 tables. A Repository class could do that nicely: maybe run a join query and a separate query, combine the results together, and return the Person object. ORMs usually tightly couple with the DB schema, which might create the risk of coupling the rest of the application as well (again, I don't know how flexible SQLAlchemy is in this). There could be some value in hiding SQLAlchemy, in case one would ever like to replace it with a better alternative. I don't have enough experience with Python to understand if that ever will be the case though. All in all, trade-offs are always important to consider. A tiny microservice consisting of a few functions: just do whatever. A growing modulith with various evolving domains which have not been fully settled yet: put some effort into decoupling and separating concerns.
- deergomoo 2y ago
  In my experience, both SQL and real-world database schema are each complex enough beasts that to ensure everything is fetched reasonably optimally, you either need tons of entity-specific (i.e. not easily interface-able) methods for every little use case, or you need to expose some sort of builder, at which point why not just use the query builder you're almost certainly already calling underneath? Repository patterns are fine for CRUD but don't really stretch to those endpoints where you really need the query with the two CTEs and the four joins onto a query selecting from another query based on the output of a window function.
SaturnIC 2y ago
> That being said, I have a number issues with other parts of it, and I have seen how dangerous it can be when inexperienced developers take it as a gospel and try to implement everything at once (which is a common problem with any collection of design patterns like this. Robert Martin is one of those examples, he did billions in damages by brainwashing inexperienced developers with his gaslighting garbage like "Clean Code". Software engineering is not a hard science so there is almost never a silver bullet, everything is trade-offs, so people that claim to know the one true way are subcriminal psychopaths or noobs
- 0xpgm 2y ago
  Clean code has lots of useful tips and techniques. When people are criticizing it they pick a concept from one or two pages out the hundreds and use it to dismiss the whole book. This is a worse mistake than introducing concepts that may be foot guns in some situations. Becoming an experienced engineer is learning how, when and where to apply tools from your toolkit.
jbs789 2y ago
This was my takeaway too. It’s interesting to see the patterns. It would be helpful for some guidance upfront around when the situations in which they are most useful to implement. If a pattern is a tool, then steering me towards when it’s used or best avoided would be helpful. I do appreciate that the pros and cons sections get to this point, so perhaps it’s just ordering and emphasis. That said, having built a small web app to enable a new business, and learning python along the way to get there, this provided me with some ideas for patterns I could implement to simplify things (but others I think I’ll avoid).
ryan-duve 2y ago
> And don't even get me started with dependency injection in Python. Could I get you started? Or could you point me to a place to get myself started? I primarily code in Python and I've found dependency injection, by which I mean giving a function all the inputs it needs to calculate via parameters, is a principle worth designing projects around.
- deleted 2y ago
  [deleted]
- d0mine 2y ago
  Here’s 1% that gives 50% of result. Replace: class C: def __init__(self): self.foo = ConcreteFoo() with: class C: def __init__(self, foo: SupportsFoo): self.foo = foo where SupportsFoo is a Protocol. That’s it.
wormlord 2y ago
> I have seen how dangerous it can be when inexperienced developers take it as a gospel and try to implement everything at once This book explicitly tells you not to do this. > Similarly, service layers and unit of work are useful when you have complex applications that cover multiple complex use cases; but in a system consisting of small services with narrow responsibilities they quickly become overly bloated using this pattern. And don't even get me started with dependency injection in Python. I have found service layers and DI really helpful for writing functional programs. I have some complex image-processing scripts in Python that I can use as plug-ins with a distributed image processing service in Celery. Service layer and DI just takes code from: ```python dependency.do_thing(params) ``` To: ```python do_thing(dependency, params) ``` Which ends up being a lot more testable. I can run image processing tasks in a live deployment with all of their I/O mocked, or I can run real image processing tasks on a mocked version of Celery. This lets me test all my different functions end-to-end before I ever do a full deploy. Also using the Result type with service layer has helped me propagate relevant error information back to the web client without crashing the program, since the failure modes are all handled in their specific service layer function.
- wbc 2y ago
  may I ask how you're mocking Celery to test? the two main methods I've seen are to run tasks eagerly, or test the underlying function and avoid test Celery .delay/etc at all
  - wormlord 2y ago
    I just rolled my own mocked Celery objects. I have mocked Groups, Chords, Chains, and Signatures, mocked Celery backend, and mocked dispatch of tasks. Everything runs eagerly because it's all just running locally in the same thread, but the workflow still runs properly-- the output of one task is fed into the next task, the tasks are updated, etc. I actually pass Celery and the functions like `signature`, `chain`, etc as a tuple into my service layer functions. It's mostly just to test that the piping of the workflow is set up correctly so I don't find out that my args are swapped later during integration tests.

kelafoja 2y ago

Could you explain how repository pattern is a "huge overkill that adds complexity with very little benefit"? I find it a very light-weight pattern and would recommend to always use it when database access is needed, to clearly separate concerns. In the end, it's just making sure that all database access for a specific entity all goes through one point (the repository for that entity). Inside the repository, you can do whatever you want (run queries yourself, use ORM, etc). A lot of the stuff written in the article under the section Repository pattern has very little to do with the pattern, and much more to do with all sorts of Python, Django, and SQLAlchemy details.

unculture 2y ago
Repository pattern is useful if you really feel like you're going to need to switch out your database layer for something else at some point in the future, but I've literally never seen this happen in my career ever. Otherwise, it's just duplicate code you have to write.
- kelafoja 2y ago
  What is the alternative that you use, how do you provide data access in a clean, separated, maintainable way? I have seen it a lot in my career, and have used it a lot. I've never used it in any situation to switch out a database layer for something else. It seems like we have very different careers. I also don't really see how it duplicates code. At the basic level, it's practically nothing more than putting database access code in one place rather than all over the place.
  - BerislavLopac 2y ago
    OK, let's first define some things. What we are talking about is a "transformation" or "mapper" layer isolating your domain entities from the persistence. If this is what we call "Repository" then yes, I absolutely agree with you -- this is the right approach to this problem. But if the "Repository pattern" means a complex structure of abstract and concrete classes and inheritance trees -- as I have usually seen it implemented -- then it is usually an overkill and rarely a good idea.
    - kelafoja 2y ago
      Thanks. In my mind, anything about complex structures of (abstract) classes and/or inheritance trees has nothing to do with a Repository pattern. As I understand it, Repository pattern is basically a generalization of the Data Access Object (DAO) pattern, and sometimes treated synonymously. The way I mean it and implement it, is basically for each entity have a separate class to provide the database access. E.g. you have a Person (not complex at all, simply a value object) and a PersonRepository to get, update, and delete Person objects. Then based on the complexity and scope of the project, Person either 1-to-1 maps to a e.g. a database table or stored object/document, or it is a somewhat more complex object in the business domain and the repository could be doing a little bit more work to fetch and construct it (e.g. perhaps some joins or more than 1 query for some data).
ryan-duve 2y ago
TL;DR, YAGNI I had a former boss who strongly pushed my team to use the repository pattern for a microservice. The team wanted to try it out since it was new to us and, like the other commenters are saying, it worked but we never actually needed it. So it just sat there as another layer of abstraction, more code, more tests, and nothing benefited from it. Anecdotally, the project was stopped after nine months because it took too long. The decision to use the repository pattern wasn't the straw that broke the camel's back, but I think using patterns that were more complicated than the usecase required was at the heart of it.
- kelafoja 2y ago
  Could you give me some insights what the possible alternative was that you would have rather seen? I am either now learning that the Repository pattern is something different than what I understand it to be, or there is misunderstanding here. I cannot understand how (basically) tucking away database access code in a repository can lead to complicated code, long development times, and the entire project failing.
  - esafak 2y ago
    It doesn't. Use it; it's easy.
  - ddejohn 2y ago
    Your understanding of the repository pattern is correct. It's the other people in this thread that seem to have misunderstood it and/or implemented it incorrectly. I use the repository pattern in virtually every service (when appropriate) and it's incredibly simple, easy to test and document, and easy to teach to coworkers. Because most of our services use the repository pattern, we can jump into any project we're not familiar with and immediately have the lay of the land, knowing where to go to find business logic or make modifications. One thing to note -- you stated in another comment that the repository pattern is just for database access, but this isn't really true. You can use the repository pattern for any type of service that requires fetching data from some other location or multiple locations -- whether that's a database, another HTTP API, a plain old file system, a gRPC server, an ftp server, a message queue, an email service... whatever. This has been hugely helpful for me as one of the things my company does is aggregate data from a lot of other APIs (whois records, stuff of that nature). Multiple times we've had to switch providers due to contract issues or because we found something better/cheaper. Being able to swap out implementations was incredibly helpful because the business logic layer and its unit tests didn't need to be touched at all. Before I started my current role, we had been using kafka for message queues. There was a huge initiative to switch over to rabbit and it was extremely painful ripping out all the kafka stuff and replacing it with rabbit stuff and it took forever and we still have issues with how the switch was executed to this day, years later. If we'd been using the repository pattern, the switch would've been a piece of cake.
mosburger 2y ago
I find repository pattern useful for testing ... it's a lot easier to mock a repository than to mock stuff that SQLalchemy might need to do.
- williamdclt 2y ago
  I rarely mock a repository. Mocking the database is nice for unit-testing, it's also a lot faster than using a real DB, but the DB and DB-application interface are some of the hottest spots for bugs: using a real DB (same engine as prod) gives me a whole lot more confidence that my code actually works. It's probably the thing I'm least likely to mock out, despite making tests more difficult to write and quite a bit slowerq
bulatb 2y ago
In theory it's a nice abstraction, and the benefit is clear. In practice, your repository likely ends up forwarding its arguments one-for-one to SQLAlchemy's select() or session.query(). That's aside from their particular example of SQLAlchemy sessions, which is extra weird because a Session is already a repository, more or less. I mean, sure, there's a difference between your repository for your things and types you might consider foreign, in theory, but how theoretical are we going to get? For what actual gain? How big of an app are we talking? You could alias Repository = Session, or define a simple protocol with stubs for some of Session's methods, just for typing, and you'd get the same amount of theoretical decoupling with no extra layer. If you want to test without a database, don't bind your models to a session. If you want to use a session anyway but still not touch the database, replace your Session's scopefunc and your tested code will never know the difference. It's not a convincing example. Building your repository layer over theirs, admittedly you stop the Query type from leaking out. But then you implement essentially the Query interface in little bits for use in different layers, just probably worse, and lacking twenty years of testing.
- kelafoja 2y ago
  Thanks, that makes a lot of sense. I don't have a whole bunch of experience with SQLAlchemy itself. In general, I prefer not to use ORMs but just write queries and map the results into value objects. That work I would put into a Repository. Also in my opinion it's important to decouple the database structure from the domain model in the code. One might have a Person type which is constructed by getting data from 3 tables. A Repository class could do that nicely: maybe run a join query and a separate query, combine the results together, and return the Person object. ORMs usually tightly couple with the DB schema, which might create the risk of coupling the rest of the application as well (again, I don't know how flexible SQLAlchemy is in this). There could be some value in hiding SQLAlchemy, in case one would ever like to replace it with a better alternative. I don't have enough experience with Python to understand if that ever will be the case though. All in all, trade-offs are always important to consider. A tiny microservice consisting of a few functions: just do whatever. A growing modulith with various evolving domains which have not been fully settled yet: put some effort into decoupling and separating concerns.
deergomoo 2y ago
In my experience, both SQL and real-world database schema are each complex enough beasts that to ensure everything is fetched reasonably optimally, you either need tons of entity-specific (i.e. not easily interface-able) methods for every little use case, or you need to expose some sort of builder, at which point why not just use the query builder you're almost certainly already calling underneath? Repository patterns are fine for CRUD but don't really stretch to those endpoints where you really need the query with the two CTEs and the four joins onto a query selecting from another query based on the output of a window function.

SaturnIC 2y ago

> That being said, I have a number issues with other parts of it, and I have seen how dangerous it can be when inexperienced developers take it as a gospel and try to implement everything at once (which is a common problem with any collection of design patterns like this. Robert Martin is one of those examples, he did billions in damages by brainwashing inexperienced developers with his gaslighting garbage like "Clean Code". Software engineering is not a hard science so there is almost never a silver bullet, everything is trade-offs, so people that claim to know the one true way are subcriminal psychopaths or noobs

0xpgm 2y ago
Clean code has lots of useful tips and techniques. When people are criticizing it they pick a concept from one or two pages out the hundreds and use it to dismiss the whole book. This is a worse mistake than introducing concepts that may be foot guns in some situations. Becoming an experienced engineer is learning how, when and where to apply tools from your toolkit.

jbs789 2y ago

This was my takeaway too. It’s interesting to see the patterns. It would be helpful for some guidance upfront around when the situations in which they are most useful to implement. If a pattern is a tool, then steering me towards when it’s used or best avoided would be helpful. I do appreciate that the pros and cons sections get to this point, so perhaps it’s just ordering and emphasis. That said, having built a small web app to enable a new business, and learning python along the way to get there, this provided me with some ideas for patterns I could implement to simplify things (but others I think I’ll avoid).

ryan-duve 2y ago

> And don't even get me started with dependency injection in Python. Could I get you started? Or could you point me to a place to get myself started? I primarily code in Python and I've found dependency injection, by which I mean giving a function all the inputs it needs to calculate via parameters, is a principle worth designing projects around.

deleted 2y ago
[deleted]
d0mine 2y ago
Here’s 1% that gives 50% of result. Replace: class C: def __init__(self): self.foo = ConcreteFoo() with: class C: def __init__(self, foo: SupportsFoo): self.foo = foo where SupportsFoo is a Protocol. That’s it.

wormlord 2y ago

> I have seen how dangerous it can be when inexperienced developers take it as a gospel and try to implement everything at once This book explicitly tells you not to do this. > Similarly, service layers and unit of work are useful when you have complex applications that cover multiple complex use cases; but in a system consisting of small services with narrow responsibilities they quickly become overly bloated using this pattern. And don't even get me started with dependency injection in Python. I have found service layers and DI really helpful for writing functional programs. I have some complex image-processing scripts in Python that I can use as plug-ins with a distributed image processing service in Celery. Service layer and DI just takes code from: ```python dependency.do_thing(params) ``` To: ```python do_thing(dependency, params) ``` Which ends up being a lot more testable. I can run image processing tasks in a live deployment with all of their I/O mocked, or I can run real image processing tasks on a mocked version of Celery. This lets me test all my different functions end-to-end before I ever do a full deploy. Also using the Result type with service layer has helped me propagate relevant error information back to the web client without crashing the program, since the failure modes are all handled in their specific service layer function.

wbc 2y ago
may I ask how you're mocking Celery to test? the two main methods I've seen are to run tasks eagerly, or test the underlying function and avoid test Celery .delay/etc at all
- wormlord 2y ago
  I just rolled my own mocked Celery objects. I have mocked Groups, Chords, Chains, and Signatures, mocked Celery backend, and mocked dispatch of tasks. Everything runs eagerly because it's all just running locally in the same thread, but the workflow still runs properly-- the output of one task is fed into the next task, the tasks are updated, etc. I actually pass Celery and the functions like `signature`, `chain`, etc as a tuple into my service layer functions. It's mostly just to test that the piping of the workflow is set up correctly so I don't find out that my args are swapped later during integration tests.