10 ms·

Keep Your ORM Out of my Controller

37 points by mononcqc 16y ago

sofuture 16y ago
Using an ORM does buy you some abstraction, though. You get to worry about your business objects and fields instead of your tables and columns. (Depending on the language/situation this can be a big deal) You're right that it doesn't provide all the abstraction you probably want, but it easily gives you a toehold on which to build that abstraction. Doing it from scratch, by hand is often a much worse place to start.
- mononcqc 16y ago
  I agree. Maybe I should have been clearer on my views. ORMs are a great tool; the way they are used with the leaky abstraction is what I have a problem with. Restricted to the model layer, they're entirely worth it and provide lots of help.
hippich 16y ago
ORM is not always Model. ORM is tool to build Model. If queries become to SQLish, add another layer on top of ORM with methods to retrieve data which encapsulate all SQLish methods and provide nice and consistent interface.
mgkimsal 16y ago
"This means that we're in the controller but we still have to grasp what the hell is going on in the database. The abstraction is leaking!" How many people JUST work in controllers and can remain oblivious to what's going on in the DB? Just because you move the ORM or any DB call to a different area of the code doesn't mean you shouldn't have a grasp of what's going on. If I wrapped all ORM calls in some service layer classes, that simply means that now when I'm looking at my service layer classes I "still have to grasp what the hell is going on in the database. The abstraction is leaking!". Doesn't it?
- mononcqc 16y ago
  Try renaming a table column. Which one of the two has more impact on the code? Of course the same programmers can work from the model up to the view, but the idea is to reduce the impact of changes in lower layers, and help with testing. The example of the database info required is used to illustrate the leak. It's not a horror in itself.
teilo 16y ago
The problem is not the ORM. Anyone who is writing such complex queries directly in the controller needs to re-think how they have designed their site. Specificity like this belongs in the View and the DB itself, not in the Controller OR the Model. Adding umpteen custom getters to the Model is not helping matters. It may be easier to test, but it is just as unwieldy and difficult to maintain. It is the wrong solution to the problem. These things need to be parametized in some way and made generic. In Django you would likely do this with Q objects, and a front-end that lets you filter down your results on-the-fly, and save the resulting query to the DB for recall later. This is even easier to test, and prevents your Models from getting cluttered with functions which are far too specific.
- mononcqc 16y ago
  I didn't know about Q objects before. What are their capabilities regarding, say, moving from an SQL database to a web-service or a third party application? I mean, this is the kind of stuff I have to do surprisingly often when maintaining legacy applications and putting them up to date. Your data model ends up changing or just not supporting all the same idioms again. You might need to join some info coming from your database without half of it coming from a RPC. I've found such changes in the foundations of where data you get is to cause a whole damn lot of problems on all the layers above. I haven't figured out a better way than wrapping lots of it in fat models, but if Q objects have something that can help with that I'd be very happy to learn about it.
  - teilo 16y ago
    Q objects are a way to represent complex SQL in a progressive fashion that would be unwieldy or impossible with the standard Django ORM chaining technique. It really doesn't address your use case.
aphyr 16y ago
I run into this question frequently: how much logic belongs in the controller, and how much in the model. The difficulty, I think, is that every method you add to the model requires a name, and the profusion of names leads to complexity and poorly named methods. An alternative is to collapse similar methods (for example, two queries differing only by "order by") into one, and use a parameter or function argument to distinguish the cases. But this pushes ORM-specific logic back up into the controller--only now with an extra layer of indirection for the programmer to remember how to use. My approach has been to define methods on the model only when they are used in more than one place or have a concise, meaningful name. Introducing the abstraction later rarely requires modifying several aspects of the application, and Sequel's def_dataset_method/chaining approach make this relatively painless.
- Xurinos 16y ago
  > how much logic belongs in the controller, and how much in the model I always thought this was really simple (at least based on the original Xerox definition of MVC): Controller transforms inputs/parameters as needed. Controller can use Model to do this. Controller creates/manages views and sends latest data to the Views. And a confusion-reducer: Controller is not between Model and Views. Views have direct access to Model. It ONLY uses Model to figure out the situation. I have found these clear separations helpful in a variety of applications. I have also found that people that apply MVC to the web confuse themselves, too. On the server side, your View is what you are returning to the browser. JavaScript is part of the View of the server. It has its own MVC when it is run in the browser environment, and the browser has its own MVC. It is perfectly acceptable to have interlocking components, each with their own MVC.
  - mononcqc 16y ago
    A note: The architecture model where the controller sits between the model and view is instead called 'Model-View-Adapter (http://en.wikipedia.org/wiki/Model-view-adapter http://en.wikipedia.org/wiki/Model-view-adapter), where the controller (or adapter) is doing the mediation between the model and the view. It supposedly allows for a stronger separation of concerns.
deleted 16y ago
[deleted]
Groxx 16y ago
For large-scale projects, this makes a lot of sense. Without another layer of abstraction to keep database-content-specific code out of your controllers, you end up with: db -> orm -> 100,000 flat-level queries tied directly to the format returned by the orm With abstraction like they suggest, you can achieve: db -> orm -> [your organization method here] -> your never-changing data you requested Significantly better if you're working at that scale, and if you ever need to change your DB/ORM. In a situation like that, I wholly agree: there should be zero ORM-tied code in your controller. BUT: tutorials are rarely-if-ever for people working at that scale. You should have some grasp of what you're doing if you're attacking a problem of that scale, or at least be beyond the tutorial stage. If tutorials addressed this, they'd remind me of most beginning stuff for Java programming: subclass this, override main, subclass that, subclass that, subclass that, build a polymorphic inheritance structure, print "hello world". Easy! Totally opaque until you understand the underlying organization of Java, and it tends to distract newcomers from things they want to achieve, by bogging them down in things should do when they have no comprehension of why. Once you understand why, it's merely annoying; before that, you really can't know if you're supposed to use an interface, inherit from Object or Array or GenericArrayListWithBellsAndWhistlesRadixSorted, or just code the damn thing by hand. ORM tutorials are for coding the damn thing by hand, a necessary starting point for building that scalable abstracted structure, and the foundation of it. Not for helping you build a 3000-table, 5-billion-row, 10,000-request-per-second Google-killer.
misterbwong 16y ago
I think the author needs to make a clearer distinction between business logic and view logic. I'm an advocate of creating a view model layer above the traditional model containing things like sorting and field suppression, etc. The app stack might look like this: DB -> ORM -> BizLogic -> ViewLogic -> View This has the effect of decoupling actual business logic (i.e. get me every british author) from view logic (i.e. sort them by name in my table) and allows the design to be more flexible in the future.
pvg 16y ago
That's an impressively verbose and obfuscated way to re-state the very basic and completely independent (of ORM, Django, Python, etc) notion that if you have common code you should abstract it into a separate function.
mattmcknight 16y ago
I think we call these Named Scopes: http://api.rubyonrails.org/classes/ActiveRecord/NamedScope/ClassMethods.html http://api.rubyonrails.org/classes/ActiveRecord/NamedScope/C... Maybe it's this in Django: http://docs.djangoproject.com/en/dev/topics/db/managers/#modifying-initial-manager-querysets http://docs.djangoproject.com/en/dev/topics/db/managers/#mod... The overall purist argument, that the controller should have no knowledge of the database seems ridiculous. If one removes or adds a field to the model, one would expect there to be cascading changes across the whole system. If you generate most of your SQL with an ORM, you can switch database with minimal fuss. Although, how often do you do that anyway? ORM provides a lot of benefit in reducing the amount of code you have to write for many operations.
- cageface 16y ago
  Beginning programmers make the mistake of abstracting too little. Intermediate programmers make the mistake of abstracting too much. Experienced programmers know when abstractions add value and when they just add another layer of extra complexity. Generally speaking, designing your code up front to answer vague & ill-defined portability concerns you may encounter down the road means you're abstracting too much & too early.
  - acangiano 16y ago
    Extremely accurate statement in my experience.
  - mononcqc 16y ago
    Can't help but feel this is directed at the post. While I can definitely agree doing it for every single ORM call for maintenance purposes is sometimes overkill (some calls just are easy to refactor, or following a rule similar to "3-strikes you're out" keeps it sane), I find it hard to imagine another way to get code that is as easy to test and to mock. Any suggestion? Then I think Groxx hit the nail on the head with his comment (http://news.ycombinator.com/item?id=1636322 http://news.ycombinator.com/item?id=1636322). The advice is mostly valid for large-scale projects, things you expect to stay forever, which is what I'm maintaining at the moment; Not for experimental or short-lived code (or whatever is logical on that spectrum).
stevenwei 16y ago
I've found the repository pattern (http://martinfowler.com/eaaCatalog/repository.html http://martinfowler.com/eaaCatalog/repository.html) quite useful in large projects to abstract away some of the detail of the ORM. Using this technique has the added benefit of being able to mock out the repository during unit testing (so you're not tied to having to setup/teardown your database between every run, which if you have a large project, can take quite a bit of time). It also helps with maintainability because all of your ORM related code is basically in one place. I do think this is overkill for smaller projects as you're just adding another layer of indirection. In most projects, I think it's best to start out by keeping things simple, and refactor towards these additional abstractions as the need grows. One thing I've always found when using an ORM, (especially in web projects although really in anything with a user interface) is that the abstraction is always leaky, especially as you start to optimize for performance. You might start out with a simple query for Books, but then realize for this specific page, you're only rendering Book.name and Book.price, so you should limit your query to those fields. Or you realize that for another page you should be eager loading Book.authors and lazy loading something else. If you don't pay attention to the fact that your data is ultimately backed by the database you could end up doing things that cause your ORM to issue more queries or request more data than is really necessary. But once you do start paying attention and customizing/optimizing those queries, your queries can end up very specific to the page that they're being rendered on.
endlessvoid94 16y ago
I hate rails. But one thing I like is that activerecord will build functions for you. e.g.: User.find_by_email(email) I've started writing these types of functions manually now and it makes EVERYTHING so much simpler. I never thought I'd say it...but...thank you rails.
- endlessvoid94 16y ago
  I guess I'm being downvoted for contributing to the discussion?
- chrismsnz 16y ago
  So you're using ORM specific functionality in the controller? :)
  - endlessvoid94 16y ago
    I...guess so. But it successfully removes the reason that becomes painful and leaky, e.g., queries. The function names are short and readable, and the execution of the query is in the ORM where it belongs.
adamilardi 16y ago
In java("an enterpise level language") we never call orm directly from a controller Typically we do 3 or more layers. controller->facade-layer(can handle transactions via annotations)->business objects->dao OR controller->business objects->dao
hp1995acer 16y ago
I'm going to make up hypothetical problem. then have the developer use the most retarded solution, and then use the resulting make believe coding chaos to prove my point. Yay Programming Blogging :/
bfjotld 16y ago
How innocent the article is written :) People are expecting to be able to build a cheap just hang in there type of flying thing, which would magically transform itself into a stealth one with minimal tweaking here and there. Of course not. And it's not about abstractions, middle-ware, etc. Everything is a tool. If you need to abstract, you abstract. If you have experience, you realize that the client is not aware of what he will want in the future. That he will want triple sorts, grouping by, etc. I recommend to the writer of the article to memorize the Tao of Programming. Only Tao is perfect. Yeah, you can claim your tool is perfect, that is magic, it's not, ask the guy that has to maintain the thing.
- mononcqc 16y ago
  I'm the guy maintaining the thing, and I'm complaining about ORMs not being perfect. I'm not sure I get your point.
  - bfjotld 16y ago
    It means that you still fight with the tool - which is a waste of energy. Accept the tools are not perfect, get your benefits out of them (either financial or intellectual - for example by making something better) and use the remaining energy in something more constructing for your self. Quote from Tao of Programming: 'There once was a master programmer who wrote unstructured programs. A novice programmer, seeking to imitate him, also began to write unstructured programs. When the novice asked the master to evaluate his progress, the master criticized him for writing unstructured programs, saying, ``What is appropriate for the master is not appropriate for the novice. You must understand the Tao before transcending structure.'
richcollins 16y ago
Now your boss has decided he wants this example search on 6 different pages (that require 6 different controllers). What do you do? Copy/paste the query within all 6 controllers Create a class with a method (or just a function) to handle it Why is the idea that you shouldn't duplicate code novel enough to make it to the front page of HN?

37 points by mononcqc 16y ago

Using an ORM does buy you some abstraction, though. You get to worry about your business objects and fields instead of your tables and columns. (Depending on the language/situation this can be a big deal) You're right that it doesn't provide all the abstraction you probably want, but it easily gives you a toehold on which to build that abstraction. Doing it from scratch, by hand is often a much worse place to start.

mononcqc 16y ago
I agree. Maybe I should have been clearer on my views. ORMs are a great tool; the way they are used with the leaky abstraction is what I have a problem with. Restricted to the model layer, they're entirely worth it and provide lots of help.

hippich 16y ago

ORM is not always Model. ORM is tool to build Model. If queries become to SQLish, add another layer on top of ORM with methods to retrieve data which encapsulate all SQLish methods and provide nice and consistent interface.

mgkimsal 16y ago

"This means that we're in the controller but we still have to grasp what the hell is going on in the database. The abstraction is leaking!" How many people JUST work in controllers and can remain oblivious to what's going on in the DB? Just because you move the ORM or any DB call to a different area of the code doesn't mean you shouldn't have a grasp of what's going on. If I wrapped all ORM calls in some service layer classes, that simply means that now when I'm looking at my service layer classes I "still have to grasp what the hell is going on in the database. The abstraction is leaking!". Doesn't it?

mononcqc 16y ago
Try renaming a table column. Which one of the two has more impact on the code? Of course the same programmers can work from the model up to the view, but the idea is to reduce the impact of changes in lower layers, and help with testing. The example of the database info required is used to illustrate the leak. It's not a horror in itself.

teilo 16y ago

The problem is not the ORM. Anyone who is writing such complex queries directly in the controller needs to re-think how they have designed their site. Specificity like this belongs in the View and the DB itself, not in the Controller OR the Model. Adding umpteen custom getters to the Model is not helping matters. It may be easier to test, but it is just as unwieldy and difficult to maintain. It is the wrong solution to the problem. These things need to be parametized in some way and made generic. In Django you would likely do this with Q objects, and a front-end that lets you filter down your results on-the-fly, and save the resulting query to the DB for recall later. This is even easier to test, and prevents your Models from getting cluttered with functions which are far too specific.

mononcqc 16y ago
I didn't know about Q objects before. What are their capabilities regarding, say, moving from an SQL database to a web-service or a third party application? I mean, this is the kind of stuff I have to do surprisingly often when maintaining legacy applications and putting them up to date. Your data model ends up changing or just not supporting all the same idioms again. You might need to join some info coming from your database without half of it coming from a RPC. I've found such changes in the foundations of where data you get is to cause a whole damn lot of problems on all the layers above. I haven't figured out a better way than wrapping lots of it in fat models, but if Q objects have something that can help with that I'd be very happy to learn about it.
- teilo 16y ago
  Q objects are a way to represent complex SQL in a progressive fashion that would be unwieldy or impossible with the standard Django ORM chaining technique. It really doesn't address your use case.

aphyr 16y ago

I run into this question frequently: how much logic belongs in the controller, and how much in the model. The difficulty, I think, is that every method you add to the model requires a name, and the profusion of names leads to complexity and poorly named methods. An alternative is to collapse similar methods (for example, two queries differing only by "order by") into one, and use a parameter or function argument to distinguish the cases. But this pushes ORM-specific logic back up into the controller--only now with an extra layer of indirection for the programmer to remember how to use. My approach has been to define methods on the model only when they are used in more than one place or have a concise, meaningful name. Introducing the abstraction later rarely requires modifying several aspects of the application, and Sequel's def_dataset_method/chaining approach make this relatively painless.

Xurinos 16y ago
> how much logic belongs in the controller, and how much in the model I always thought this was really simple (at least based on the original Xerox definition of MVC): Controller transforms inputs/parameters as needed. Controller can use Model to do this. Controller creates/manages views and sends latest data to the Views. And a confusion-reducer: Controller is not between Model and Views. Views have direct access to Model. It ONLY uses Model to figure out the situation. I have found these clear separations helpful in a variety of applications. I have also found that people that apply MVC to the web confuse themselves, too. On the server side, your View is what you are returning to the browser. JavaScript is part of the View of the server. It has its own MVC when it is run in the browser environment, and the browser has its own MVC. It is perfectly acceptable to have interlocking components, each with their own MVC.
- mononcqc 16y ago
  A note: The architecture model where the controller sits between the model and view is instead called 'Model-View-Adapter (http://en.wikipedia.org/wiki/Model-view-adapter http://en.wikipedia.org/wiki/Model-view-adapter), where the controller (or adapter) is doing the mediation between the model and the view. It supposedly allows for a stronger separation of concerns.

deleted 16y ago

[deleted]

Groxx 16y ago

For large-scale projects, this makes a lot of sense. Without another layer of abstraction to keep database-content-specific code out of your controllers, you end up with: db -> orm -> 100,000 flat-level queries tied directly to the format returned by the orm With abstraction like they suggest, you can achieve: db -> orm -> [your organization method here] -> your never-changing data you requested Significantly better if you're working at that scale, and if you ever need to change your DB/ORM. In a situation like that, I wholly agree: there should be zero ORM-tied code in your controller. BUT: tutorials are rarely-if-ever for people working at that scale. You should have some grasp of what you're doing if you're attacking a problem of that scale, or at least be beyond the tutorial stage. If tutorials addressed this, they'd remind me of most beginning stuff for Java programming: subclass this, override main, subclass that, subclass that, subclass that, build a polymorphic inheritance structure, print "hello world". Easy! Totally opaque until you understand the underlying organization of Java, and it tends to distract newcomers from things they want to achieve, by bogging them down in things should do when they have no comprehension of why. Once you understand why, it's merely annoying; before that, you really can't know if you're supposed to use an interface, inherit from Object or Array or GenericArrayListWithBellsAndWhistlesRadixSorted, or just code the damn thing by hand. ORM tutorials are for coding the damn thing by hand, a necessary starting point for building that scalable abstracted structure, and the foundation of it. Not for helping you build a 3000-table, 5-billion-row, 10,000-request-per-second Google-killer.

misterbwong 16y ago

I think the author needs to make a clearer distinction between business logic and view logic. I'm an advocate of creating a view model layer above the traditional model containing things like sorting and field suppression, etc. The app stack might look like this: DB -> ORM -> BizLogic -> ViewLogic -> View This has the effect of decoupling actual business logic (i.e. get me every british author) from view logic (i.e. sort them by name in my table) and allows the design to be more flexible in the future.

pvg 16y ago

That's an impressively verbose and obfuscated way to re-state the very basic and completely independent (of ORM, Django, Python, etc) notion that if you have common code you should abstract it into a separate function.

mattmcknight 16y ago

I think we call these Named Scopes: http://api.rubyonrails.org/classes/ActiveRecord/NamedScope/ClassMethods.html http://api.rubyonrails.org/classes/ActiveRecord/NamedScope/C... Maybe it's this in Django: http://docs.djangoproject.com/en/dev/topics/db/managers/#modifying-initial-manager-querysets http://docs.djangoproject.com/en/dev/topics/db/managers/#mod... The overall purist argument, that the controller should have no knowledge of the database seems ridiculous. If one removes or adds a field to the model, one would expect there to be cascading changes across the whole system. If you generate most of your SQL with an ORM, you can switch database with minimal fuss. Although, how often do you do that anyway? ORM provides a lot of benefit in reducing the amount of code you have to write for many operations.

cageface 16y ago
Beginning programmers make the mistake of abstracting too little. Intermediate programmers make the mistake of abstracting too much. Experienced programmers know when abstractions add value and when they just add another layer of extra complexity. Generally speaking, designing your code up front to answer vague & ill-defined portability concerns you may encounter down the road means you're abstracting too much & too early.
- acangiano 16y ago
  Extremely accurate statement in my experience.
- mononcqc 16y ago
  Can't help but feel this is directed at the post. While I can definitely agree doing it for every single ORM call for maintenance purposes is sometimes overkill (some calls just are easy to refactor, or following a rule similar to "3-strikes you're out" keeps it sane), I find it hard to imagine another way to get code that is as easy to test and to mock. Any suggestion? Then I think Groxx hit the nail on the head with his comment (http://news.ycombinator.com/item?id=1636322 http://news.ycombinator.com/item?id=1636322). The advice is mostly valid for large-scale projects, things you expect to stay forever, which is what I'm maintaining at the moment; Not for experimental or short-lived code (or whatever is logical on that spectrum).

stevenwei 16y ago

I've found the repository pattern (http://martinfowler.com/eaaCatalog/repository.html http://martinfowler.com/eaaCatalog/repository.html) quite useful in large projects to abstract away some of the detail of the ORM. Using this technique has the added benefit of being able to mock out the repository during unit testing (so you're not tied to having to setup/teardown your database between every run, which if you have a large project, can take quite a bit of time). It also helps with maintainability because all of your ORM related code is basically in one place. I do think this is overkill for smaller projects as you're just adding another layer of indirection. In most projects, I think it's best to start out by keeping things simple, and refactor towards these additional abstractions as the need grows. One thing I've always found when using an ORM, (especially in web projects although really in anything with a user interface) is that the abstraction is always leaky, especially as you start to optimize for performance. You might start out with a simple query for Books, but then realize for this specific page, you're only rendering Book.name and Book.price, so you should limit your query to those fields. Or you realize that for another page you should be eager loading Book.authors and lazy loading something else. If you don't pay attention to the fact that your data is ultimately backed by the database you could end up doing things that cause your ORM to issue more queries or request more data than is really necessary. But once you do start paying attention and customizing/optimizing those queries, your queries can end up very specific to the page that they're being rendered on.

endlessvoid94 16y ago

I hate rails. But one thing I like is that activerecord will build functions for you. e.g.: User.find_by_email(email) I've started writing these types of functions manually now and it makes EVERYTHING so much simpler. I never thought I'd say it...but...thank you rails.

endlessvoid94 16y ago
I guess I'm being downvoted for contributing to the discussion?
chrismsnz 16y ago
So you're using ORM specific functionality in the controller? :)
- endlessvoid94 16y ago
  I...guess so. But it successfully removes the reason that becomes painful and leaky, e.g., queries. The function names are short and readable, and the execution of the query is in the ORM where it belongs.

adamilardi 16y ago

In java("an enterpise level language") we never call orm directly from a controller Typically we do 3 or more layers. controller->facade-layer(can handle transactions via annotations)->business objects->dao OR controller->business objects->dao

hp1995acer 16y ago

I'm going to make up hypothetical problem. then have the developer use the most retarded solution, and then use the resulting make believe coding chaos to prove my point. Yay Programming Blogging :/

bfjotld 16y ago

How innocent the article is written :) People are expecting to be able to build a cheap just hang in there type of flying thing, which would magically transform itself into a stealth one with minimal tweaking here and there. Of course not. And it's not about abstractions, middle-ware, etc. Everything is a tool. If you need to abstract, you abstract. If you have experience, you realize that the client is not aware of what he will want in the future. That he will want triple sorts, grouping by, etc. I recommend to the writer of the article to memorize the Tao of Programming. Only Tao is perfect. Yeah, you can claim your tool is perfect, that is magic, it's not, ask the guy that has to maintain the thing.

mononcqc 16y ago
I'm the guy maintaining the thing, and I'm complaining about ORMs not being perfect. I'm not sure I get your point.
- bfjotld 16y ago
  It means that you still fight with the tool - which is a waste of energy. Accept the tools are not perfect, get your benefits out of them (either financial or intellectual - for example by making something better) and use the remaining energy in something more constructing for your self. Quote from Tao of Programming: 'There once was a master programmer who wrote unstructured programs. A novice programmer, seeking to imitate him, also began to write unstructured programs. When the novice asked the master to evaluate his progress, the master criticized him for writing unstructured programs, saying, ``What is appropriate for the master is not appropriate for the novice. You must understand the Tao before transcending structure.'

richcollins 16y ago

Now your boss has decided he wants this example search on 6 different pages (that require 6 different controllers). What do you do? Copy/paste the query within all 6 controllers Create a class with a method (or just a function) to handle it Why is the idea that you shouldn't duplicate code novel enough to make it to the front page of HN?