5 ms·

Actually "writing queries in some language other than SQL" which has static typing and catching issues in compile time is quite big for me. Add automatic databa

by ozim 8y ago

Actually "writing queries in some language other than SQL" which has static typing and catching issues in compile time is quite big for me. Add automatic database migrations that are also keeping types in line with code and whole bunch of "mess of mapping code" goes away. Though I use .NET EntityFramework which by now is really mature and heavily invested into by MS. Not sure how it is with other environments but I think I have it too easy.

JamesBarney 8y ago
Not to mention code reuse. Say for instance we have one query that finds eligible bachelors near me and another that finds newly eligible bachelors. We need two queries but they both rely on the same underlying domain concept of "eligible bachelors". In entity framework it's easy to save that as an expression that can be reused in multiple places. It's hard to do that in sql in a maintainable way. Not to mention refactoring issues. If I decide to change a datetime field called time to split it into a datetime and offset that change is far safer in c# than it is SQL.
- polygotdomain 8y ago
  > It's hard to do that in sql in a maintainable way <cough>Views</cough>
  - btilly 8y ago
    I have spent enough of my life untangling a mess of views that create performance problems to doubt that views are maintainable. Also, there is usually no good solution for keeping the database in sync with code releases. With the result that database+code change synchronization tends to be a source of deployment complexity. Keeping the logic in one place is better than the alternative. And of the two, the tools to manage source control are better.
    - lenkite 8y ago
      We use liquibase https://www.liquibase.org/ https://www.liquibase.org/ for tracking, managing and applying any schema changes. The liquibase changelogs are part of our project releases.
      - OJFord 8y ago
        > Supports XML, YAML, JSON and SQL formats XML is the default on the page, but wow the SQL is _so_ much shorter and more readable than the others! To the extent that I wonder what's the point, the obvious is machine readabaility, but one not three, and is SQL really so machine-unreadable that it's worth the extra depth and complexity?
        radiowave 8y ago
        With Liquibase, when you write changesets using SQL, you have to write the rollback code yourself. When you write them with Liquibase's built-in operations in (for example) XML, the rollback is derived automatically. I write SQL almost daily, but for the things that can easily be expressed with Liquibase's operations, I can write them faster and with fewer errors in XML, so long as I'm using a text editor with XML schema support (hence: autocompletion and validation as-you-type). PyCharm works well for this.
    - polygotdomain 8y ago
      >I have spent enough of my life untangling a mess of views that create performance problems to doubt that views are maintainable. Just like concepts like inheritance can be a burden when overused, views can be overused as well. Like inheritance, having views go too many levels deep generally winds up causing more harm than good. Used correctly though, views are able to capture and centralize concepts in a way that are reusable, can be more easily tuneable, and make maintenance easier in the long run. Queries can be simpler and encompass less responsibility as the common logic is implemented by the view, and the logic that is specific to that query is more readily apparent. IMHO thinking of the DB as just a collection of tables misses out on a key db feature. >Also, there is usually no good solution for keeping the database in sync with code releases. With the result that database+code change synchronization tends to be a source of deployment complexity. I disagree that there's no good solution, but I understand that it's not as "out of the box" as many ORMs offer. That being said, writing SQL scripts that coordinate schema changes is not hard, and any CI/CD pipeline that can run commandline utilities can run a sql script against a database. At my work, we use DbUp (https://dbup.github.io/ https://dbup.github.io/) and are able to easily write small, isolated SQL scripts that correspond with code changes and get committed to source control, run all of those scripts with our CI/CD pipeline across multiple sharded databases, and do so quite seamlessly. We don't have any questions about migrations or what a tool did, as running the SQL script individually is the same as it runs on deployment. Coordinating db changes on developer machines is a breeze as well. >And of the two, the tools to manage source control are better. All of our sql scripts are committed into source control. It's not any easier or harder than looking at a code file, an html file, or a config file to get to the logic your working with.
- dragonwriter 8y ago
  > Say for instance we have one query that finds eligible bachelors near me and another that finds newly eligible bachelors. We need two queries but they both rely on the same underlying domain concept of "eligible bachelors". [...] It's hard to do that in sql in a maintainable way. Seems a textbook case of using a view for sharing query logic. > If I decide to change a datetime field called time to split it into a datetime and offset that change is far safer in c# than it is SQL. If you can do it in C# instead of SQL, you must be talking about a change in the interface used by an app (so, with good DB design, a view, not a base table). But this is very safe in SQL, you are changing a view definition and, if its a read/write view, insert and update triggers.
  - Dayshine 8y ago
    >Seems a textbook case of using a view for sharing query logic. Views are just terrible outside of data analyst style work. Unmaintainable, restricted to SQL structures, require absurd hacks or custom dlls, not properly source controlled, difficult to perform performance analysis on, hard to update. You can't combine views easily, you can't cache results, you don't get static type checking.
    - GordonS 8y ago
      I don't like to say "you're doing it wrong", but TBH it sounds like you are. > restricted to SQL structures So is your ORM; it just adds a layer of abstraction > require absurd hacks or custom dlls Literally no idea what you're doing that views require anything out of the ordinary, or indeed "custom DLLs" > not properly source controlled Eh? You can store your view DDL files in source control just fine. I mean, they're just SQL files; why couldn't you? > difficult to perform performance analysis on I don't see how ORMs offer any advantage here - if anything, SQL has the edge here, since your typical SQL GUI will draw pretty graphs to help visualise EXPLAIN, and some (e.g. SSMS) will even suggest missing indexes > hard to update It's just SQL, and for views typically rather basic SQL at that. I use ORMs on 99% of the projects I work on, sometimes heavyweights like Entity Framework Core or Marten (which I love, BTW), and sometimes more lightweight like Dapper - but I know when to use the right tool for the job, and sometimes that's SQL views. I just don't get any of your arguments against them.
jeltz 8y ago
I have tied to use automatic migrations and I do not see the point. Either your diff is trivial and it is trivial to write a migration or your diff is complex and you have to care about the lock levels of schema changes, the time complexity of changes and that the migration transforms data correctly. The code generating the automatic migrations needs knowledge about the structure of the production data.
- alexnewman 8y ago
  exactly
GordonS 8y ago
I'm very comfortable with SQL, but I also like the static typing and compile-time safety that linq brings. Automatic migrations I like less. In particular, Entity Framework and Entity Framework Core migrations. Firstly, it seems that the second you stray from the most basic of scenarios, thry generate incorrect DDL. Secondly, I'm just not a fan of 'magic' code gen. Thirdly, every entity/model change a dev makes does not need to equate to a schema migration - that's how you end up quickly accumulating hundreds of schema migrations. When I have to use MS SQL Server, the approach I favour is to use CQRS, using EF Core for the majority of cases, then using Dapper when I need more control. And for migrations I like DbUp, because it provides me with full control over when it's time to create a migration file, and what better DSL for migrations than... SQL.
- rawoke083600 8y ago
  Damn that's a lot of enterprise level words and acronyms to just "store some data". Lol not saying you wrong... just don't like this part of our industry. Is it too late to go back to Turbo Pascal (record-files) ? :)
  - GordonS 8y ago
    Not sure I really mentioned much 'enterprise-level', beyond SQL Server? I guess I'm making it sound more complicated than it really is then, because this approach is actually a lot simpler than the 'N-tier' abstraction-hell of yesteryear. Jimmy Bogard has a simple example here: https://github.com/jbogard/ContosoUniversityDotNetCore https://github.com/jbogard/ContosoUniversityDotNetCore Some code can probably explain it better than me ;)
mmusc 8y ago
I much prefer having an vs sql project side by side with the ef models. Keep the schema up to date in there. It can generate migration scripts by comparing the dB you want to target. Then just generate ef models from the dB.
dllthomas 8y ago
I've been wanting to build static typing for raw sql atop https://github.com/uber/queryparser https://github.com/uber/queryparser
SOLAR_FIELDS 8y ago
Lack of adequate support for versioning your schema and in general bad support for anything versioning with RDBMS is one of the main dislikes I have for SQL. I find SQL to be a beautifully expressive language when you get the hang of it, but no RDBMS that I know of has been able to adequately tackle the versioning problem at both the schema and the script level. Everyone's schema evolves over time, why is it so difficult for the RDBMS to try to capture the idea of database evolutions? This is the single biggest selling point for me in ORM land, because with regular (read: non-SQL) code you have all the typical version control mechanisms to help you manage changes to the schema. It's still not ideal, but a lot better than trying to manage this kind of thing in any of the tools that mainstream RDBMS have.
- mikeryan 8y ago
  This is a big thing for me. Most ORMs I’ve used have decent migration tools which is a huge benefit for managing your schema. We enforce changing Schemas via migrations as opposed to any sort of auto-migrations some packages offer. Makes life a ton easier being able to put these changes into a build pipeline.
  - SOLAR_FIELDS 8y ago
    Migrations (or evolutions, depending on the language you are using) are a joy when implemented properly. Alembic and Play Evolutions are two frameworks that I have dealt with specifically that are pretty great and hard to go back from.
- drwiggly 8y ago
  Yes I agree. I ended up making a small DSL to describe tables. Then another program to translate that in to a script that checks everything through meta data in the target db (atm it does mysql). I check the table structure files into version control. If I wanted an old version pull that checkout and run the generator script. I end up with a pure SQL script that creates a blank db with bootstrap data or asserts all the columns/indexes exist for that version of the code. There is another small dsl for bootstrap data, and I just check in all stored procs like normal code. What this doesn't give you is good change column type semantics but any other tool I've ever used when this happens, can't be trusted anyway you have manage it yourself. This type of change is pretty infrequent though. For me it works better to assert the schema I want now. Drop columns arn't done due to similar thorny issues that usually a human has to deal with. Or if its simple I put those in a cleanup proc that runs as the last part of the script. Using normal code to describe structure I guess is similar its just you're fighting the language with meta data extensions, and also whatever you do isn't language agnostic. You're also at the mercy of whatever magic is going to happen when you change schema with that lib.

by ozim 8y ago

JamesBarney 8y ago

Not to mention code reuse. Say for instance we have one query that finds eligible bachelors near me and another that finds newly eligible bachelors. We need two queries but they both rely on the same underlying domain concept of "eligible bachelors". In entity framework it's easy to save that as an expression that can be reused in multiple places. It's hard to do that in sql in a maintainable way. Not to mention refactoring issues. If I decide to change a datetime field called time to split it into a datetime and offset that change is far safer in c# than it is SQL.

polygotdomain 8y ago
> It's hard to do that in sql in a maintainable way <cough>Views</cough>
- btilly 8y ago
  I have spent enough of my life untangling a mess of views that create performance problems to doubt that views are maintainable. Also, there is usually no good solution for keeping the database in sync with code releases. With the result that database+code change synchronization tends to be a source of deployment complexity. Keeping the logic in one place is better than the alternative. And of the two, the tools to manage source control are better.
  - lenkite 8y ago
    We use liquibase https://www.liquibase.org/ https://www.liquibase.org/ for tracking, managing and applying any schema changes. The liquibase changelogs are part of our project releases.
    - OJFord 8y ago
      > Supports XML, YAML, JSON and SQL formats XML is the default on the page, but wow the SQL is _so_ much shorter and more readable than the others! To the extent that I wonder what's the point, the obvious is machine readabaility, but one not three, and is SQL really so machine-unreadable that it's worth the extra depth and complexity?
      - radiowave 8y ago
        With Liquibase, when you write changesets using SQL, you have to write the rollback code yourself. When you write them with Liquibase's built-in operations in (for example) XML, the rollback is derived automatically. I write SQL almost daily, but for the things that can easily be expressed with Liquibase's operations, I can write them faster and with fewer errors in XML, so long as I'm using a text editor with XML schema support (hence: autocompletion and validation as-you-type). PyCharm works well for this.
  - polygotdomain 8y ago
    >I have spent enough of my life untangling a mess of views that create performance problems to doubt that views are maintainable. Just like concepts like inheritance can be a burden when overused, views can be overused as well. Like inheritance, having views go too many levels deep generally winds up causing more harm than good. Used correctly though, views are able to capture and centralize concepts in a way that are reusable, can be more easily tuneable, and make maintenance easier in the long run. Queries can be simpler and encompass less responsibility as the common logic is implemented by the view, and the logic that is specific to that query is more readily apparent. IMHO thinking of the DB as just a collection of tables misses out on a key db feature. >Also, there is usually no good solution for keeping the database in sync with code releases. With the result that database+code change synchronization tends to be a source of deployment complexity. I disagree that there's no good solution, but I understand that it's not as "out of the box" as many ORMs offer. That being said, writing SQL scripts that coordinate schema changes is not hard, and any CI/CD pipeline that can run commandline utilities can run a sql script against a database. At my work, we use DbUp (https://dbup.github.io/ https://dbup.github.io/) and are able to easily write small, isolated SQL scripts that correspond with code changes and get committed to source control, run all of those scripts with our CI/CD pipeline across multiple sharded databases, and do so quite seamlessly. We don't have any questions about migrations or what a tool did, as running the SQL script individually is the same as it runs on deployment. Coordinating db changes on developer machines is a breeze as well. >And of the two, the tools to manage source control are better. All of our sql scripts are committed into source control. It's not any easier or harder than looking at a code file, an html file, or a config file to get to the logic your working with.
dragonwriter 8y ago
> Say for instance we have one query that finds eligible bachelors near me and another that finds newly eligible bachelors. We need two queries but they both rely on the same underlying domain concept of "eligible bachelors". [...] It's hard to do that in sql in a maintainable way. Seems a textbook case of using a view for sharing query logic. > If I decide to change a datetime field called time to split it into a datetime and offset that change is far safer in c# than it is SQL. If you can do it in C# instead of SQL, you must be talking about a change in the interface used by an app (so, with good DB design, a view, not a base table). But this is very safe in SQL, you are changing a view definition and, if its a read/write view, insert and update triggers.
- Dayshine 8y ago
  >Seems a textbook case of using a view for sharing query logic. Views are just terrible outside of data analyst style work. Unmaintainable, restricted to SQL structures, require absurd hacks or custom dlls, not properly source controlled, difficult to perform performance analysis on, hard to update. You can't combine views easily, you can't cache results, you don't get static type checking.
  - GordonS 8y ago
    I don't like to say "you're doing it wrong", but TBH it sounds like you are. > restricted to SQL structures So is your ORM; it just adds a layer of abstraction > require absurd hacks or custom dlls Literally no idea what you're doing that views require anything out of the ordinary, or indeed "custom DLLs" > not properly source controlled Eh? You can store your view DDL files in source control just fine. I mean, they're just SQL files; why couldn't you? > difficult to perform performance analysis on I don't see how ORMs offer any advantage here - if anything, SQL has the edge here, since your typical SQL GUI will draw pretty graphs to help visualise EXPLAIN, and some (e.g. SSMS) will even suggest missing indexes > hard to update It's just SQL, and for views typically rather basic SQL at that. I use ORMs on 99% of the projects I work on, sometimes heavyweights like Entity Framework Core or Marten (which I love, BTW), and sometimes more lightweight like Dapper - but I know when to use the right tool for the job, and sometimes that's SQL views. I just don't get any of your arguments against them.

jeltz 8y ago

I have tied to use automatic migrations and I do not see the point. Either your diff is trivial and it is trivial to write a migration or your diff is complex and you have to care about the lock levels of schema changes, the time complexity of changes and that the migration transforms data correctly. The code generating the automatic migrations needs knowledge about the structure of the production data.

alexnewman 8y ago
exactly

GordonS 8y ago

I'm very comfortable with SQL, but I also like the static typing and compile-time safety that linq brings. Automatic migrations I like less. In particular, Entity Framework and Entity Framework Core migrations. Firstly, it seems that the second you stray from the most basic of scenarios, thry generate incorrect DDL. Secondly, I'm just not a fan of 'magic' code gen. Thirdly, every entity/model change a dev makes does not need to equate to a schema migration - that's how you end up quickly accumulating hundreds of schema migrations. When I have to use MS SQL Server, the approach I favour is to use CQRS, using EF Core for the majority of cases, then using Dapper when I need more control. And for migrations I like DbUp, because it provides me with full control over when it's time to create a migration file, and what better DSL for migrations than... SQL.

rawoke083600 8y ago
Damn that's a lot of enterprise level words and acronyms to just "store some data". Lol not saying you wrong... just don't like this part of our industry. Is it too late to go back to Turbo Pascal (record-files) ? :)
- GordonS 8y ago
  Not sure I really mentioned much 'enterprise-level', beyond SQL Server? I guess I'm making it sound more complicated than it really is then, because this approach is actually a lot simpler than the 'N-tier' abstraction-hell of yesteryear. Jimmy Bogard has a simple example here: https://github.com/jbogard/ContosoUniversityDotNetCore https://github.com/jbogard/ContosoUniversityDotNetCore Some code can probably explain it better than me ;)

mmusc 8y ago

I much prefer having an vs sql project side by side with the ef models. Keep the schema up to date in there. It can generate migration scripts by comparing the dB you want to target. Then just generate ef models from the dB.

dllthomas 8y ago

I've been wanting to build static typing for raw sql atop https://github.com/uber/queryparser https://github.com/uber/queryparser

SOLAR_FIELDS 8y ago

Lack of adequate support for versioning your schema and in general bad support for anything versioning with RDBMS is one of the main dislikes I have for SQL. I find SQL to be a beautifully expressive language when you get the hang of it, but no RDBMS that I know of has been able to adequately tackle the versioning problem at both the schema and the script level. Everyone's schema evolves over time, why is it so difficult for the RDBMS to try to capture the idea of database evolutions? This is the single biggest selling point for me in ORM land, because with regular (read: non-SQL) code you have all the typical version control mechanisms to help you manage changes to the schema. It's still not ideal, but a lot better than trying to manage this kind of thing in any of the tools that mainstream RDBMS have.

mikeryan 8y ago
This is a big thing for me. Most ORMs I’ve used have decent migration tools which is a huge benefit for managing your schema. We enforce changing Schemas via migrations as opposed to any sort of auto-migrations some packages offer. Makes life a ton easier being able to put these changes into a build pipeline.
- SOLAR_FIELDS 8y ago
  Migrations (or evolutions, depending on the language you are using) are a joy when implemented properly. Alembic and Play Evolutions are two frameworks that I have dealt with specifically that are pretty great and hard to go back from.
drwiggly 8y ago
Yes I agree. I ended up making a small DSL to describe tables. Then another program to translate that in to a script that checks everything through meta data in the target db (atm it does mysql). I check the table structure files into version control. If I wanted an old version pull that checkout and run the generator script. I end up with a pure SQL script that creates a blank db with bootstrap data or asserts all the columns/indexes exist for that version of the code. There is another small dsl for bootstrap data, and I just check in all stored procs like normal code. What this doesn't give you is good change column type semantics but any other tool I've ever used when this happens, can't be trusted anyway you have manage it yourself. This type of change is pretty infrequent though. For me it works better to assert the schema I want now. Drop columns arn't done due to similar thorny issues that usually a human has to deal with. Or if its simple I put those in a cleanup proc that runs as the last part of the script. Using normal code to describe structure I guess is similar its just you're fighting the language with meta data extensions, and also whatever you do isn't language agnostic. You're also at the mercy of whatever magic is going to happen when you change schema with that lib.