5 ms·

I strongly disagree. For many backends/purposes, SQL is really the thing doing all the real work and the backend's just some fancy authentication and validation

by dagss 7y ago

I strongly disagree. For many backends/purposes, SQL is really the thing doing all the real work and the backend's just some fancy authentication and validation layer. Stick close to the technology doing the heavy lifting. If people abstract away SQL too much they just start reimplementing things in buggy underperforming code, using 1000 lines of Go or Java or whatever what should be done in 20 lines of SQL. Also, don't get med started on people thinking they can abstract away tye database for testing. The automated test suite should always include the real database you are working against and all queries executed. I will never write a backend again without testing against the real database (whether sql or nosql). Some people justify abstracting away the database saying "one needs to be able to switch to an arbitrary storage engine". Well..storing things is probably the main purpose of your backend. I sometimes turn that around and say, "I want to write things in SQL so that I can easily change what language the backend is written it".

xrisk 7y ago
Testing against the real database? Won't keeping a copy of the database just for that be expensive? Also, should devs really be seeing potentially sensitive data? On a side note, "I use SQL so I can change the backend language" is the hottest take I've heard in some time heh.
- dagss 7y ago
  With database I meant an instance of whatever database engine you are using, not the actual data. My current project uses mssql. Each test run probably spins up and destroys 100 databases (inside one mssql docker container that is spun up for each test run). Each test function populates a DB from scratch (using the same sql migration scripts that we have used in prod), runs the test, drops the database. Can do that quite some times in the minute it takes to run the full suite. Point is to actually execute the SQL (or whatever NoSQL you use) as part of the test.
Aperocky 7y ago
And that's a problem - my opinion remain that individual sql queries are better left for analyst and data engineers to craft so they can discover whatever correlation they might be interested in. But for a full scale app it shouldn't be in the forefront, at least not in the same package as other logic. The entire idea of programming is the one where you abstract away things and then abstract away the thing you're currently writing. I agree that it has gone too far in some of the places, but abstracting SQL away is the first and almost one of the best abstraction you can make.
- dagss 7y ago
  Analysts and data engineers these days mostly work on seperate data warehouses? Do they get involved in the OLTP work in your case? I was talking about using SQL specifically for the OLTP workload not analysis. If a query that is necessary for a backend response in some REST API ends up taking 10 minutes, but could have taken 100ms if the backend developer just knew what they were really doing over in the SQL land....the backend developer will probably waste a lot of time doing silly things with the tools available to him/her (perhaps introduce redis to cache stuff there is no need for caching, build their own fragile homegrown index table or aggregation table manually using 100 to 1000 lines of backend code plus tests, and so on). I mean, noone ships a totally broken application that noone can use because it is so slow then asks the SQL expert to optimize it onve it has shipped!! Instead any lack of knowledge of underlying database will just mean reinvention and needless and fragile cruft in the backend code. Note: I am against abstracting away the specific database technology you sit on (whether some SQL or not). If you are on SQL, know about indexes and materialized views (and their limitations) and so on and use them to implement efficient API endpoints, let your model in SQL directly handle any questions about idempotency or races in your API, and so on. If you are talking about just abstracting away the SQL syntax, not the DB technology as such... such abstractions I am "meh" about, not very against them but they are usually very leaky so... They don't seem to bring anything fundamental for or against just syntax candy (getting in the way in my case).

by dagss 7y ago

xrisk 7y ago

Testing against the real database? Won't keeping a copy of the database just for that be expensive? Also, should devs really be seeing potentially sensitive data? On a side note, "I use SQL so I can change the backend language" is the hottest take I've heard in some time heh.

dagss 7y ago
With database I meant an instance of whatever database engine you are using, not the actual data. My current project uses mssql. Each test run probably spins up and destroys 100 databases (inside one mssql docker container that is spun up for each test run). Each test function populates a DB from scratch (using the same sql migration scripts that we have used in prod), runs the test, drops the database. Can do that quite some times in the minute it takes to run the full suite. Point is to actually execute the SQL (or whatever NoSQL you use) as part of the test.

Aperocky 7y ago

And that's a problem - my opinion remain that individual sql queries are better left for analyst and data engineers to craft so they can discover whatever correlation they might be interested in. But for a full scale app it shouldn't be in the forefront, at least not in the same package as other logic. The entire idea of programming is the one where you abstract away things and then abstract away the thing you're currently writing. I agree that it has gone too far in some of the places, but abstracting SQL away is the first and almost one of the best abstraction you can make.

dagss 7y ago
Analysts and data engineers these days mostly work on seperate data warehouses? Do they get involved in the OLTP work in your case? I was talking about using SQL specifically for the OLTP workload not analysis. If a query that is necessary for a backend response in some REST API ends up taking 10 minutes, but could have taken 100ms if the backend developer just knew what they were really doing over in the SQL land....the backend developer will probably waste a lot of time doing silly things with the tools available to him/her (perhaps introduce redis to cache stuff there is no need for caching, build their own fragile homegrown index table or aggregation table manually using 100 to 1000 lines of backend code plus tests, and so on). I mean, noone ships a totally broken application that noone can use because it is so slow then asks the SQL expert to optimize it onve it has shipped!! Instead any lack of knowledge of underlying database will just mean reinvention and needless and fragile cruft in the backend code. Note: I am against abstracting away the specific database technology you sit on (whether some SQL or not). If you are on SQL, know about indexes and materialized views (and their limitations) and so on and use them to implement efficient API endpoints, let your model in SQL directly handle any questions about idempotency or races in your API, and so on. If you are talking about just abstracting away the SQL syntax, not the DB technology as such... such abstractions I am "meh" about, not very against them but they are usually very leaky so... They don't seem to bring anything fundamental for or against just syntax candy (getting in the way in my case).