4 ms·

But software developers often struggle to use sql and prefer using ORMs or analytical APIs like polars; the people who excel at sql are typically not programmer

But software developers often struggle to use sql and prefer using ORMs or analytical APIs like polars; the people who excel at sql are typically not programmers, they’re data engineers, DBAs, analysts, etc. Maybe a similar bifurcation will arise where there are vibe coders who use LLMs to write everything, and there are real engineers who avoid LLMs. Maybe we’re seeing the beginning of that with the whole bifurcation of programmers into two camps: heavy AI users and AI skeptics.

nathanfig 1y ago
Claude made this point while reviewing my blog for me: the mechanization of farms created a whole lot more specialization of roles. The person editing CAD diagrams of next year's combine harvester may not be a farmer strictly speaking, but farming is still where their livelihood comes from.
- swader999 1y ago
  Land is scarce though. The amount of software work that needs doing might not be, it could be infinite or probably more tied to electrical capacity.
- deleted 1y ago
  [deleted]
- dredmorbius 1y ago
  Strictly speaking, farming is where all our livelihoods come from, in the greatest part. We're all living off the surplus value of food production. (Also of other food, energy, and materials sourcing: fishing, forestry, mining, etc.) This was the insight of the French economist François Quesnay in his Tableau économique, foundation of the Physiocratic school of economics.
  - eru 1y ago
    You might find https://en.wikipedia.org/wiki/Margarine#Coal_butter https://en.wikipedia.org/wiki/Margarine#Coal_butter fascinating. > Strictly speaking, farming is where all our livelihoods come from, in the greatest part. We're all living off the surplus value of food production. I don't think farming is special here, because food isn't special. You could make exactly the same argument for water (or even air) instead of food, and all of a sudden all our livelihoods would derive ultimately from the local municipal waterworks. Whether that's a reductio ad absurdum of the original argument, or a valuable new perspective on the local waterworks is left as an exercise to the reader.
    - dredmorbius 1y ago
      Food (and other fuels) are special. Water largely isn't fundamenally transformed with use (unless it's involved in a chemical reaction, though that's a minute fraction of all water usage), though it may be dispersed or degraded (usually contaminated with something). But it can recover its earlier state with appropriate applications of process and energy. Water (and much else we consume) is a material input rather than an energy input. With energy inputs it is the energy potential itself which provides value, and that potential is intrinsically consumed in their use. Water, wood, iron, aluminium, lithium, helium, etc., can all be recycled, restored to their useful state, at comparatively little cost. Collecting the waste products of food don't give you that, on two counts. First, most of the actual metabolic output is gaseous and lost to the atmosphere at large (CO2 and water vapour in your breath), and to the extent that solid and liquid human waste are useful in producing new food, it's a nutrient fertilisers which enable energy conversion of sunlight to fuel, and not the primary energy input itself (sunlight).
      - eru 1y ago
        I think you would want to be talking about entropy, not energy? Recycling all the materials you mentioned costs 'energy' (to use your terminology). The same for food: we can use used-up food and a lot of energy and grow new food. The process for 'recycling' wood is basically the same that for recycling food: you grow some plants. The waste products of used up wood are also basically the same as those for used up food. --- In any case, I don't see how any of this makes food more special than eg petrol or sunlight? And you can argue that food is only useful, if we have air, ie oxygen to burn it with.
    - bluGill 1y ago
      Except in a few places drinkable water is in such abundance that nobody every spent significant effort trying to get it. Likewise for air, few people have ever spent much effort getting air to breathe - even in the worst polluted areas bottled air was reserved for airplanes, hospitals (and a few people with medical conditions), and scuba divers.
      - 1y ago
- lipowitz 1y ago
  Removing jobs that could only be performed by those living near the particular fields with those that can be done anywhere makes jobs for the person willing to take the least satisfactory compensation for the most skill and work. Working the summer fields was one of the least desirable jobs but still gave local students with no particular skills a good supplemental income appropriate for whichever region.
  - eru 1y ago
    Yes, but automating these away means that food becomes cheaper. We increase the overall total prosperity with that automation.
    - lipowitz 1y ago
      Increasing total prosperity is the wrong goal if distribution is completely unregulated. Investor and real estate owning classes like the 1% get more, the salaries can trend down because food costs are down, in a deflation spiral the youth are perpetual dependents and/or debtors who can't possibly earn enough over day to day costs given global competition includes people with no debts or debts from an economy that was less wealthy.
      - eru 1y ago
        When and where has that ever happened? Btw, most countries have taxes and welfare anyway.
  - miki123211 1y ago
    depending on the job, it may also allow you to select for talent much better, which creates intense competition and raises salaries significantly. A good example of this phenomenon is sports. Even thought it can't be done remotely, it's so talent dependent that it's often better to find a great player in a foreign country and ask them to work for you, rather than relying exclusively on local talent. If it could be a remote job, this effect would be even greater.
idiotsecant 1y ago
'real' engineers can use SQL just fine. This is a strange position to take.
- collingreen 1y ago
  No true Scotsman would struggle with sql
  - idiotsecant 1y ago
    Are you saying that expecting software developers to know how to use SQL is a no true scotsman fallacy? What's next? Do we stop expecting people to know how FOR loops work?
    - collingreen 1y ago
      Do you think that's where I'm going with that? Expectations for people and for loops? > 'real' engineers can use SQL just fine. To be very explicit instead of snarky about my point: I think it is both factually incorrect and unnecessary gatekeeping to say real engineers know sql and imply not knowing sql marks someone as not a real engineer, hence the "no true Scotsman" fallacy. I probably agree with most of your opinions here within the context of the thread - I think good engineers can learn sql and other tools as necessary. I don't, however, think experience with any particular technology is a valid bar for a good or bad engineer at this point and I'm happy to speak against that when I see it.
adalacelove 1y ago
I'm a developer and: - I hate ORMs, they are the source for a lot of obscure errors behind layers and layers of abstractions. - I prefer analytical APIs for technical reasons, not just the language. Reasons: - I can compose queries, which in turn makes them easier to decompose - It's easier to spot errors - I avoid parsing SQL strings - It's easier to interact with the rest of the code, both functions and objects If I need to make just a query I gladly write SQL
- eru 1y ago
  Well, the problem in ORM is the O. Objection-orientation is just a worse way to organise your data and logic than relational algebra. It's just a shame that many languages don't support relational algebra well. We had relations as a datatype and all the relevant operations over them (like join) in a project I was working on. It was great! Very useful for expressing business logic.
  - tpm 1y ago
    The problem in ORM is the M, the mapping is always lossy and a leaky abstraction.
    - eru 1y ago
      Yes. So if you have the same thing in both the database and in your program, you don't need no mapping. And I suggest that having relations in both places is the way to go.
      - tpm 1y ago
        It's not enough, there is no way to represent some db stuff (not relational, more impl oriented) or some special queries in the program with an ORM. No ORM I have seen comes close to 100% of what you could do with the DB. So still the best way to get closest to the db is using raw queries in the program, but of course that does not scale with more tables, does not scale with more db engines, and does not scale with juniors taking over your codebase and not understanding it. So to conclude, objection-orientation is fine and relational is fine too, the issue is that the optimal way to translate between them does not scale.
sanderjd 1y ago
> But software developers often struggle to use sql Is this true? It doesn't seem true to me.
- winnie_ua 1y ago
  Oh, sweet summer child. Yes, there are so many so called developers in backend field of work who do not know how to do basic SQL. Anything bigger than s9imple WHERE clause. I wouldn't even talk about using indexes in database.
  - sanderjd 1y ago
    I've been doing this for over fifteen years. I haven't ever worked with anyone who is good at the job in general but can't figure out SQL.
ruszki 1y ago
What you can achieve with the standard SQL is taught on universities. The whole package. I’ve never met a developer, who struggled with that. When you use ORMs you need to follow SQL’s logic anyway. People use ORMs to avoid painful data conversions. Not to avoid the logic. Data engineers, DBAs, analysts, etc excel in specific databases, not in “SQL”.
- FranzFerdiNaN 1y ago
  Ive worked in BI and data engineering my whole career and I’ve met plenty of programmers who struggled immensely with SQL once it went further than select and group by. And don’t get me started about their database design skills. It’s way too often a disaster hidden behind “it works for the software so good enough”. Im more surprised by software engineers who do know these things than by the ones who don’t.
  - maccard 1y ago
    I’ve worked with gameplay programmers who can’t do simple 3D math, c++ programmers who fundamentally don’t understand pointers, backend developers who didn’t understand globals were shared state and cause race conditions, etc. It’s not that SQL is hard, it’s that for any discipline the vast majority of people don’t have a solid grasp of the tools they’re using. Ask most tradespeople about the underlying thing they’re working with and you’ll have the same problem.
atomicnumber3 1y ago
Aren't data engineers programmers? That is to say, a data engineer is-a software engineer? I share your sentiment though - I'm a data engineer (8 years) turned product engineer (3 years) and it astounds me how little SQL "normal" programmers know. It honestly changed my opinion on ORMs - it's not like the SQL people would write exceeds the basic select/filter/count patterns that is the most that non-data people know.
downrightmike 1y ago
They're all real programmers John

nathanfig 1y ago

Claude made this point while reviewing my blog for me: the mechanization of farms created a whole lot more specialization of roles. The person editing CAD diagrams of next year's combine harvester may not be a farmer strictly speaking, but farming is still where their livelihood comes from.

swader999 1y ago
Land is scarce though. The amount of software work that needs doing might not be, it could be infinite or probably more tied to electrical capacity.
deleted 1y ago
[deleted]
dredmorbius 1y ago
Strictly speaking, farming is where all our livelihoods come from, in the greatest part. We're all living off the surplus value of food production. (Also of other food, energy, and materials sourcing: fishing, forestry, mining, etc.) This was the insight of the French economist François Quesnay in his Tableau économique, foundation of the Physiocratic school of economics.
- eru 1y ago
  You might find https://en.wikipedia.org/wiki/Margarine#Coal_butter https://en.wikipedia.org/wiki/Margarine#Coal_butter fascinating. > Strictly speaking, farming is where all our livelihoods come from, in the greatest part. We're all living off the surplus value of food production. I don't think farming is special here, because food isn't special. You could make exactly the same argument for water (or even air) instead of food, and all of a sudden all our livelihoods would derive ultimately from the local municipal waterworks. Whether that's a reductio ad absurdum of the original argument, or a valuable new perspective on the local waterworks is left as an exercise to the reader.
  - dredmorbius 1y ago
    Food (and other fuels) are special. Water largely isn't fundamenally transformed with use (unless it's involved in a chemical reaction, though that's a minute fraction of all water usage), though it may be dispersed or degraded (usually contaminated with something). But it can recover its earlier state with appropriate applications of process and energy. Water (and much else we consume) is a material input rather than an energy input. With energy inputs it is the energy potential itself which provides value, and that potential is intrinsically consumed in their use. Water, wood, iron, aluminium, lithium, helium, etc., can all be recycled, restored to their useful state, at comparatively little cost. Collecting the waste products of food don't give you that, on two counts. First, most of the actual metabolic output is gaseous and lost to the atmosphere at large (CO2 and water vapour in your breath), and to the extent that solid and liquid human waste are useful in producing new food, it's a nutrient fertilisers which enable energy conversion of sunlight to fuel, and not the primary energy input itself (sunlight).
    - eru 1y ago
      I think you would want to be talking about entropy, not energy? Recycling all the materials you mentioned costs 'energy' (to use your terminology). The same for food: we can use used-up food and a lot of energy and grow new food. The process for 'recycling' wood is basically the same that for recycling food: you grow some plants. The waste products of used up wood are also basically the same as those for used up food. --- In any case, I don't see how any of this makes food more special than eg petrol or sunlight? And you can argue that food is only useful, if we have air, ie oxygen to burn it with.
  - bluGill 1y ago
    Except in a few places drinkable water is in such abundance that nobody every spent significant effort trying to get it. Likewise for air, few people have ever spent much effort getting air to breathe - even in the worst polluted areas bottled air was reserved for airplanes, hospitals (and a few people with medical conditions), and scuba divers.
    - 1y ago
lipowitz 1y ago
Removing jobs that could only be performed by those living near the particular fields with those that can be done anywhere makes jobs for the person willing to take the least satisfactory compensation for the most skill and work. Working the summer fields was one of the least desirable jobs but still gave local students with no particular skills a good supplemental income appropriate for whichever region.
- eru 1y ago
  Yes, but automating these away means that food becomes cheaper. We increase the overall total prosperity with that automation.
  - lipowitz 1y ago
    Increasing total prosperity is the wrong goal if distribution is completely unregulated. Investor and real estate owning classes like the 1% get more, the salaries can trend down because food costs are down, in a deflation spiral the youth are perpetual dependents and/or debtors who can't possibly earn enough over day to day costs given global competition includes people with no debts or debts from an economy that was less wealthy.
    - eru 1y ago
      When and where has that ever happened? Btw, most countries have taxes and welfare anyway.
- miki123211 1y ago
  depending on the job, it may also allow you to select for talent much better, which creates intense competition and raises salaries significantly. A good example of this phenomenon is sports. Even thought it can't be done remotely, it's so talent dependent that it's often better to find a great player in a foreign country and ask them to work for you, rather than relying exclusively on local talent. If it could be a remote job, this effect would be even greater.

idiotsecant 1y ago

'real' engineers can use SQL just fine. This is a strange position to take.

collingreen 1y ago
No true Scotsman would struggle with sql
- idiotsecant 1y ago
  Are you saying that expecting software developers to know how to use SQL is a no true scotsman fallacy? What's next? Do we stop expecting people to know how FOR loops work?
  - collingreen 1y ago
    Do you think that's where I'm going with that? Expectations for people and for loops? > 'real' engineers can use SQL just fine. To be very explicit instead of snarky about my point: I think it is both factually incorrect and unnecessary gatekeeping to say real engineers know sql and imply not knowing sql marks someone as not a real engineer, hence the "no true Scotsman" fallacy. I probably agree with most of your opinions here within the context of the thread - I think good engineers can learn sql and other tools as necessary. I don't, however, think experience with any particular technology is a valid bar for a good or bad engineer at this point and I'm happy to speak against that when I see it.

adalacelove 1y ago

I'm a developer and: - I hate ORMs, they are the source for a lot of obscure errors behind layers and layers of abstractions. - I prefer analytical APIs for technical reasons, not just the language. Reasons: - I can compose queries, which in turn makes them easier to decompose - It's easier to spot errors - I avoid parsing SQL strings - It's easier to interact with the rest of the code, both functions and objects If I need to make just a query I gladly write SQL

eru 1y ago
Well, the problem in ORM is the O. Objection-orientation is just a worse way to organise your data and logic than relational algebra. It's just a shame that many languages don't support relational algebra well. We had relations as a datatype and all the relevant operations over them (like join) in a project I was working on. It was great! Very useful for expressing business logic.
- tpm 1y ago
  The problem in ORM is the M, the mapping is always lossy and a leaky abstraction.
  - eru 1y ago
    Yes. So if you have the same thing in both the database and in your program, you don't need no mapping. And I suggest that having relations in both places is the way to go.
    - tpm 1y ago
      It's not enough, there is no way to represent some db stuff (not relational, more impl oriented) or some special queries in the program with an ORM. No ORM I have seen comes close to 100% of what you could do with the DB. So still the best way to get closest to the db is using raw queries in the program, but of course that does not scale with more tables, does not scale with more db engines, and does not scale with juniors taking over your codebase and not understanding it. So to conclude, objection-orientation is fine and relational is fine too, the issue is that the optimal way to translate between them does not scale.

sanderjd 1y ago

> But software developers often struggle to use sql Is this true? It doesn't seem true to me.

winnie_ua 1y ago
Oh, sweet summer child. Yes, there are so many so called developers in backend field of work who do not know how to do basic SQL. Anything bigger than s9imple WHERE clause. I wouldn't even talk about using indexes in database.
- sanderjd 1y ago
  I've been doing this for over fifteen years. I haven't ever worked with anyone who is good at the job in general but can't figure out SQL.

ruszki 1y ago

What you can achieve with the standard SQL is taught on universities. The whole package. I’ve never met a developer, who struggled with that. When you use ORMs you need to follow SQL’s logic anyway. People use ORMs to avoid painful data conversions. Not to avoid the logic. Data engineers, DBAs, analysts, etc excel in specific databases, not in “SQL”.

FranzFerdiNaN 1y ago
Ive worked in BI and data engineering my whole career and I’ve met plenty of programmers who struggled immensely with SQL once it went further than select and group by. And don’t get me started about their database design skills. It’s way too often a disaster hidden behind “it works for the software so good enough”. Im more surprised by software engineers who do know these things than by the ones who don’t.
- maccard 1y ago
  I’ve worked with gameplay programmers who can’t do simple 3D math, c++ programmers who fundamentally don’t understand pointers, backend developers who didn’t understand globals were shared state and cause race conditions, etc. It’s not that SQL is hard, it’s that for any discipline the vast majority of people don’t have a solid grasp of the tools they’re using. Ask most tradespeople about the underlying thing they’re working with and you’ll have the same problem.

atomicnumber3 1y ago

Aren't data engineers programmers? That is to say, a data engineer is-a software engineer? I share your sentiment though - I'm a data engineer (8 years) turned product engineer (3 years) and it astounds me how little SQL "normal" programmers know. It honestly changed my opinion on ORMs - it's not like the SQL people would write exceeds the basic select/filter/count patterns that is the most that non-data people know.

downrightmike 1y ago

They're all real programmers John