7 ms·

I always found it disheartening to read their changelogs and see how every release (I've looked at) fixes some "N+1 SELECTs" problem [1]. This will be an unpop

I always found it disheartening to read their changelogs and see how every release (I've looked at) fixes some "N+1 SELECTs" problem [1]. This will be an unpopular opinion here, but I feel we've (collectively) failed. Between ORMs and the dynamic language du jour, we have become so far removed from how computers work that most software development is a waste of electricity. We're using tens of servers and complex architectures for simple sites, just because we think that's the only way to scale them [2]. And our computers are slower than they ever were [3] [4]. I only gave a couple of examples, but look around, you can find them everywhere. Everything is awful. [1] The issue tracker is now down, but you can try it when it comes back: https://about.gitlab.com/releases/ https://about.gitlab.com/releases/. [2] https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos... [3] On older computers GNOME can't even keep up to update the mouse cursor. [4] I'm honestly not sure that Windows 10 boots faster from an SSD than XP from an HDD.

overcast 8y ago
SQL optimization is a dying art. Amazes me when I dig through software we've purchased to find abominations that I have to refactor. I've sent emails to developers with the fixes for shit we've paid for.
- pc86 8y ago
  Ah yes I love receiving Smug Reports[0] in my inbox. A 30,000' fix provided by someone with a 10' view of the code base and no business understanding outside their own organization. [0] https://blog.codinghorror.com/new-programming-jargon/ https://blog.codinghorror.com/new-programming-jargon/ Number 4
  - slacka 8y ago
    Could it be that you have the 30,000' view, while they had a ground view and climbed up a mountain to fix the view for all those in their town? Out of frustration I've broke out IDA Pro enough times to scratch my own itch that it's turned me to into a near Stallman-esque fan of open source software. When you work for small companies, bug reports on proprietary software issue trackers often get neglected.
  - overcast 8y ago
    Ah yes, I love receiving Smug Responses from developers who think they know what's best for their spaghetti code. I've been working with SQL longer than some of these kids have been alive. I know my shit, and I know when a query pings my slow query logs because of inefficient nonsense. Generally your end users have put it through far more paces, than you who probably don't use it in any production capacity. There's ZERO excuse for poor indexes, or NONE at all! Do what you want with the provided fix, because I've already applied to our systems to keep the server from crumbling.
  - throwawaymath 8y ago
    That seems like a pretty dismissive take. I've been on the other side of some truly ignorant bug reports, but I've also sent reports with code that, to the best of my knowledge, resolves the problem. If I received a bug report wherein someone was clearly annoyed with software they paid me for but still put the effort into trying to understand and resolve on their own, I'd probably prioritize that email and try to be empathetic about it.
- postila 8y ago
  Better, automated tools for optimization are needed. Postgres ecosystem right now lacks a lot of things that enterprise systems have for decades.
- tylerjwilk00 8y ago
  This. I fear the usage of ORM are slowing the web and developers SQL knowledge is atrophying along with it. I miss the days of crushing a query from 0.4s to 0.0014s. I've see some pretty ludicrous SQL execution plans generated by ORMs.
  - overcast 8y ago
    No joke, finding that eureka moment of cutting query times down a few orders a magnitude is one of my great pleasures.
xfour 8y ago
I really think you're on to something here. I see at my place of work adding additional complexity as something to be celebrated in terms of overcomplicated pipelines, "Big Data" for questionable reasons etc. I'm starting to become more and more sure this is just that people don't actually know how any of it works. My mantra that I'm thinking to myself is everything can and will fail and you're adding layers of things to will fail so your pipeline just becomes extremely brittle. And, while there's nothing more frustrating that an procedural dump of linear code, at least debugging it is possible. With these overly complex systems, your layers of abstraction just start fighting against you, often they'll be on different servers etc. All this is hard, and if it's not necessary don't just jump there.
- cirgue 8y ago
  In the tech industry, our ability to obtain new tools and find new problems to solve vastly outstrips the speed at which we can properly understand either the tools or the problems. When relevance in an organization is tied to your ability to leverage the new sexy, of course the equibrium behavior is to be half-cocked 100% of the time.
- andrewstuart2 8y ago
  I think it's a cargo cult problem. We see massive companies showing off their solutions to gigantic engineering problems that can only be solved in a certain way, and we think that we also must solve problems that way. Or that we'll get promoted if we solve problems "like <insert-big-name-tech-company>" in our architecture. Chances are, all you need is to properly use a relational database or another similar, proven, highly flexible and optimized tools, and you'll be fine until you have the time and money to solve the scale issues.
  - lovich 8y ago
    I think it might go deeper than just the engineers, although we are definitely a large part of it. The basic issue is that most companies don't compensate employees for compentently building software. Technical debt is always a future managers problem so it's both, never worked on and preventing it is never seen as a big accomplishment when evaluating an engineer. Secondly managers are people and people like flashy things. When someone says they built the website in vanilla js because it works and we won't need something more complicated for years, the manager isn't going to get that excited or remember when raise time comes around. If the engineer says they rewrote an application from [legacy language] to some hip new language that the manager has heard about from their peers in other companies who are also rewriting their applications, it's going to stick in the managers head when they are rating their employees. You get what you pay for and modern companies are paying for semi working but flashy software more than they pay for solid but boring software
majidazimi 8y ago
> This will be an unpopular opinion here, but I feel we've (collectively) failed. We? NO. But those who refrained to learn SQL, yes. They've egregiously failed.
somethingsimple 8y ago
> This will be an unpopular opinion here, but I feel we've (collectively) failed. Between ORMs and the dynamic language du jour, we have become so far removed from how computers work that most software development is a waste of electricity. I just wanted to say that I was kinda relieved reading that, because I often feel like I’m the only person who thinks that.
- kgdinesh 8y ago
  you guys need to read this blog https://mechanical-sympathy.blogspot.com/ https://mechanical-sympathy.blogspot.com/
dtech 8y ago
Why is this general rant on the state of software development relevant to a Gitlab outage?
- GrayShade 8y ago
  This issue is different, but gitlab.com usually has poor availability which I'm sure it's caused in part by technical decisions they made.
  - jrs95 8y ago
    Is it? There are plenty of Rails sites, Basecamp included, that have pretty high uptime. GitHub is also based on Rails, and while it could be better, it is substantially more stable than GitLab IMO.
- pknopf 8y ago
  The constant issues GitLab has had with updates/deployments echoes back to a common thread that many people feel about software in general, which is that we have needleslly over-complicated things, making things brittle. Wether it's GitLab's stack that is brittle, or that the team doesn't have experience dealing with growing pains, it doesn't change the point.
- throwawaymath 8y ago
  The parent commenter is saying (or rather, insinuating) that GitLab's outage is symptomatic of a broader problem in the software industry, vis á vis unnecessary complexity. I think it was worded a bit harshly, but there's a kernel of truth there. I don't mean to pile onto GitLab in particular, but take a look at this open proposal for hardware upgrades at GitLab in late 2016: https://about.gitlab.com/2016/12/11/proposed-server-purchase-for-gitlab-com/ https://about.gitlab.com/2016/12/11/proposed-server-purchase.... I don't think I can do a better job of explaining what's wrong with that proposal than this thread, from the ensuing HN discussion: https://news.ycombinator.com/item?id=13153860 https://news.ycombinator.com/item?id=13153860. Our industry is plagued with architecture decisions that pile layers and layers of complexity onto otherwise straightforward technical problems in pursuit of scalable abstraction. Even companies that legitimately have to think about "scale" choose architectures which technically impressive but brittle in practice.
adtac 8y ago
This is why I have a mild dislike for ORMs. They're pretty handy, but most of the time, I prefer to hand-make SQL statements. Are they more work? Yes. But no ORM will ever give me the actual flexibility and performance of raw SQL queries (unless it becomes SQL itself).
- GrayShade 8y ago
  I'm fine with "thin" ORMs that map rows to objects, and not much more. You can just write SQL (or use a view) while avoiding some repetitive code.
- dimgl 8y ago
  In a shop I used to work at all of the devs were looking into Entity Framework as an abstraction layer for access to an MSSQL database. They were using T4 generators to create classes based off of the tables they were interested in (it was hundreds) and so they were looking into unit of work/repository patterns. I told them I had plenty of experience going down this route (my most popular StackOverflow question is on UOW/repo) and to just use Dapper and not look back. There is a pleasant simplicity in using plain old C# classes and simple queries that you can visualize just by looking at them without having to run a trace to figure out what's actually being executed. This was one battle of many. I lost that battle.
  - GrayShade 8y ago
    You'll love this: https://www.earthli.com/news/view_article.php?id=2288 https://www.earthli.com/news/view_article.php?id=2288 > This was one battle of many. I lost that battle. Have a hug :-).
    - spencercw 8y ago
      We use Hibernate at work and I've seen it generate some pretty gnarly queries (like, tens of kilobytes of just SQL). People like to rag on Oracle, but it's kind of impressive that it stands up to such abuse without complaining.
- 8y ago
btrettel 8y ago
The Jevons paradox applies to computers [1]. Additional computational power is mostly wasted. That's why I use Xfce. The additional bloat doesn't accomplish anything in my view. (And I might switch to something lighter in the future.) [1] https://en.wikipedia.org/wiki/Jevons_paradox https://en.wikipedia.org/wiki/Jevons_paradox
- nextos 8y ago
  Regarding Xfce and lightweight desktops, no desktop environment, a few services, X, a tiling window manager (e.g. stumpwm or i3) and a terminal can easily fit into less than 256 MB of RAM. My Arch and Nix setups are in fact close to 128. When casual users insist on needing 16 GB of RAM to surf the web, watch videos and chat it makes me cringe.
  - greggyb 8y ago
    Regarding RAM consumption. I'm on a Windows box right now. The Windows Desktop Manager takes ~75MB RAM, but my Firefox process (with most tabs suspended) takes >2GB RAM. I run into the same on my Linux and BSD machines at home. I run i3 and the XFCE settings daemon for convenience. Sure, I could optimize more, but it's really not the area where I'll see a lot of benefit. My best optimization for standard users is an automatic tab suspender.
deleted 8y ago
[deleted]
rovek 8y ago
I don't know what you mean; I've just been working on 3 Node lambdas, deployed by CircleCI to AWS infrastructure provisioned by terraform which take a CSV from an email and output multiple CSVs.
- jedberg 8y ago
  Hey, at least you know the system will probably scale when you have to deal with a million emails and ten million outputs!
remyp 8y ago
I consider ORMs to be a form of technical debt -- they're a great way to get to market quickly. Unfortunately like most technical debt, it usually just gets ignored until it's completely unsustainable.
noncoml 8y ago
Yes and no. When starting a new project it makes absolutely sense to use ORM and a dynamic language; the cost of "wasted electricity" is negligible in front of the development cost saved.
smarx007 8y ago
I think the better way to put it that "we" stopped caring about performance both on the client (think Electron) and on the server (ORMs & dynamic languages, as already mentioned) at some point of time in favour of faster time to market, cheaper. Just so that the output is "good enough". But honestly, I agree with that approach to a certain (!) extent. I don't want to write all of my backend in C++. I use Java or Python instead. Thousand reasons: maven/pip, portability etc. So, I would say we need to come to terms with ourselves and agree that the productivity is sometimes increased at the expense of the system quality. After that, I think it's okay to keep doing that if both us and our customers are OK with the tradeoff. Just to illustrate: do I like the RAM consumption & the startup time of my Slack app? No. But I am a free user, so I can't complain. In the end, the Slack team probably made the right choice. UPD: 500 error is gone. Gitlab also needs to manage costs and be friendly to the OSS contributors. Ruby was also probably a good choice for them. If you want top-notch software, be prepared to stomach NASA-grade costs, because the rumour is that 1 LOC in Shuttle ended up costing $1500 [1]. P.S. Thanks a lot for the [2] paper reference! [1] https://www.johndcook.com/blog/2009/10/08/nasa-buggy-software/ https://www.johndcook.com/blog/2009/10/08/nasa-buggy-softwar...
- xigency 8y ago
  > I don't want to write all of my backend in C++. I use Java or Python instead. Thousand reasons: maven/pip, portability etc. Is this really a good excuse with containers being used so frequently?
  - smarx007 8y ago
    Maven/pip is still a good excuse. I don't want to wait x months until a new release of some library lands in Debian repos. Portability – not so much, but then it saves you so much headache because switching from 16.04 to 18.04 should not break anything in your system.
- isostatic 8y ago
  > rumour is that 1 LOC in Shuttle ended up costing $1500 [1]. Should have used K&R braces
raziel2p 8y ago
I don't think it's a matter of not understanding how SQL selects work etc., it's more that ORMs don't make it easy to find all the places in our code which might need this relationship prefetched. The fact that it's often a simple fix once you've found a place like that is something to celebrate, honestly.
drinchev 8y ago
Unfortunately the smartest minds in CS are exactly suggesting “don’t reinvent the wheel”. Maybe your comment is quite relevant for GitLab, because of their admirable publicity on their outages and code base, but I’m quite sure that even your TV has the same weird layers over layers that do something for the sake of “ease of developing” I’m always surprised when I see a working complicated software, because I know that lots of projects, today are duck-taped patchwork.
- majidazimi 8y ago
  > Unfortunately the smartest minds in CS are exactly suggesting “don’t reinvent the wheel” Smartest minds never said that. Here is the correct quote: "Don't reinvent the wheel, but please learn how a wheel works".
btown 8y ago
Well-implemented internal GraphQL APIs, and intelligent caching on the client, are a step in the right direction, separating query optimization from the data needed by the view layer. (Though you could argue that this is wasting cycles to have that level of abstraction, but abstraction can be useful in its own right.) This is still ongoing work; for instance, see https://github.com/tfoxy/graphene-django-optimizer https://github.com/tfoxy/graphene-django-optimizer which is in progress as of two days ago, and attempts to do away with the N+1 problem systematically. But more generally, if you're worried about this, choose to work on projects whose impact on the world (however you want to define it) is greater than the electricity your servers use. And if you're working in that direction, your speed to market is much more important than your electricity cost, or your support of legacy hardware. Design abstractions (ORMs, APIs) so that they can be optimized in the future, imagine that they'll be rewritten at that point in Go or Rust by dedicated teams, but in the meantime, stay laser-focused on your vision. This applies whether you're at the smallish vNext team at Microsoft, or a startup nobody's heard about. It's "performance debt," and it's often the most efficient way to capitalize your initiative.
kenforthewin 8y ago
1. This is not an unpopular opinion on HN, quite the opposite, I see comments like this every day 2. Don't attribute a failure of a programmer to the ORM. Tech like ORMs allow more people access to the underlying technology, yes. So there will be more errors like N+1. But that's not the purpose of ORM, just a side effect. The purpose is to speed development for those who have already learned SQL. Can't blame an N+1 error on the ORM unless it somehow prohibits you from addressing such a problem, which no modern ORM does.
- bjpbakker 8y ago
  > Don't attribute a failure of a programmer to the ORM While I sympathize with this, I would like ORMs to make it a lot harder to make these mistakes. Especially frameworks for dynamic languages (in my experience) have performance issues by default (and when following their docs). That should probably change.
adreamingsoul 8y ago
I spend most of my engineering time on the front-end, and I 100% agree. We are wasting so many resources that I'm genuinely disgusted by it.
ajross 8y ago
Interestingly, git itself is a counterexample to your very point. It's written in reasonably tight code and without major runtime dependencies to run fast on a single system with a simple filesystem backing. And depending on who you ask and what their tastes are, it does maybe 80% of the job that all these giant web sites are being asked to do internally, without any rigging or support beyond a remote ssh account. So "we" haven't collectively failed, just some of us.
jgalentine007 8y ago
I don't think orms are really to blame - can you imagine all of the sql injected,barely functioning select/joins the internet would be dealing with right now? The real problem: lazy developers are going to be lazy!
superflyguy 8y ago
This seems somewhat pertinent: https://www.stilldrinking.org/programming-sucks https://www.stilldrinking.org/programming-sucks
OliverJones 8y ago
All this complexity is a sure sign that our trade is growing up. We've been around for long enough that the world depends on multigenerational legacy systems we've created. The economics of developing and maintaining all kinds of systems push us in the direction of easy-to-inspect and easy-to-develop code rather than highly optimized code. We use existing systems (like SQL, like UNIX derivatives, like various internet protocols, like git, like Ruby, Javascript and the rest) because they're almost completely debugged, even if they're sometimes clunky. Even if it's hard to maintain our mission-critical legacy systems, it's not as hard as maintaining the signaling in the New York City subways. Yet. And nobody HAS to use Gnome or the wizziest GUI sugar on other systems. Yes, it's discouraging when things don't work smoothly. But our job, as hackers, is to do our best to keep things running.
mschuster91 8y ago
> [4] I'm honestly not sure that Windows 10 boots faster from an SSD than XP from an HDD. Win 10 LTSB on HDD would leave XP on SSD dead in the water, I believe - the sheer amount of crap that Windows has fed into the consumer and even the professional Windows editions, though, is unbelievable.
Jach 8y ago
This complaint of software eating up all the gains made by hardware has been noted since the 80s: https://en.wikipedia.org/wiki/Wirth%27s_law https://en.wikipedia.org/wiki/Wirth%27s_law Which is to say there's not going to be any sort of "victory" industry-wide here. Still, ranting is fun. My favorite recent-ish writing is linked from https://twitter.com/CarloPescio/status/903603552890834947 https://twitter.com/CarloPescio/status/903603552890834947 Programming as if the Domain (and Performance) Mattered You can learn new things by keeping up with rants too. The best algorithm depends not only on the broad architecture but even the microarchitecture. Newer generations of Intel CPUs for instance throw many older optimizations out the window, but you'll still get the occasional suggestions for those optimizations from people who haven't kept up to date. I'm not sure Ruby counts as a dynamic language du jour given how old it is, but I wish more dynamic languages followed the Lisp tradition instead of the interpreted Basic tradition, especially with capabilities that let you optimize dynamically as needed. * (defun add-opt (x y) (declare (optimize (speed 3) (space 3) (safety 0) (debug 0)) ; not all these matter in this example, just showing options (type fixnum x y)) (the fixnum (+ x y))) ADD-OPT * (disassemble 'add-opt) ; disassembly for ADD-OPT ; Size: 9 bytes ; 02D0C2FF: 4801FA ADD RDX, RDI ; no-arg-parsing entry point ; 302: 488BE5 MOV RSP, RBP ; 305: F8 CLC ; 306: 5D POP RBP ; 307: C3 RET
- bjpbakker 8y ago
  > I'm not sure Ruby counts as a dynamic language du jour given how old it is I believe it still did when Gitlab started
cthart 8y ago
Agree completely here. 20 years ago I was involved in a project that rolled out an integrated SAP/GIS system. The GIS server had 3Gb RAM and was able to support 100 users. These days we can't even roll out a client computer with that little RAM -- and the end users can't get any more work done than they could 20 years ago.

overcast 8y ago

SQL optimization is a dying art. Amazes me when I dig through software we've purchased to find abominations that I have to refactor. I've sent emails to developers with the fixes for shit we've paid for.

pc86 8y ago
Ah yes I love receiving Smug Reports[0] in my inbox. A 30,000' fix provided by someone with a 10' view of the code base and no business understanding outside their own organization. [0] https://blog.codinghorror.com/new-programming-jargon/ https://blog.codinghorror.com/new-programming-jargon/ Number 4
- slacka 8y ago
  Could it be that you have the 30,000' view, while they had a ground view and climbed up a mountain to fix the view for all those in their town? Out of frustration I've broke out IDA Pro enough times to scratch my own itch that it's turned me to into a near Stallman-esque fan of open source software. When you work for small companies, bug reports on proprietary software issue trackers often get neglected.
- overcast 8y ago
  Ah yes, I love receiving Smug Responses from developers who think they know what's best for their spaghetti code. I've been working with SQL longer than some of these kids have been alive. I know my shit, and I know when a query pings my slow query logs because of inefficient nonsense. Generally your end users have put it through far more paces, than you who probably don't use it in any production capacity. There's ZERO excuse for poor indexes, or NONE at all! Do what you want with the provided fix, because I've already applied to our systems to keep the server from crumbling.
- throwawaymath 8y ago
  That seems like a pretty dismissive take. I've been on the other side of some truly ignorant bug reports, but I've also sent reports with code that, to the best of my knowledge, resolves the problem. If I received a bug report wherein someone was clearly annoyed with software they paid me for but still put the effort into trying to understand and resolve on their own, I'd probably prioritize that email and try to be empathetic about it.
postila 8y ago
Better, automated tools for optimization are needed. Postgres ecosystem right now lacks a lot of things that enterprise systems have for decades.
tylerjwilk00 8y ago
This. I fear the usage of ORM are slowing the web and developers SQL knowledge is atrophying along with it. I miss the days of crushing a query from 0.4s to 0.0014s. I've see some pretty ludicrous SQL execution plans generated by ORMs.
- overcast 8y ago
  No joke, finding that eureka moment of cutting query times down a few orders a magnitude is one of my great pleasures.

xfour 8y ago

I really think you're on to something here. I see at my place of work adding additional complexity as something to be celebrated in terms of overcomplicated pipelines, "Big Data" for questionable reasons etc. I'm starting to become more and more sure this is just that people don't actually know how any of it works. My mantra that I'm thinking to myself is everything can and will fail and you're adding layers of things to will fail so your pipeline just becomes extremely brittle. And, while there's nothing more frustrating that an procedural dump of linear code, at least debugging it is possible. With these overly complex systems, your layers of abstraction just start fighting against you, often they'll be on different servers etc. All this is hard, and if it's not necessary don't just jump there.

cirgue 8y ago
In the tech industry, our ability to obtain new tools and find new problems to solve vastly outstrips the speed at which we can properly understand either the tools or the problems. When relevance in an organization is tied to your ability to leverage the new sexy, of course the equibrium behavior is to be half-cocked 100% of the time.
andrewstuart2 8y ago
I think it's a cargo cult problem. We see massive companies showing off their solutions to gigantic engineering problems that can only be solved in a certain way, and we think that we also must solve problems that way. Or that we'll get promoted if we solve problems "like <insert-big-name-tech-company>" in our architecture. Chances are, all you need is to properly use a relational database or another similar, proven, highly flexible and optimized tools, and you'll be fine until you have the time and money to solve the scale issues.
- lovich 8y ago
  I think it might go deeper than just the engineers, although we are definitely a large part of it. The basic issue is that most companies don't compensate employees for compentently building software. Technical debt is always a future managers problem so it's both, never worked on and preventing it is never seen as a big accomplishment when evaluating an engineer. Secondly managers are people and people like flashy things. When someone says they built the website in vanilla js because it works and we won't need something more complicated for years, the manager isn't going to get that excited or remember when raise time comes around. If the engineer says they rewrote an application from [legacy language] to some hip new language that the manager has heard about from their peers in other companies who are also rewriting their applications, it's going to stick in the managers head when they are rating their employees. You get what you pay for and modern companies are paying for semi working but flashy software more than they pay for solid but boring software

majidazimi 8y ago

> This will be an unpopular opinion here, but I feel we've (collectively) failed. We? NO. But those who refrained to learn SQL, yes. They've egregiously failed.

somethingsimple 8y ago

> This will be an unpopular opinion here, but I feel we've (collectively) failed. Between ORMs and the dynamic language du jour, we have become so far removed from how computers work that most software development is a waste of electricity. I just wanted to say that I was kinda relieved reading that, because I often feel like I’m the only person who thinks that.

kgdinesh 8y ago
you guys need to read this blog https://mechanical-sympathy.blogspot.com/ https://mechanical-sympathy.blogspot.com/

dtech 8y ago

Why is this general rant on the state of software development relevant to a Gitlab outage?

GrayShade 8y ago
This issue is different, but gitlab.com usually has poor availability which I'm sure it's caused in part by technical decisions they made.
- jrs95 8y ago
  Is it? There are plenty of Rails sites, Basecamp included, that have pretty high uptime. GitHub is also based on Rails, and while it could be better, it is substantially more stable than GitLab IMO.
pknopf 8y ago
The constant issues GitLab has had with updates/deployments echoes back to a common thread that many people feel about software in general, which is that we have needleslly over-complicated things, making things brittle. Wether it's GitLab's stack that is brittle, or that the team doesn't have experience dealing with growing pains, it doesn't change the point.
throwawaymath 8y ago
The parent commenter is saying (or rather, insinuating) that GitLab's outage is symptomatic of a broader problem in the software industry, vis á vis unnecessary complexity. I think it was worded a bit harshly, but there's a kernel of truth there. I don't mean to pile onto GitLab in particular, but take a look at this open proposal for hardware upgrades at GitLab in late 2016: https://about.gitlab.com/2016/12/11/proposed-server-purchase-for-gitlab-com/ https://about.gitlab.com/2016/12/11/proposed-server-purchase.... I don't think I can do a better job of explaining what's wrong with that proposal than this thread, from the ensuing HN discussion: https://news.ycombinator.com/item?id=13153860 https://news.ycombinator.com/item?id=13153860. Our industry is plagued with architecture decisions that pile layers and layers of complexity onto otherwise straightforward technical problems in pursuit of scalable abstraction. Even companies that legitimately have to think about "scale" choose architectures which technically impressive but brittle in practice.

adtac 8y ago

This is why I have a mild dislike for ORMs. They're pretty handy, but most of the time, I prefer to hand-make SQL statements. Are they more work? Yes. But no ORM will ever give me the actual flexibility and performance of raw SQL queries (unless it becomes SQL itself).

GrayShade 8y ago
I'm fine with "thin" ORMs that map rows to objects, and not much more. You can just write SQL (or use a view) while avoiding some repetitive code.
dimgl 8y ago
In a shop I used to work at all of the devs were looking into Entity Framework as an abstraction layer for access to an MSSQL database. They were using T4 generators to create classes based off of the tables they were interested in (it was hundreds) and so they were looking into unit of work/repository patterns. I told them I had plenty of experience going down this route (my most popular StackOverflow question is on UOW/repo) and to just use Dapper and not look back. There is a pleasant simplicity in using plain old C# classes and simple queries that you can visualize just by looking at them without having to run a trace to figure out what's actually being executed. This was one battle of many. I lost that battle.
- GrayShade 8y ago
  You'll love this: https://www.earthli.com/news/view_article.php?id=2288 https://www.earthli.com/news/view_article.php?id=2288 > This was one battle of many. I lost that battle. Have a hug :-).
  - spencercw 8y ago
    We use Hibernate at work and I've seen it generate some pretty gnarly queries (like, tens of kilobytes of just SQL). People like to rag on Oracle, but it's kind of impressive that it stands up to such abuse without complaining.
8y ago

btrettel 8y ago

The Jevons paradox applies to computers [1]. Additional computational power is mostly wasted. That's why I use Xfce. The additional bloat doesn't accomplish anything in my view. (And I might switch to something lighter in the future.) [1] https://en.wikipedia.org/wiki/Jevons_paradox https://en.wikipedia.org/wiki/Jevons_paradox

nextos 8y ago
Regarding Xfce and lightweight desktops, no desktop environment, a few services, X, a tiling window manager (e.g. stumpwm or i3) and a terminal can easily fit into less than 256 MB of RAM. My Arch and Nix setups are in fact close to 128. When casual users insist on needing 16 GB of RAM to surf the web, watch videos and chat it makes me cringe.
- greggyb 8y ago
  Regarding RAM consumption. I'm on a Windows box right now. The Windows Desktop Manager takes ~75MB RAM, but my Firefox process (with most tabs suspended) takes >2GB RAM. I run into the same on my Linux and BSD machines at home. I run i3 and the XFCE settings daemon for convenience. Sure, I could optimize more, but it's really not the area where I'll see a lot of benefit. My best optimization for standard users is an automatic tab suspender.

deleted 8y ago

[deleted]

rovek 8y ago

I don't know what you mean; I've just been working on 3 Node lambdas, deployed by CircleCI to AWS infrastructure provisioned by terraform which take a CSV from an email and output multiple CSVs.

jedberg 8y ago
Hey, at least you know the system will probably scale when you have to deal with a million emails and ten million outputs!

remyp 8y ago

I consider ORMs to be a form of technical debt -- they're a great way to get to market quickly. Unfortunately like most technical debt, it usually just gets ignored until it's completely unsustainable.

noncoml 8y ago

Yes and no. When starting a new project it makes absolutely sense to use ORM and a dynamic language; the cost of "wasted electricity" is negligible in front of the development cost saved.

smarx007 8y ago

I think the better way to put it that "we" stopped caring about performance both on the client (think Electron) and on the server (ORMs & dynamic languages, as already mentioned) at some point of time in favour of faster time to market, cheaper. Just so that the output is "good enough". But honestly, I agree with that approach to a certain (!) extent. I don't want to write all of my backend in C++. I use Java or Python instead. Thousand reasons: maven/pip, portability etc. So, I would say we need to come to terms with ourselves and agree that the productivity is sometimes increased at the expense of the system quality. After that, I think it's okay to keep doing that if both us and our customers are OK with the tradeoff. Just to illustrate: do I like the RAM consumption & the startup time of my Slack app? No. But I am a free user, so I can't complain. In the end, the Slack team probably made the right choice. UPD: 500 error is gone. Gitlab also needs to manage costs and be friendly to the OSS contributors. Ruby was also probably a good choice for them. If you want top-notch software, be prepared to stomach NASA-grade costs, because the rumour is that 1 LOC in Shuttle ended up costing $1500 [1]. P.S. Thanks a lot for the [2] paper reference! [1] https://www.johndcook.com/blog/2009/10/08/nasa-buggy-software/ https://www.johndcook.com/blog/2009/10/08/nasa-buggy-softwar...

xigency 8y ago
> I don't want to write all of my backend in C++. I use Java or Python instead. Thousand reasons: maven/pip, portability etc. Is this really a good excuse with containers being used so frequently?
- smarx007 8y ago
  Maven/pip is still a good excuse. I don't want to wait x months until a new release of some library lands in Debian repos. Portability – not so much, but then it saves you so much headache because switching from 16.04 to 18.04 should not break anything in your system.
isostatic 8y ago
> rumour is that 1 LOC in Shuttle ended up costing $1500 [1]. Should have used K&R braces

raziel2p 8y ago

I don't think it's a matter of not understanding how SQL selects work etc., it's more that ORMs don't make it easy to find all the places in our code which might need this relationship prefetched. The fact that it's often a simple fix once you've found a place like that is something to celebrate, honestly.

drinchev 8y ago

Unfortunately the smartest minds in CS are exactly suggesting “don’t reinvent the wheel”. Maybe your comment is quite relevant for GitLab, because of their admirable publicity on their outages and code base, but I’m quite sure that even your TV has the same weird layers over layers that do something for the sake of “ease of developing” I’m always surprised when I see a working complicated software, because I know that lots of projects, today are duck-taped patchwork.

majidazimi 8y ago
> Unfortunately the smartest minds in CS are exactly suggesting “don’t reinvent the wheel” Smartest minds never said that. Here is the correct quote: "Don't reinvent the wheel, but please learn how a wheel works".

btown 8y ago

Well-implemented internal GraphQL APIs, and intelligent caching on the client, are a step in the right direction, separating query optimization from the data needed by the view layer. (Though you could argue that this is wasting cycles to have that level of abstraction, but abstraction can be useful in its own right.) This is still ongoing work; for instance, see https://github.com/tfoxy/graphene-django-optimizer https://github.com/tfoxy/graphene-django-optimizer which is in progress as of two days ago, and attempts to do away with the N+1 problem systematically. But more generally, if you're worried about this, choose to work on projects whose impact on the world (however you want to define it) is greater than the electricity your servers use. And if you're working in that direction, your speed to market is much more important than your electricity cost, or your support of legacy hardware. Design abstractions (ORMs, APIs) so that they can be optimized in the future, imagine that they'll be rewritten at that point in Go or Rust by dedicated teams, but in the meantime, stay laser-focused on your vision. This applies whether you're at the smallish vNext team at Microsoft, or a startup nobody's heard about. It's "performance debt," and it's often the most efficient way to capitalize your initiative.

kenforthewin 8y ago

1. This is not an unpopular opinion on HN, quite the opposite, I see comments like this every day 2. Don't attribute a failure of a programmer to the ORM. Tech like ORMs allow more people access to the underlying technology, yes. So there will be more errors like N+1. But that's not the purpose of ORM, just a side effect. The purpose is to speed development for those who have already learned SQL. Can't blame an N+1 error on the ORM unless it somehow prohibits you from addressing such a problem, which no modern ORM does.

bjpbakker 8y ago
> Don't attribute a failure of a programmer to the ORM While I sympathize with this, I would like ORMs to make it a lot harder to make these mistakes. Especially frameworks for dynamic languages (in my experience) have performance issues by default (and when following their docs). That should probably change.

adreamingsoul 8y ago

I spend most of my engineering time on the front-end, and I 100% agree. We are wasting so many resources that I'm genuinely disgusted by it.

ajross 8y ago

Interestingly, git itself is a counterexample to your very point. It's written in reasonably tight code and without major runtime dependencies to run fast on a single system with a simple filesystem backing. And depending on who you ask and what their tastes are, it does maybe 80% of the job that all these giant web sites are being asked to do internally, without any rigging or support beyond a remote ssh account. So "we" haven't collectively failed, just some of us.

jgalentine007 8y ago

I don't think orms are really to blame - can you imagine all of the sql injected,barely functioning select/joins the internet would be dealing with right now? The real problem: lazy developers are going to be lazy!

superflyguy 8y ago

This seems somewhat pertinent: https://www.stilldrinking.org/programming-sucks https://www.stilldrinking.org/programming-sucks

OliverJones 8y ago

All this complexity is a sure sign that our trade is growing up. We've been around for long enough that the world depends on multigenerational legacy systems we've created. The economics of developing and maintaining all kinds of systems push us in the direction of easy-to-inspect and easy-to-develop code rather than highly optimized code. We use existing systems (like SQL, like UNIX derivatives, like various internet protocols, like git, like Ruby, Javascript and the rest) because they're almost completely debugged, even if they're sometimes clunky. Even if it's hard to maintain our mission-critical legacy systems, it's not as hard as maintaining the signaling in the New York City subways. Yet. And nobody HAS to use Gnome or the wizziest GUI sugar on other systems. Yes, it's discouraging when things don't work smoothly. But our job, as hackers, is to do our best to keep things running.

mschuster91 8y ago

> [4] I'm honestly not sure that Windows 10 boots faster from an SSD than XP from an HDD. Win 10 LTSB on HDD would leave XP on SSD dead in the water, I believe - the sheer amount of crap that Windows has fed into the consumer and even the professional Windows editions, though, is unbelievable.

Jach 8y ago

This complaint of software eating up all the gains made by hardware has been noted since the 80s: https://en.wikipedia.org/wiki/Wirth%27s_law https://en.wikipedia.org/wiki/Wirth%27s_law Which is to say there's not going to be any sort of "victory" industry-wide here. Still, ranting is fun. My favorite recent-ish writing is linked from https://twitter.com/CarloPescio/status/903603552890834947 https://twitter.com/CarloPescio/status/903603552890834947 Programming as if the Domain (and Performance) Mattered You can learn new things by keeping up with rants too. The best algorithm depends not only on the broad architecture but even the microarchitecture. Newer generations of Intel CPUs for instance throw many older optimizations out the window, but you'll still get the occasional suggestions for those optimizations from people who haven't kept up to date. I'm not sure Ruby counts as a dynamic language du jour given how old it is, but I wish more dynamic languages followed the Lisp tradition instead of the interpreted Basic tradition, especially with capabilities that let you optimize dynamically as needed. * (defun add-opt (x y) (declare (optimize (speed 3) (space 3) (safety 0) (debug 0)) ; not all these matter in this example, just showing options (type fixnum x y)) (the fixnum (+ x y))) ADD-OPT * (disassemble 'add-opt) ; disassembly for ADD-OPT ; Size: 9 bytes ; 02D0C2FF: 4801FA ADD RDX, RDI ; no-arg-parsing entry point ; 302: 488BE5 MOV RSP, RBP ; 305: F8 CLC ; 306: 5D POP RBP ; 307: C3 RET

bjpbakker 8y ago
> I'm not sure Ruby counts as a dynamic language du jour given how old it is I believe it still did when Gitlab started

cthart 8y ago

Agree completely here. 20 years ago I was involved in a project that rolled out an integrated SAP/GIS system. The GIS server had 3Gb RAM and was able to support 100 users. These days we can't even roll out a client computer with that little RAM -- and the end users can't get any more work done than they could 20 years ago.