8 ms·

Every time this topic comes up, it reminds me of a web app I wrote back around 2007 that was deployed to a over 2000 locations. I deliberately used "boring" tec

Every time this topic comes up, it reminds me of a web app I wrote back around 2007 that was deployed to a over 2000 locations. I deliberately used "boring" technologies. The entire front-end used under 100 lines of JavaScript. The backend was simply SQL Server, and the queries were written in SQL instead of some ORM. The output was just HTML. No special tooling was used, no "minification" or "tree shaking", or any such thing. Just hit the build button and "copy to deploy". For about a decade I used to turn up to that customer annually for "maintenance", which primarily involved importing some CSVs that changed every year, and also updating the logo images and icons to match any rebrands. In that time the system had two million users, went through 4 OS upgrades, 3 database upgrades, and went through the 32-bit to 64-bit upgrade also. The underlying runtime had 3 or 4 major updates, depending on how you count it. Zero outages, no problems, only the occasional performance regression that could be fixed by poking the database statistics to get things back in their groove. The problem was... You see, all of the above was a problem, because it didn't keep me employed. I was not the "hero" for saving the day. Entire teams of people weren't involved. There was no visibility at the senior management level. Nobody got scared, or had to throw money at it, or hire consultants to review it. So it had to go. It was replaced by a system that cost about 500x as much (a 9-digit sum), got rolled back for failing to meet requirements, and then got additional funding and was eventually forced upon its hapless users. That, apparently, was doing things "properly". That got everybody involved. Everyone got their beak wet. All the way up to government ministers and their lobbyists. Multiple consultancies were engaged. Reports. Audits. Reviews. This is why we can't have simplicity: because it doesn't scale.

agumonkey 4y ago
> You see, all of the above was a problem, because it didn't keep me employed. I was not the "hero" for saving the day. Entire teams of people weren't involved. There was no visibility at the senior management level. Nobody got scared, or had to throw money at it, or hire consultants to review it. there are a few other instances of that: - an old article about Michelin (french tire manufacturer) quoting some scientist of theirs "We can make a million hour tire.. but what would we sell" - recently people said their rust code cause too much downtime for coders because it was too stable too early flip side of the same issue: - very often people game their work to ensure benefits: stash duties for later so you can appear busy, or overwhelmed (and claim promotion because you have so much to do) The global system doesn't reward to true optimization, it allocates people on useless tasks, at best for lower risk, but smart people doing things solid and fast could be using their talents on other problems.
bdg 4y ago
I understand the annoyance of it being replaced for a more complex more expensive system, but I would also like to know: What was the reasoning provided and what did decision makers truly believe about the whole thing?
- jiggawatts 4y ago
  > What was the reasoning provided Reasoning? Hah-ha. Haaaaa... It was an open government tender process, for which I, or the company I worked for was not eligible, despite the tender being "open". You see, a decade-long pedigree of actually having implemented the software used for this purpose did not qualify us for replacing it with a v2.0. There are rules, you see? They have to be followed! Or else. Or else bad things might happen, like money being wasted. The fact that the end-result of this process was that a 9-digit sum was spent on something I spat out in my spare time in under a year -- and was used for a decade -- was of no relevance. > what did decision makers truly believe about the whole thing? Their concerns started only when the whole thing blew up and started making headlines. Then nothing happened to them personally, so their concerns evaporated along with the taxpayer funds they had wasted.
  - listenallyall 4y ago
    I think most readers, including myself, empathize with you and understand the frustration and absurdity. But you are also telling just one side of the story (yours) and I imagine that v.2.0 specs had certain requirements and features, possibly required by legislation, that needed to be followed and implemented. When you say, dismissively, There are rules, you see? They have to be followed! that's when I, and likely others, start to wonder if you are really providing the full story, or if you actually even understand the differences between your simple app and the updated version.
    - majkinetor 4y ago
      Nah, this is full story 99.9% of the time. I worked for government and this happened all the time - nobody ever got fired for choosing 500x more expensive IMB general solution versus something that you customized for the stakeholders and has 0 issues and million users and 0 incidents. I had personaly many such products being on the side of the government once, and on the side of the private vendor after that. One example - I created Help Desk system for the public finances of entire country using Redmine and other FOSS tools. The cost was 0, the time to implement it was single year of not so focused work and it served hundreeds of thousands of people. Then IBM took over with its service desk, implmenting it for years and costing infinity. They could get into tender, I could not since I and my team are small company. The funny thing is that stakeholder subteam abandoned it and returned to my solution (with 0 maintenance since I left the company). This is typical. You need to know how government works to understand it. I understand it, but do not approve it. I am also not frustrated about it, its just how this world works currently, in majority of the countries as far as I know.
eurasiantiger 4y ago
How secure was the app compared to the new one?
- jiggawatts 4y ago
  Fantastically. The new one had "hand rolled cryptography", which should make you twitch uncontrollably if you know anything about security. The new application had, among other failings, hard-coded (unchangeable!) RSA keys used for communication channels. As in, all customers shared the same keys. I can't remember the exact specifics, but I swear at some point there was something like encrypted JSON in XML. Or was it encrypted XML in JSON? Does it matter which? The old app that I wrote would happily take JavaScript or SQL snippets as inputs to any text field and do The Right Thing. You don't want to know what happened to the new app when it was tested with malicious inputs. The testing team were told "not to go too hard on it", because that would "derail the project".
  - omegalulw 4y ago
    > The old app that I wrote would happily take JavaScript or SQL snippets as inputs to any text field and do The Right Thing. Confused here. Where were input validation checks in your implementation? How did you guard against SQL injection, etc?
    - thow232329 4y ago
      Just put the queries in procedures with parameters. Only store the procedure calls in your backend, disable arbitrary queries completely in your database permissions.
    - Winsaucerer 4y ago
      Presumably they just did whatever the standard provided mechanisms for their SQL driver were (such as parameterised queries). User inputs text in a comment box, and you insert it into database using such a mechanism and it's safe. And if you're using, for example, Go's templating library, then it automatically escapes everything in HTML templates unless you explicitly override this default behaviour.
imiric 4y ago
I think you may be biased towards the reasons to replace it, given that it was your creation, and we're not hearing the whole story. It may well be that the system was difficult to maintain _because_ it used a bespoke framework of vanilla JS and handcrafted SQL Server queries. Or that they wanted to improve the workflow of importing CSVs, and build modern features around it, which would be a mountain of work. Or that the company outgrew it and it was difficult to scale. Or, you might be right, and it was politically and financially driven. But then the technology choice wouldn't have mattered, and you could've chosen a more complex stack just as well. I appreciate the sentiment of trying to keep things simple and not jumping on the bandwagon of the latest trends, but sometimes choosing a popular framework is not a bad idea. Particularly in corporate environments where the project is not owned by a single person, churn is high, and new developers are expected to eventually take over maintenance.
- jiggawatts 4y ago
  One day of maintenance per year is not "difficult", that's basically the point! I didn't use or create any JS frameworks, which is a part of why the maintenance was easy! The customer was a government department, and their scale changed only with population. That is: slowly. > sometimes choosing a popular framework is not a bad idea. Ironically, the replacement product used a popular but out-of-date technologies such as Enterprise Java Beans. They overused OO paradigms to a hilarious degree, and needed something like 2000x the server capacity to host their application. Keep in mind that the data, userbase, requiements, etc... are all identical. This is a like-for-like replacement. They needed an entire team of people just to babysit the infrastructure, which now took a decent chunk of a data center. My app could have handled the production workload while running on my laptop.
  - imiric 4y ago
    Alright, it sounds like a typical government gig then. Then a lesson in this case is to use large corporate approved technologies, regardless of how inefficient, costly and complex they might be.
- spaniard89277 4y ago
xcambar 4y ago
It is also possible that this project and you being the only one handling it were analyzed as a risk. It doesn't justify how the handover/rewrite happened, but why it did is somewhat understandable, if not plain reasonable. Typically, employees with walled gardens (willingly or not) are a massive liability to their company.
- j-bos 4y ago
  What details must be missinh for 100 lines of js and a sql backend with or without orm to be a walled garden?
- throw827474737 4y ago
  That's not what he said, sounds more like anybody (reasonable) could have handled it alone, or for redundancy in a small team. Boring standardized battletested tech is the opposite of creating a walled garden, and he states (understandably) that this was the problem, for him and the product itself. Very sad story, but seeing what lasagnacode over-and-underengineered-madnesses-at-the-same-timr are put up today, more than completely believable..
- ratww 4y ago
  Nowhere in the grandparent's post says that it was a "walled garden", or even that it was closed source. The fact that only one person was needed doesn't mean there's only one person available. OP even said he worked for a company in a reply. The rationalisation automatically assumes that the grandparent is either incompetent or lying by omission, which is very uncharitable. Even if all those problems were true, if it was really analysed as risky, the proper thing to do is to bring in one or two more engineers, perform audits, ask for the full source if it's not available. Ask for documentation. Heck, OP said it's not minified: try to reverse engineer it, if need be. Perhaps it's not even necessary! There's absolutely no need to bring a 9-digit-sum team to replace a working system made by one person, even if this is common practice in our industry. Not before all other rational avenues are pursued if there are problems. What also pisses me off is that what happened on the other side might have been caused by companies like the ones I worked for. For a long time I worked for consultancies, and it was routine for sales to "translate" our feature lists into procurement rules (sorry don't know the term in english) and give that to companies and government so we would be the only ones able to answer. And the worst part is that software engineers go on with this tune because they enjoy so much overengineering everything from scratch.
sorokod 4y ago
You did get to keep your integrity as an engineer and have a good story to tell. That is worth a lot.
rTX5CMRXIfFG 4y ago
I honestly think you did the right thing--your conclusion is more a cynical take than a truthful claim. You clearly had organizational problems, and that's beyond the scope of your code.
Isamu 4y ago
I have personally seen this within organizations. Certain leaders can appreciate the simple approach, but not others, because it doesn’t increase headcount. For other leadership types it is drama and increasing headcount that drive their careers. Also you the developer will get scant recognition for finding the simplest solution. That’s the kind of thing that doesn’t get appreciation up the management chain (usually.) It should but it doesn’t. The perception is not: wow this will save us millions over time by allowing us to do more with less. The perception is: so this guy did this project that turned out to be simple.
- WJW 4y ago
  It's the same problem as preventing versus curing: the latter is much more expensive but much more flashy. In most companies the owners are the only ones who would care about doing things as efficiently as possible, but those are often also the most removed the line work. They get all their information filtered through middle managers who are competing with each other for their next promotion, so unspectacular news often doesn't make the cut for being passed upwards. The main exception I've found so far is in making tools/systems for myself, since then it is easy to convince the owner about the benefits of simplicity and easy maintenance :)
patrickthebold 4y ago
It sounds like your project ended up working out well (ignoring the replacement). But one thing that would be hard for me when starting a project like this: How do you know that over time it won't grow into something terribly unmaintainable? You don't have an ORM, but then perhaps over time you re-implement most of the functionality of an ORM, and now new people need to learn that. Of course, you can start with out one and bring one in when it is needed. But in my experience that's hard to actually do because feature N + 1 needs to be implemented now and there's no time to migrate everything over to (ORM that would have been nice to have to make feature N + 1 easy to implement.) I'm just using ORM as an example, or course. Anyhoo, I think there's probably some other dimensions than "boring". Seems like you used "less" tech, but I'd say in the java world Spring and Hibernate are boring, or at least "popular", in the sense you can hire devs anywhere with some experience.
- mike_hock 4y ago
  By devoting time to code maintenance and refactoring in between features N and N+1 (or at least N+M). The code doesn't just magically go from 5-10 SQL queries to being completely unmaintainable without an ORM overnight. When and if it grows into that, you'll see it coming. That doesn't work, of course, if you're not considered to be "working" unless you're hacking on a new feature right now that'll be deployable by the end of the week, but it seems like OP was allowed to develop in a sane way.
  - throw827474737 4y ago
    > The code doesn't just magically go from 5-10 SQL queries to being completely unmaintainable without an ORM overnight. I'm not sure in whih world an ORM helps with maintainability...
- ratww 4y ago
  You either start with or without an ORM, depending on your assessment of whether the project is gonna need one. If you start without one, you still have to partition your code well enough so that retrofitting one doesn't cause a huge mess. Basically keep your "raw SQL queries" in a centralised place (file or folder), rather than strewn together in controllers/views/services. And you should do exactly the same if you use an ORM. Isolate the fuck out of it and make it "easily removable" or "easily replaceable". Also keep the "effects" of your ORM or your non-ORM away from those other parts too: your controllers, views and services should be totally agnostic to whatever you're using in the data layer. When you add subtle coupling you lose the possibility of changing it, but it also makes your project less maintainable. This is easier said than done: in dynamic languages or with structural typing like Typescript it's very easy: it's all objects, anyway, so ORM or no ORM it's the same. In stricter languages like Java it might lead to lots of intermediate data structures which are verbose and causes problems in itself. Or the middle ground: use primitives (lists and maps) rather than classes and objects, although ORMs like Hibernate will make things difficult for you, since they're not too flexible about how they're used and their types tend to "creep" all over your project. - Most unmaintainable projects don't become unmaintainable because people "forgot to prepare". They become unmaintainable because people assumed everything is permanent, so there's no penalty to using everything-everywhere. So there are "traces" of the ORM in the controllers and views, the serialisation library and serialisation code is called in models in services as a "quick hack", the authorisation library is called from everywhere because why not. You quickly lose the ability to easily reason about code. The same applies other areas. I could make a treatise of how game developers love sticking keyboard-reading code absolutely everywhere in the codebase.
geokon 4y ago
Isn't the moral of the story that you didn't charge for it correctly? If you charged a fixed annual maintenance fee then you would have felt very clever having made a ton of money not having to do anything Administrators would have also felt good that they had you as insurance of sorts bc the way things stand you make no money and can disappear at any moment
- lr4444lr 4y ago
  I think they would have still been uneasy that he was a SPOF, even if he had charged a lot more. And there's also a cognitive disconnect if a one-man deliverable costs above a certain threshold.
  - steveBK123 4y ago
    The SPOF fallacy is always managements favorite way to waste money. My current shop we had a system maintained by 1 guy, as like 10-20% of his job/time. I can guarantee you he is not paid even $500K. This of course was deemed bad & risky, so we must engage a set of vendors to deliver a replacement. 2 years, a dozen subcontractors, and 7-figure annual bill later.. and they still haven't replaced 10-20% of this SPOF. Literally spent $5M so far against max $100k of this guys salary. There are still no signs that SPOF can give up the responsibility even next year. No one has been fired over this. In fact the decider has been promoted.
  - samatman 4y ago
    Both of these issues are often solved by incorporation. A one-vendor deliverable can cost whatever, and corporations can certainly agree to keep supporting something for a certain time. Doesn't actually matter if the corporation has one member, and no coherent plan for what happens to the contracts if that person steps in front of the wrong bus. It makes the relationship legible in the way that the contracting party is comfortable with.
    - pojzon 4y ago
      Its called business continuity plan and tbh most companies drop the ball in that regard.
calvinmorrison 4y ago
we don't have simplicity because people are inept. I would trust you to write a small CRUD php database without any security issues, however, the next goon that comes along and jams in a bunch of $id = $GET[id]; insert into where $is; and you have a major security issue Frameworks exist so you don't roll your own stupidity into a bigger problem.
kyleyeats 4y ago
This comment was better than the post!
simonebrunozzi 4y ago
> This is why we can't have simplicity: because it doesn't scale. No, it's because it doesn't sell.
- RockingGoodNite 4y ago
  Agreed. Half of people want simple, maintainable solutions. The other half want complex solutions that people can glam onto, like how politics works.
osigurdson 4y ago
>> deployed to over 2000 locations Were there 2000 independent systems / SQL server instances running or just one? 2K separate deployments to manage (with 1K users each), does sound a little scary. Of course, perhaps that is not what is going on at all.
- skeeter2020 4y ago
  Which is actually kinda funny, because some of the "complex" technology the OP is railing against allows us today to manage thousands of databases both easily and efficiently... IF the systems are built with a more current approach. This is why I try to understand ALL of the motivations for disruptive change and not immediately assume incompetence and self-interest bordering on criminal.
  - osigurdson 4y ago
    Sometimes complexity is conflated with lack of familiarity. Instead of using the term "complexity", we should state what the actual problem is.
  - hedora 4y ago
    Option one: Write a cloud formation / terraform template that involves O(1) machines and deploy 2000 identical copies. Option two: Write a template that deploys O(N = 2000) interdependent services across roughly 3-10x as many machines, and deploy one copy. From what I can tell, you are arguing for option 2. It is strictly worse than option one. In addition to being more complex, it has a few nines less reliability, and costs 3-10x more for the hardware. The dev and CI hardware budgets is going to be 10x more because you can't test it on one machine, and it has bugs that only manifest at scale. Source: I do this for a living, and have been on both sides of this fence. Option 1 typically has 5-6 nines (measured in chance a given customer sees a 10 second outage), option 2 never gets past 3-4 nines (measured in at least N% of customers are not seeing an outage). The modern vs old technology debate has nothing to do with this tradeoff. If you want, you can build option 2 with EJB + CORBA on an IBM mainframe, and option 1 with rust and json on an exokernel FAAS.
skeeter2020 4y ago
This definitely one narrative, but there are lots of reasons that could also contribute to a change. A few: 1. Your system was likely more complex than you describe here; What's generating HTML with dynamic data between the database and the client? Did you "100 lines of JS" have any dependencies? 2. Maybe your company wasn't charging for this simplicity and peace of mind correctly. Companies would pay for for SaaS-style products that looked a lot like subscriptions even back in 2007, we just didn't call them that. 3. It sounds like this was run on-prem. That's expensive (and scary) for a lot of companies if supporting software is not part of their core skill set. 4. We're not solving the same problems today as 2007; much of that low-hanging fruit has been picked. I'm guessing your original system was internal facing at your clients; everyone wants to integrate into much broader client-facing workflows now. 5. If you were only doing annual updates not much was changing. That's awesome but implies a pretty static problem domain. There are countless more motivations, and the baseline has shifted dramatically. I'm not saying the reason you present is wrong or not the primary one, but it's dangerous to attribute malicious intent when there are lots of "simpler" reasons as well. >> This is why we can't have simplicity: because it doesn't scale. I'm not sure your example leads to this conclusion. Simplicity is a set of abstractions. When we expand the domain broadly enough they start to leak. This is related to, but not the same thing as scaling.
- dc-programmer 4y ago
  I do feel like the necessary complexity of SQL maintenance and dependency patching was thrown under the rug here. But then again maybe the client completely firewalled development and operations
norwalkbear 4y ago
Yea government work is full of that. Kickbacks galore, my brother works in ed-tech. He said one state rewrites their public school report card system every two years like clockwork because of that.
- steveBK123 4y ago
  You know the crazy thing is that moderately sized private companies I've worked (500-5000 people) have plenty of that kind of BS. I have seen a piece of mediocre software or service vendors sold as a panacea to my last 2 shops, skip POC phase and just get purchased off the back of someone (not in IT) very very senior being buddies with the founders. No users or technical people were asking for it.. just gets rammed down from place up high until a reasonable enough niche is found to put it into PROD. It ends up being a solution in search of a problem, with a couple years finding where to use it, a couple years finding it inadequate, and then a couple years removing it again. Great money if you can get it.
jameshart 4y ago
100 lines of JavaScript and a backend consisting of stored procedures that output HTML does not sound like a ‘boring’ technology choice. It sounds very exciting - anyone approaching this codebase to do things to it will likely have lots of very interesting questions about how this system handles lots of things. The solution might have good answers to all of those questions! It is perfectly possible to build a well-engineered system using those technologies! But on almost every level the answers to those questions are going to be surprising. Whereas the same thing built using webpack and a Ruby on Rails to Postgres backend will be much more legible. ‘Boring’ technology does not mean no technology.
dub 4y ago
Figuring out how to reward simplicity, reliability, and maintainability feels like one of the most important unsolved social/human/economic issues in the software industry. Seems there's only incentive to simplify at small companies where the employees feel they can save their own time or increase the value of their equity by delivering value to customers more efficiently. At large companies employees work 40-hour weeks regardless of their output and they're trying to impress a performance review committee, not customers.
- danieljacksonno 4y ago
  Stop billing hourly. Instead bill for the value you provide. How you provide it should be immaterial, and so you can do it as cheaply and efficiently as possible while still reaping heaps of money as long as it creates value. That's how all other markets work. Billing hourly is the death of progress.
hedora 4y ago
I've seen this at startups. Management starts off informal, and promotes people that have good visibility. In the early stage, these are people that quickly churn through feature lists and deliver solid code. Later, drunk on its own success, management continues to promote based on visibility. In a mid sized company, the most visible engineers are the ones that manage to ship broken code, then respond to nail-biter business continuity ending pages at 2AM. At this point, the engineers that built the product and keep the lights on vest out, are passed over for promotion, and then leave. In the next phase, the company's product stops delivering on its core competency, but hopefully it has monopoly leverage, so whatever. Finally, the big company has a come to Jesus moment, and tries to course correct. This step is fraught with peril. It rarely works, and instead usually leads to a revolving door of process-fiddling / agile-promoting execs. This happens because execs that could solve the problem necessarily realize the root issue is middle and upper management; organizational antibodies pick up on this and isolate such threats to the staus quo. The only execs that succeed at this point are the ones that somehow delegate 100% to low level managers and ICs while giving their peers the impression they are micromanaging and making massive organizational realignments or something.
richardhod 4y ago
I think you mean: we can't have simplicity, because it doesn't sell!
x86x87 4y ago
I call this the tragedy of software development. Well written, minimal, roubust systems are not appreciated or welcomed. Heck, most of the times people don't know they exist. In our industry the heroes are the firefighters and the firefighters are usually the pyromaniacs.

agumonkey 4y ago

> You see, all of the above was a problem, because it didn't keep me employed. I was not the "hero" for saving the day. Entire teams of people weren't involved. There was no visibility at the senior management level. Nobody got scared, or had to throw money at it, or hire consultants to review it. there are a few other instances of that: - an old article about Michelin (french tire manufacturer) quoting some scientist of theirs "We can make a million hour tire.. but what would we sell" - recently people said their rust code cause too much downtime for coders because it was too stable too early flip side of the same issue: - very often people game their work to ensure benefits: stash duties for later so you can appear busy, or overwhelmed (and claim promotion because you have so much to do) The global system doesn't reward to true optimization, it allocates people on useless tasks, at best for lower risk, but smart people doing things solid and fast could be using their talents on other problems.

bdg 4y ago

I understand the annoyance of it being replaced for a more complex more expensive system, but I would also like to know: What was the reasoning provided and what did decision makers truly believe about the whole thing?

jiggawatts 4y ago
> What was the reasoning provided Reasoning? Hah-ha. Haaaaa... It was an open government tender process, for which I, or the company I worked for was not eligible, despite the tender being "open". You see, a decade-long pedigree of actually having implemented the software used for this purpose did not qualify us for replacing it with a v2.0. There are rules, you see? They have to be followed! Or else. Or else bad things might happen, like money being wasted. The fact that the end-result of this process was that a 9-digit sum was spent on something I spat out in my spare time in under a year -- and was used for a decade -- was of no relevance. > what did decision makers truly believe about the whole thing? Their concerns started only when the whole thing blew up and started making headlines. Then nothing happened to them personally, so their concerns evaporated along with the taxpayer funds they had wasted.
- listenallyall 4y ago
  I think most readers, including myself, empathize with you and understand the frustration and absurdity. But you are also telling just one side of the story (yours) and I imagine that v.2.0 specs had certain requirements and features, possibly required by legislation, that needed to be followed and implemented. When you say, dismissively, There are rules, you see? They have to be followed! that's when I, and likely others, start to wonder if you are really providing the full story, or if you actually even understand the differences between your simple app and the updated version.
  - majkinetor 4y ago
    Nah, this is full story 99.9% of the time. I worked for government and this happened all the time - nobody ever got fired for choosing 500x more expensive IMB general solution versus something that you customized for the stakeholders and has 0 issues and million users and 0 incidents. I had personaly many such products being on the side of the government once, and on the side of the private vendor after that. One example - I created Help Desk system for the public finances of entire country using Redmine and other FOSS tools. The cost was 0, the time to implement it was single year of not so focused work and it served hundreeds of thousands of people. Then IBM took over with its service desk, implmenting it for years and costing infinity. They could get into tender, I could not since I and my team are small company. The funny thing is that stakeholder subteam abandoned it and returned to my solution (with 0 maintenance since I left the company). This is typical. You need to know how government works to understand it. I understand it, but do not approve it. I am also not frustrated about it, its just how this world works currently, in majority of the countries as far as I know.

eurasiantiger 4y ago

How secure was the app compared to the new one?

jiggawatts 4y ago
Fantastically. The new one had "hand rolled cryptography", which should make you twitch uncontrollably if you know anything about security. The new application had, among other failings, hard-coded (unchangeable!) RSA keys used for communication channels. As in, all customers shared the same keys. I can't remember the exact specifics, but I swear at some point there was something like encrypted JSON in XML. Or was it encrypted XML in JSON? Does it matter which? The old app that I wrote would happily take JavaScript or SQL snippets as inputs to any text field and do The Right Thing. You don't want to know what happened to the new app when it was tested with malicious inputs. The testing team were told "not to go too hard on it", because that would "derail the project".
- omegalulw 4y ago
  > The old app that I wrote would happily take JavaScript or SQL snippets as inputs to any text field and do The Right Thing. Confused here. Where were input validation checks in your implementation? How did you guard against SQL injection, etc?
  - thow232329 4y ago
    Just put the queries in procedures with parameters. Only store the procedure calls in your backend, disable arbitrary queries completely in your database permissions.
  - Winsaucerer 4y ago
    Presumably they just did whatever the standard provided mechanisms for their SQL driver were (such as parameterised queries). User inputs text in a comment box, and you insert it into database using such a mechanism and it's safe. And if you're using, for example, Go's templating library, then it automatically escapes everything in HTML templates unless you explicitly override this default behaviour.

imiric 4y ago

I think you may be biased towards the reasons to replace it, given that it was your creation, and we're not hearing the whole story. It may well be that the system was difficult to maintain _because_ it used a bespoke framework of vanilla JS and handcrafted SQL Server queries. Or that they wanted to improve the workflow of importing CSVs, and build modern features around it, which would be a mountain of work. Or that the company outgrew it and it was difficult to scale. Or, you might be right, and it was politically and financially driven. But then the technology choice wouldn't have mattered, and you could've chosen a more complex stack just as well. I appreciate the sentiment of trying to keep things simple and not jumping on the bandwagon of the latest trends, but sometimes choosing a popular framework is not a bad idea. Particularly in corporate environments where the project is not owned by a single person, churn is high, and new developers are expected to eventually take over maintenance.

jiggawatts 4y ago
One day of maintenance per year is not "difficult", that's basically the point! I didn't use or create any JS frameworks, which is a part of why the maintenance was easy! The customer was a government department, and their scale changed only with population. That is: slowly. > sometimes choosing a popular framework is not a bad idea. Ironically, the replacement product used a popular but out-of-date technologies such as Enterprise Java Beans. They overused OO paradigms to a hilarious degree, and needed something like 2000x the server capacity to host their application. Keep in mind that the data, userbase, requiements, etc... are all identical. This is a like-for-like replacement. They needed an entire team of people just to babysit the infrastructure, which now took a decent chunk of a data center. My app could have handled the production workload while running on my laptop.
- imiric 4y ago
  Alright, it sounds like a typical government gig then. Then a lesson in this case is to use large corporate approved technologies, regardless of how inefficient, costly and complex they might be.
spaniard89277 4y ago

xcambar 4y ago

It is also possible that this project and you being the only one handling it were analyzed as a risk. It doesn't justify how the handover/rewrite happened, but why it did is somewhat understandable, if not plain reasonable. Typically, employees with walled gardens (willingly or not) are a massive liability to their company.

j-bos 4y ago
What details must be missinh for 100 lines of js and a sql backend with or without orm to be a walled garden?
throw827474737 4y ago
That's not what he said, sounds more like anybody (reasonable) could have handled it alone, or for redundancy in a small team. Boring standardized battletested tech is the opposite of creating a walled garden, and he states (understandably) that this was the problem, for him and the product itself. Very sad story, but seeing what lasagnacode over-and-underengineered-madnesses-at-the-same-timr are put up today, more than completely believable..
ratww 4y ago
Nowhere in the grandparent's post says that it was a "walled garden", or even that it was closed source. The fact that only one person was needed doesn't mean there's only one person available. OP even said he worked for a company in a reply. The rationalisation automatically assumes that the grandparent is either incompetent or lying by omission, which is very uncharitable. Even if all those problems were true, if it was really analysed as risky, the proper thing to do is to bring in one or two more engineers, perform audits, ask for the full source if it's not available. Ask for documentation. Heck, OP said it's not minified: try to reverse engineer it, if need be. Perhaps it's not even necessary! There's absolutely no need to bring a 9-digit-sum team to replace a working system made by one person, even if this is common practice in our industry. Not before all other rational avenues are pursued if there are problems. What also pisses me off is that what happened on the other side might have been caused by companies like the ones I worked for. For a long time I worked for consultancies, and it was routine for sales to "translate" our feature lists into procurement rules (sorry don't know the term in english) and give that to companies and government so we would be the only ones able to answer. And the worst part is that software engineers go on with this tune because they enjoy so much overengineering everything from scratch.

sorokod 4y ago

You did get to keep your integrity as an engineer and have a good story to tell. That is worth a lot.

rTX5CMRXIfFG 4y ago

I honestly think you did the right thing--your conclusion is more a cynical take than a truthful claim. You clearly had organizational problems, and that's beyond the scope of your code.

Isamu 4y ago

I have personally seen this within organizations. Certain leaders can appreciate the simple approach, but not others, because it doesn’t increase headcount. For other leadership types it is drama and increasing headcount that drive their careers. Also you the developer will get scant recognition for finding the simplest solution. That’s the kind of thing that doesn’t get appreciation up the management chain (usually.) It should but it doesn’t. The perception is not: wow this will save us millions over time by allowing us to do more with less. The perception is: so this guy did this project that turned out to be simple.

WJW 4y ago
It's the same problem as preventing versus curing: the latter is much more expensive but much more flashy. In most companies the owners are the only ones who would care about doing things as efficiently as possible, but those are often also the most removed the line work. They get all their information filtered through middle managers who are competing with each other for their next promotion, so unspectacular news often doesn't make the cut for being passed upwards. The main exception I've found so far is in making tools/systems for myself, since then it is easy to convince the owner about the benefits of simplicity and easy maintenance :)

patrickthebold 4y ago

It sounds like your project ended up working out well (ignoring the replacement). But one thing that would be hard for me when starting a project like this: How do you know that over time it won't grow into something terribly unmaintainable? You don't have an ORM, but then perhaps over time you re-implement most of the functionality of an ORM, and now new people need to learn that. Of course, you can start with out one and bring one in when it is needed. But in my experience that's hard to actually do because feature N + 1 needs to be implemented now and there's no time to migrate everything over to (ORM that would have been nice to have to make feature N + 1 easy to implement.) I'm just using ORM as an example, or course. Anyhoo, I think there's probably some other dimensions than "boring". Seems like you used "less" tech, but I'd say in the java world Spring and Hibernate are boring, or at least "popular", in the sense you can hire devs anywhere with some experience.

mike_hock 4y ago
By devoting time to code maintenance and refactoring in between features N and N+1 (or at least N+M). The code doesn't just magically go from 5-10 SQL queries to being completely unmaintainable without an ORM overnight. When and if it grows into that, you'll see it coming. That doesn't work, of course, if you're not considered to be "working" unless you're hacking on a new feature right now that'll be deployable by the end of the week, but it seems like OP was allowed to develop in a sane way.
- throw827474737 4y ago
  > The code doesn't just magically go from 5-10 SQL queries to being completely unmaintainable without an ORM overnight. I'm not sure in whih world an ORM helps with maintainability...
ratww 4y ago
You either start with or without an ORM, depending on your assessment of whether the project is gonna need one. If you start without one, you still have to partition your code well enough so that retrofitting one doesn't cause a huge mess. Basically keep your "raw SQL queries" in a centralised place (file or folder), rather than strewn together in controllers/views/services. And you should do exactly the same if you use an ORM. Isolate the fuck out of it and make it "easily removable" or "easily replaceable". Also keep the "effects" of your ORM or your non-ORM away from those other parts too: your controllers, views and services should be totally agnostic to whatever you're using in the data layer. When you add subtle coupling you lose the possibility of changing it, but it also makes your project less maintainable. This is easier said than done: in dynamic languages or with structural typing like Typescript it's very easy: it's all objects, anyway, so ORM or no ORM it's the same. In stricter languages like Java it might lead to lots of intermediate data structures which are verbose and causes problems in itself. Or the middle ground: use primitives (lists and maps) rather than classes and objects, although ORMs like Hibernate will make things difficult for you, since they're not too flexible about how they're used and their types tend to "creep" all over your project. - Most unmaintainable projects don't become unmaintainable because people "forgot to prepare". They become unmaintainable because people assumed everything is permanent, so there's no penalty to using everything-everywhere. So there are "traces" of the ORM in the controllers and views, the serialisation library and serialisation code is called in models in services as a "quick hack", the authorisation library is called from everywhere because why not. You quickly lose the ability to easily reason about code. The same applies other areas. I could make a treatise of how game developers love sticking keyboard-reading code absolutely everywhere in the codebase.

geokon 4y ago

Isn't the moral of the story that you didn't charge for it correctly? If you charged a fixed annual maintenance fee then you would have felt very clever having made a ton of money not having to do anything Administrators would have also felt good that they had you as insurance of sorts bc the way things stand you make no money and can disappear at any moment

lr4444lr 4y ago
I think they would have still been uneasy that he was a SPOF, even if he had charged a lot more. And there's also a cognitive disconnect if a one-man deliverable costs above a certain threshold.
- steveBK123 4y ago
  The SPOF fallacy is always managements favorite way to waste money. My current shop we had a system maintained by 1 guy, as like 10-20% of his job/time. I can guarantee you he is not paid even $500K. This of course was deemed bad & risky, so we must engage a set of vendors to deliver a replacement. 2 years, a dozen subcontractors, and 7-figure annual bill later.. and they still haven't replaced 10-20% of this SPOF. Literally spent $5M so far against max $100k of this guys salary. There are still no signs that SPOF can give up the responsibility even next year. No one has been fired over this. In fact the decider has been promoted.
- samatman 4y ago
  Both of these issues are often solved by incorporation. A one-vendor deliverable can cost whatever, and corporations can certainly agree to keep supporting something for a certain time. Doesn't actually matter if the corporation has one member, and no coherent plan for what happens to the contracts if that person steps in front of the wrong bus. It makes the relationship legible in the way that the contracting party is comfortable with.
  - pojzon 4y ago
    Its called business continuity plan and tbh most companies drop the ball in that regard.

calvinmorrison 4y ago

we don't have simplicity because people are inept. I would trust you to write a small CRUD php database without any security issues, however, the next goon that comes along and jams in a bunch of $id = $GET[id]; insert into where $is; and you have a major security issue Frameworks exist so you don't roll your own stupidity into a bigger problem.

kyleyeats 4y ago

This comment was better than the post!

simonebrunozzi 4y ago

> This is why we can't have simplicity: because it doesn't scale. No, it's because it doesn't sell.

RockingGoodNite 4y ago
Agreed. Half of people want simple, maintainable solutions. The other half want complex solutions that people can glam onto, like how politics works.

osigurdson 4y ago

>> deployed to over 2000 locations Were there 2000 independent systems / SQL server instances running or just one? 2K separate deployments to manage (with 1K users each), does sound a little scary. Of course, perhaps that is not what is going on at all.

skeeter2020 4y ago
Which is actually kinda funny, because some of the "complex" technology the OP is railing against allows us today to manage thousands of databases both easily and efficiently... IF the systems are built with a more current approach. This is why I try to understand ALL of the motivations for disruptive change and not immediately assume incompetence and self-interest bordering on criminal.
- osigurdson 4y ago
  Sometimes complexity is conflated with lack of familiarity. Instead of using the term "complexity", we should state what the actual problem is.
- hedora 4y ago
  Option one: Write a cloud formation / terraform template that involves O(1) machines and deploy 2000 identical copies. Option two: Write a template that deploys O(N = 2000) interdependent services across roughly 3-10x as many machines, and deploy one copy. From what I can tell, you are arguing for option 2. It is strictly worse than option one. In addition to being more complex, it has a few nines less reliability, and costs 3-10x more for the hardware. The dev and CI hardware budgets is going to be 10x more because you can't test it on one machine, and it has bugs that only manifest at scale. Source: I do this for a living, and have been on both sides of this fence. Option 1 typically has 5-6 nines (measured in chance a given customer sees a 10 second outage), option 2 never gets past 3-4 nines (measured in at least N% of customers are not seeing an outage). The modern vs old technology debate has nothing to do with this tradeoff. If you want, you can build option 2 with EJB + CORBA on an IBM mainframe, and option 1 with rust and json on an exokernel FAAS.

skeeter2020 4y ago

This definitely one narrative, but there are lots of reasons that could also contribute to a change. A few: 1. Your system was likely more complex than you describe here; What's generating HTML with dynamic data between the database and the client? Did you "100 lines of JS" have any dependencies? 2. Maybe your company wasn't charging for this simplicity and peace of mind correctly. Companies would pay for for SaaS-style products that looked a lot like subscriptions even back in 2007, we just didn't call them that. 3. It sounds like this was run on-prem. That's expensive (and scary) for a lot of companies if supporting software is not part of their core skill set. 4. We're not solving the same problems today as 2007; much of that low-hanging fruit has been picked. I'm guessing your original system was internal facing at your clients; everyone wants to integrate into much broader client-facing workflows now. 5. If you were only doing annual updates not much was changing. That's awesome but implies a pretty static problem domain. There are countless more motivations, and the baseline has shifted dramatically. I'm not saying the reason you present is wrong or not the primary one, but it's dangerous to attribute malicious intent when there are lots of "simpler" reasons as well. >> This is why we can't have simplicity: because it doesn't scale. I'm not sure your example leads to this conclusion. Simplicity is a set of abstractions. When we expand the domain broadly enough they start to leak. This is related to, but not the same thing as scaling.

dc-programmer 4y ago
I do feel like the necessary complexity of SQL maintenance and dependency patching was thrown under the rug here. But then again maybe the client completely firewalled development and operations

norwalkbear 4y ago

Yea government work is full of that. Kickbacks galore, my brother works in ed-tech. He said one state rewrites their public school report card system every two years like clockwork because of that.

steveBK123 4y ago
You know the crazy thing is that moderately sized private companies I've worked (500-5000 people) have plenty of that kind of BS. I have seen a piece of mediocre software or service vendors sold as a panacea to my last 2 shops, skip POC phase and just get purchased off the back of someone (not in IT) very very senior being buddies with the founders. No users or technical people were asking for it.. just gets rammed down from place up high until a reasonable enough niche is found to put it into PROD. It ends up being a solution in search of a problem, with a couple years finding where to use it, a couple years finding it inadequate, and then a couple years removing it again. Great money if you can get it.

jameshart 4y ago

100 lines of JavaScript and a backend consisting of stored procedures that output HTML does not sound like a ‘boring’ technology choice. It sounds very exciting - anyone approaching this codebase to do things to it will likely have lots of very interesting questions about how this system handles lots of things. The solution might have good answers to all of those questions! It is perfectly possible to build a well-engineered system using those technologies! But on almost every level the answers to those questions are going to be surprising. Whereas the same thing built using webpack and a Ruby on Rails to Postgres backend will be much more legible. ‘Boring’ technology does not mean no technology.

dub 4y ago

Figuring out how to reward simplicity, reliability, and maintainability feels like one of the most important unsolved social/human/economic issues in the software industry. Seems there's only incentive to simplify at small companies where the employees feel they can save their own time or increase the value of their equity by delivering value to customers more efficiently. At large companies employees work 40-hour weeks regardless of their output and they're trying to impress a performance review committee, not customers.

danieljacksonno 4y ago
Stop billing hourly. Instead bill for the value you provide. How you provide it should be immaterial, and so you can do it as cheaply and efficiently as possible while still reaping heaps of money as long as it creates value. That's how all other markets work. Billing hourly is the death of progress.

hedora 4y ago

I've seen this at startups. Management starts off informal, and promotes people that have good visibility. In the early stage, these are people that quickly churn through feature lists and deliver solid code. Later, drunk on its own success, management continues to promote based on visibility. In a mid sized company, the most visible engineers are the ones that manage to ship broken code, then respond to nail-biter business continuity ending pages at 2AM. At this point, the engineers that built the product and keep the lights on vest out, are passed over for promotion, and then leave. In the next phase, the company's product stops delivering on its core competency, but hopefully it has monopoly leverage, so whatever. Finally, the big company has a come to Jesus moment, and tries to course correct. This step is fraught with peril. It rarely works, and instead usually leads to a revolving door of process-fiddling / agile-promoting execs. This happens because execs that could solve the problem necessarily realize the root issue is middle and upper management; organizational antibodies pick up on this and isolate such threats to the staus quo. The only execs that succeed at this point are the ones that somehow delegate 100% to low level managers and ICs while giving their peers the impression they are micromanaging and making massive organizational realignments or something.

richardhod 4y ago

I think you mean: we can't have simplicity, because it doesn't sell!

x86x87 4y ago

I call this the tragedy of software development. Well written, minimal, roubust systems are not appreciated or welcomed. Heck, most of the times people don't know they exist. In our industry the heroes are the firefighters and the firefighters are usually the pyromaniacs.