24 ms·

It happened to me twice. The first time was in a start-up at the beginning of the century, we were developing an electronic health record and we had outsourced

It happened to me twice. The first time was in a start-up at the beginning of the century, we were developing an electronic health record and we had outsourced the database abstraction layer to a company in Greece. In the beginning things went fine but after a while the development of the DAL went slower and slower and it became unstable as well. Eventually the word came out: the main developer of the DAL framework had left the company and, according to the Greek CEO, she had been 'too smart' which meant that nobody understood her code. They had tried adding features but that had made things only worse and the DAL had started to crash randomly. We tried to take over the framework by ourselves but it was written in Eiffel and the code was a horrible entangled mess. Eventually we rewrote it in Java but, being a start-up, we lost too much precious time already and eventually went almost bankrupt and were bought up by a competitor. The second time was in a small company whose product was a search engine for consumers. The web layer was written in a mixture of JSF, JQuery and Ajax. While that combination already slowed down development on the front end, the main problem was the performance of JSF on the server. Because JSF is rendered on the backend, it placed a massive load on our server for certain heavily used pages and we just couldn't scale any further. Skipping JSF for a framework that was rendered on the front-end would be the solution but that was a massive refactor for which the company just didn't have enough resources. Eventually the company had to skip their search product and change their business model to a more community based website.

tboyd47 10y ago
For the second one, you must have been receiving a lot of traffic for template rendering to be such a bottleneck. Why not upgrade the server?
- misja111 10y ago
  They upgraded the server of course, to as much as they could afford. But it wasn't enough, the rendering load soon caught up. First of all because their number of visitors grew, but also because they wanted to add new features to their JSF pages and every new feature required extra rendering power as well.
  - olavgg 10y ago
    Could you not scale this horizontally? We do all template rendering server side, though it is JSP and not JSF.
    - misja111 10y ago
      That was considered but it would of course take some refactoring on the back end, and it would still cost quite much in hardware. The thing with JSP and JSF is, they do ok as long as your content is relatively static, because then rendered content can be cached. In case of this company, their most visited page was the list with search results which by its very nature was not very static at all.
      - sbov 10y ago
        Every problem is different, so I hate to judge, but what you're saying doesn't add up to any experience I've had. It sounds like your company seriously screwed up the design if you can't scale your web tier code horizontally. I've also never had a view technology take up a significant chunk of cpu resources - it's always the Java code carrying out the functionality. E.g. I would expect the largest factor in CPU usage in the list of search results to be... generating the data for the search result. If the largest factor was rendering the result, then something was probably seriously wrong.
taeric 10y ago
This sounds less like technical debt, and more like liabilities of over engineering. Possibly feature creep. That is, technical debt is not necessarily tangled over-engineered code. It is more compromises that were made to actually ship and operate in the world. You can see this in the world with devices. Consider, technical debt is the reason you have AC delivered to your house going through as many converters as you do devices. Often to the same target power characteristics for those devices. It is not the reason that your coffee machine that also grinds and whatever, is likely to fail within the year. Another example; Technical debt is the reason we are still predominantly using petrol for automobiles. It is not the reason the dashboards are horribly non-responsive on modern cars.
- CydeWeys 10y ago
  > Consider, technical debt is the reason you have AC delivered to your house going through as many converters as you do devices. Often to the same target power characteristics for those devices. Bad example. AC power has many desirable characteristics for the local transmission grid. If you were to do the grid over from scratch you'd still use AC. You're also too focused on household electronic usage, which is a very tiny percentage of the overall electricity used.
  - taeric 10y ago
    My understanding is that HVDC had advantages. That said, I was also intending that to include the distribution in your house.
    - napoleond 10y ago
      HVDC does have advantages in certain scenarios (very long transmission lines, for example) but parent is still correct--the majority of the grid makes way more sense with AC.
      - taeric 10y ago
        I meant my follow-on to be a concession, but worded it poorly. I thought it had advantages, but yes, I was thinking small appliance mainly. In particular, in home. And not just computers, but lights and control panels. Seems many things all use the same power characteristics and are now becoming complicated by dealing with AC. Which, amusingly, is fitting for the tech debt debate. Eradicating some choices from the project is likely to be missing the point. Just as eradicating AC from all power would be short sighted/wrong.
kisstheblade 10y ago
You had a search engine and rendering the search results was the bottleneck? That's really weird. Don't know a lot about JSF but other templating languages are usually really not ever the bottleneck. Maybe if you have some giant table with thousands of cells each with its own complicated template directive (for loop with conditionals etc).
agrafix 10y ago
What was/is the product/company called if i may ask?
klibertp 10y ago
> We tried to take over the framework by ourselves but it was written in Eiffel and the code was a horrible entangled mess. Eventually we rewrote it in Java but, being a start-up, we lost too much precious time already I wonder, would the result be different if you had access to competent Eiffel developers? How large was the Eiffel codebase? Eiffel is an interesting language, with a somewhat unique feature-set (I think only Ada is coming close). Design by contract and static typing as core language features - if used right - should greatly help with both stability and ease of refactoring. How large the codebase was is an important question, also how bad it really was. I saw a similar story - external codebase getting worse and worse from some point on - with Clojure at the center. The code quality was quite ok for a couple of months, then it worsened. At that point and for a couple of following months the codebase was possible to save - a single competent Clojure programmer would make a difference, I think. The project was less than 10k LOC then. However, more than 1.5 years and 60k LOC later, doing anything became nearly impossible for anyone, including original authors.
edblarney 10y ago
"Eventually the word came out: the main developer of the DAL framework had left the company and, according to the Greek CEO, she had been 'too smart' which meant that nobody understood her code." OMG no - run for the hills. 95% of software systems are not inherently sophisticated - they are 'complex' - yes - maybe there are many features, and moving parts - but there are no pieces of the system that should be hard to understand by anyone. Decent architecture + decent design and coding and an entire banks system should read like a long, but well articulated user manual. Unless you're doing super low-level stuff, complex algorithms, heavy math stuff, or issues with massive scale or performance etc. ... the end result should almost be mundane in most cases.

tboyd47 10y ago

For the second one, you must have been receiving a lot of traffic for template rendering to be such a bottleneck. Why not upgrade the server?

misja111 10y ago
They upgraded the server of course, to as much as they could afford. But it wasn't enough, the rendering load soon caught up. First of all because their number of visitors grew, but also because they wanted to add new features to their JSF pages and every new feature required extra rendering power as well.
- olavgg 10y ago
  Could you not scale this horizontally? We do all template rendering server side, though it is JSP and not JSF.
  - misja111 10y ago
    That was considered but it would of course take some refactoring on the back end, and it would still cost quite much in hardware. The thing with JSP and JSF is, they do ok as long as your content is relatively static, because then rendered content can be cached. In case of this company, their most visited page was the list with search results which by its very nature was not very static at all.
    - sbov 10y ago
      Every problem is different, so I hate to judge, but what you're saying doesn't add up to any experience I've had. It sounds like your company seriously screwed up the design if you can't scale your web tier code horizontally. I've also never had a view technology take up a significant chunk of cpu resources - it's always the Java code carrying out the functionality. E.g. I would expect the largest factor in CPU usage in the list of search results to be... generating the data for the search result. If the largest factor was rendering the result, then something was probably seriously wrong.

taeric 10y ago

This sounds less like technical debt, and more like liabilities of over engineering. Possibly feature creep. That is, technical debt is not necessarily tangled over-engineered code. It is more compromises that were made to actually ship and operate in the world. You can see this in the world with devices. Consider, technical debt is the reason you have AC delivered to your house going through as many converters as you do devices. Often to the same target power characteristics for those devices. It is not the reason that your coffee machine that also grinds and whatever, is likely to fail within the year. Another example; Technical debt is the reason we are still predominantly using petrol for automobiles. It is not the reason the dashboards are horribly non-responsive on modern cars.

CydeWeys 10y ago
> Consider, technical debt is the reason you have AC delivered to your house going through as many converters as you do devices. Often to the same target power characteristics for those devices. Bad example. AC power has many desirable characteristics for the local transmission grid. If you were to do the grid over from scratch you'd still use AC. You're also too focused on household electronic usage, which is a very tiny percentage of the overall electricity used.
- taeric 10y ago
  My understanding is that HVDC had advantages. That said, I was also intending that to include the distribution in your house.
  - napoleond 10y ago
    HVDC does have advantages in certain scenarios (very long transmission lines, for example) but parent is still correct--the majority of the grid makes way more sense with AC.
    - taeric 10y ago
      I meant my follow-on to be a concession, but worded it poorly. I thought it had advantages, but yes, I was thinking small appliance mainly. In particular, in home. And not just computers, but lights and control panels. Seems many things all use the same power characteristics and are now becoming complicated by dealing with AC. Which, amusingly, is fitting for the tech debt debate. Eradicating some choices from the project is likely to be missing the point. Just as eradicating AC from all power would be short sighted/wrong.

kisstheblade 10y ago

You had a search engine and rendering the search results was the bottleneck? That's really weird. Don't know a lot about JSF but other templating languages are usually really not ever the bottleneck. Maybe if you have some giant table with thousands of cells each with its own complicated template directive (for loop with conditionals etc).

agrafix 10y ago

What was/is the product/company called if i may ask?

klibertp 10y ago

> We tried to take over the framework by ourselves but it was written in Eiffel and the code was a horrible entangled mess. Eventually we rewrote it in Java but, being a start-up, we lost too much precious time already I wonder, would the result be different if you had access to competent Eiffel developers? How large was the Eiffel codebase? Eiffel is an interesting language, with a somewhat unique feature-set (I think only Ada is coming close). Design by contract and static typing as core language features - if used right - should greatly help with both stability and ease of refactoring. How large the codebase was is an important question, also how bad it really was. I saw a similar story - external codebase getting worse and worse from some point on - with Clojure at the center. The code quality was quite ok for a couple of months, then it worsened. At that point and for a couple of following months the codebase was possible to save - a single competent Clojure programmer would make a difference, I think. The project was less than 10k LOC then. However, more than 1.5 years and 60k LOC later, doing anything became nearly impossible for anyone, including original authors.

edblarney 10y ago

"Eventually the word came out: the main developer of the DAL framework had left the company and, according to the Greek CEO, she had been 'too smart' which meant that nobody understood her code." OMG no - run for the hills. 95% of software systems are not inherently sophisticated - they are 'complex' - yes - maybe there are many features, and moving parts - but there are no pieces of the system that should be hard to understand by anyone. Decent architecture + decent design and coding and an entire banks system should read like a long, but well articulated user manual. Unless you're doing super low-level stuff, complex algorithms, heavy math stuff, or issues with massive scale or performance etc. ... the end result should almost be mundane in most cases.