5 ms·

But that is the case now too and in my experience it swung to paying through the nose for hardware in general; as more or less a sidetrack I take on projects wh

But that is the case now too and in my experience it swung to paying through the nose for hardware in general; as more or less a sidetrack I take on projects where I optimise (mostly online) systems. Example; a few weeks ago a startup asked me to check out their setup as they were spending almost 30k$/mo on AWS. I spent a few days optimising and now they are down to less than 10k$. With some more work it will be a few 1000$; there is still so much wrong. But that is less low hanging fruit so it will be a lot more expensive. Still well worth it imho. People really bought into the ‘people are more expensive than hardware’ as an excuse to get screwed like this. For $5k in human cost, these guys (and their investors) now save 200k/year in hosting. And this is not an isolated story; I am working on another one at this very moment. Programmers have become so incredibly sloppy with the ‘autoscaling’ and ‘serverless’ cloud ‘revolution’.

ledauphin 7y ago
You're quite right that there's plenty to optimize. It's not that there isn't money in optimizing. It's that there's often not _enough_ money in optimizing to rise to the level of the top N priorities for a business.
- tluyben2 7y ago
  Agreed, until you raise it at the right level at the right time. People do not find me for nothing... Usually after the initial launch euphoria dies down and someone looks at the books and asks why such a large % of the expenditure goes there. People start looking around online and see things like ‘our application serves 200k requests/day with one 50$/mo server’ and compare that with their 30k/mo setup barely serving 50k/day and start poking around. It is usually apples and pears, but more often than not there are massive issues. Most of them I would consider beginner issues but they are not made by beginners; many senior programmers I meet simply do not know about normal forms, proper types (all are stringy), proper indexes, O(n^2) etc; they trust cloud scaling to solve it all. And it does! But it costs...
- tluyben2 7y ago
  And ofcourse, there is a limit to what you want to spend even if it might make some profit long term. You need to be able to find programmers to maintain things etc as well. If I needed something handling massive traffic while handling real business logic but not allowed to cost more than a few bucks in hosting, I would use something like [0]. But that would be silly for maintenance reasons alone. Does anyone know a modern (well maintained I mean really) equivalent though? I played around with this a long time ago and it is incredibly efficient. [0] http://datadraw.sourceforge.net/ http://datadraw.sourceforge.net/ (github; https://github.com/waywardgeek/datadraw https://github.com/waywardgeek/datadraw as sourceforge seems down) Edit; maybe I answered that last question by finding a github version: seems waywardgeek does maintain at least to keep it running.
  - zozbot234 7y ago
    > Does anyone know a modern (well maintained I mean really) equivalent though? I played around with this a long time ago and it is incredibly efficient. https://diesel.rs https://diesel.rs ? Maybe https://tql.antoyo.xyz/ https://tql.antoyo.xyz/ if you care more about ease of use.
    - tluyben2 7y ago
      Datadraw is not an ORM; it is more comparable to a statically compiled Redis. So it is far less flexible, but it is very efficient/fast. One of the purposes of Datadraw is for instance to build SQL databases on top of.
zozbot234 7y ago
> almost 30k$/mo That's like, a couple full-time developers, AIUI? Maybe even less than that. Perhaps the people who say "people are more expensive than hardware" have a point - at least in the Bay Area. Or you can move to the Rust Belt if you'd like a change.
- tluyben2 7y ago
  Sure, but my point was that they cut that bill with 20k PER month by giving me 5k one off... They gave me 10k runway to poke around but 5k was enough to fix it; it was simply that bad to start with. The low hanging fruit in most systems I see is really trivial to fix; they just have no one to do it... I bet other people here have seen that before when thrown into an existing project (and I read Spolsky at an impressionable point in my career so I am usually the one against rewriting the whole thing outright).
  - johnny22 7y ago
    seems like you deserve more of a cut than that.
    - tluyben2 7y ago
      Well, the premise going in after a quick (very quick) review of the system was: 'I will check what I can do in 5 days at $10k; I believe I can help, but if I cannot, you lose $10k. If I can help you in less time, you only pay that time.'. I do not think I can move that to some other deal with that premise. Maybe if I say; 'I will do this for 50% of the money you save in 12 months after I am done' that would work, but this is is a side thing which I do because I like optimizing things; if I sell it in another way, it's not bound to time which will make it a timesink and risk. It is a choice.
      - pdimitar 7y ago
        I am curious how do you even find a side job like that. I am definitely a spiritual brother with you because I love optimising things. But I am very unsure how do I even start a side career with that premise. Any advice?
herval 7y ago
What sort of waste you tend to see more, if you do this regularly? Is it the case that people are aware of the cost and “don’t care”, or is it surprising/hidden cost?
- tluyben2 7y ago
  There are 2 types: 1) they know the costs and thought it would scale infinitely with money but it doesn't (crashes, hangs, etc) 2) they knew it would cost more to scale but they did not expect it going up quite that fast as it does with more traffic (not linear).
falcolas 7y ago
> paying through the nose for hardware in general You also have to consider that there are limits to how parallel an application can be - Amdahl's Law - at some point even throwing hardware at a scaling issues has its limits. Of course, there's also a truism that the team who implemented the first pass won't have to support (financially or as a developer) the software when it no longer scales.
- dnautics 7y ago
  No, amdahl's law is (roughly speaking) a limit to how parallel an algorithm can be. Applications (in the sense of web apps) generally have the potential to scale via Gustafson's law, but we are (IMO) largely held back by framework and old ways of programming. https://en.wikipedia.org/wiki/Gustafson's_law https://en.wikipedia.org/wiki/Gustafson's_law
  - falcolas 7y ago
    So long as an application needs to share state between worker processes, (database, redis cluster, etc) then Amdahl’s law still applies. There’s very few modern applications that can truly scale linearly.
    - dnautics 7y ago
      Share consistent state. Eventually consistent models (most web apps) are often generally okay.
    - imtringued 7y ago
      You're confusing something. Fully consistent databases are primarily limited by the speed of communication because they need to replicate writes and queries to all nodes and wait for a response even if a node is on the other side of the planet. Unless your CPU is extremely slow (clock frequencies of a few kilo Herz) the speed of light is a significantly more important limit. This is actually a usecase where modern CPUs are more than fast enough and we don't need a significant improvement in processing speed. Faster storage and networks are welcome though.
deleted 7y ago
[deleted]
pchiusano 7y ago
I don’t know if you feel this way, but my complaint about the more hyped cloud services is that not only can they be expensive (fine) but the promised time-savings and simplicity of operating the system often doesn’t really materialize either, except in restricted circumstances that you don’t appreciate in advance and only find out later after you’ve already committed. If it really did save time and were simpler, some companies would (quite reasonably) be willing to pay a premium for that - time is money and all that. In reality it seems like people often end up with the worst of both worlds - it’s expensive, complicated, still needs a huge staff to maintain, and doesn’t even work that well.
- tluyben2 7y ago
  Well it is better than making even trivial architectures with actual hardware (I have some pictures of me hauling servers over xmas a long time ago; that was cheap (in monthlies and hardware, not in hours!) but I would and do pay a premium for that). Otherwise I do agree somewhat; most overbearing systems can be done much simpler but we are all preparing (and thus paying) for eventualities that most likely will never occur or will not influence the bottom line. Tech like AWS Lambda (of which I like the theoretical idea) are meant to remedy the issues with complexity for a premium. But that premium makes, personally, my eyes water. I cannot see any high volume operation justifying going live with it. Are there big examples of those? And how is it justified vs the alternatives (which are, besides some programmer+admin time and scalability) far more efficient?
  - garethmcc 7y ago
    There are some significant high volume cases. We work with companies doing billions of Lambda invocations per month and realising large cost saving benefits. Lambda itself is usually the smallest part of the bill as one of the advantages of building serverless applications is you shift the responsibility of certain execution to specially designed managed services as opposed to code consuming CPU cycles; for example API Gateway takes over routing, S3 takes over file system calls, etc. A large portion of savings organisations see though is in time to production, as well as the overhead of managing servers and container clusters which is a lot more costly than you might think. Especially in the environment we are in now where qualified Dev Ops talent is hard to come by and at a premium. Sure, a developer can take some time to try and learn how to put together some infrastructure, but that's time taken away from adding direct value to business needs and not to mention the fall out when things go pear shaped later because it turns out a few hours Googling doesn't turn someone into a DevOps expert.

ledauphin 7y ago

You're quite right that there's plenty to optimize. It's not that there isn't money in optimizing. It's that there's often not _enough_ money in optimizing to rise to the level of the top N priorities for a business.

tluyben2 7y ago
Agreed, until you raise it at the right level at the right time. People do not find me for nothing... Usually after the initial launch euphoria dies down and someone looks at the books and asks why such a large % of the expenditure goes there. People start looking around online and see things like ‘our application serves 200k requests/day with one 50$/mo server’ and compare that with their 30k/mo setup barely serving 50k/day and start poking around. It is usually apples and pears, but more often than not there are massive issues. Most of them I would consider beginner issues but they are not made by beginners; many senior programmers I meet simply do not know about normal forms, proper types (all are stringy), proper indexes, O(n^2) etc; they trust cloud scaling to solve it all. And it does! But it costs...
tluyben2 7y ago
And ofcourse, there is a limit to what you want to spend even if it might make some profit long term. You need to be able to find programmers to maintain things etc as well. If I needed something handling massive traffic while handling real business logic but not allowed to cost more than a few bucks in hosting, I would use something like [0]. But that would be silly for maintenance reasons alone. Does anyone know a modern (well maintained I mean really) equivalent though? I played around with this a long time ago and it is incredibly efficient. [0] http://datadraw.sourceforge.net/ http://datadraw.sourceforge.net/ (github; https://github.com/waywardgeek/datadraw https://github.com/waywardgeek/datadraw as sourceforge seems down) Edit; maybe I answered that last question by finding a github version: seems waywardgeek does maintain at least to keep it running.
- zozbot234 7y ago
  > Does anyone know a modern (well maintained I mean really) equivalent though? I played around with this a long time ago and it is incredibly efficient. https://diesel.rs https://diesel.rs ? Maybe https://tql.antoyo.xyz/ https://tql.antoyo.xyz/ if you care more about ease of use.
  - tluyben2 7y ago
    Datadraw is not an ORM; it is more comparable to a statically compiled Redis. So it is far less flexible, but it is very efficient/fast. One of the purposes of Datadraw is for instance to build SQL databases on top of.

zozbot234 7y ago

> almost 30k$/mo That's like, a couple full-time developers, AIUI? Maybe even less than that. Perhaps the people who say "people are more expensive than hardware" have a point - at least in the Bay Area. Or you can move to the Rust Belt if you'd like a change.

tluyben2 7y ago
Sure, but my point was that they cut that bill with 20k PER month by giving me 5k one off... They gave me 10k runway to poke around but 5k was enough to fix it; it was simply that bad to start with. The low hanging fruit in most systems I see is really trivial to fix; they just have no one to do it... I bet other people here have seen that before when thrown into an existing project (and I read Spolsky at an impressionable point in my career so I am usually the one against rewriting the whole thing outright).
- johnny22 7y ago
  seems like you deserve more of a cut than that.
  - tluyben2 7y ago
    Well, the premise going in after a quick (very quick) review of the system was: 'I will check what I can do in 5 days at $10k; I believe I can help, but if I cannot, you lose $10k. If I can help you in less time, you only pay that time.'. I do not think I can move that to some other deal with that premise. Maybe if I say; 'I will do this for 50% of the money you save in 12 months after I am done' that would work, but this is is a side thing which I do because I like optimizing things; if I sell it in another way, it's not bound to time which will make it a timesink and risk. It is a choice.
    - pdimitar 7y ago
      I am curious how do you even find a side job like that. I am definitely a spiritual brother with you because I love optimising things. But I am very unsure how do I even start a side career with that premise. Any advice?

herval 7y ago

What sort of waste you tend to see more, if you do this regularly? Is it the case that people are aware of the cost and “don’t care”, or is it surprising/hidden cost?

tluyben2 7y ago
There are 2 types: 1) they know the costs and thought it would scale infinitely with money but it doesn't (crashes, hangs, etc) 2) they knew it would cost more to scale but they did not expect it going up quite that fast as it does with more traffic (not linear).

falcolas 7y ago

> paying through the nose for hardware in general You also have to consider that there are limits to how parallel an application can be - Amdahl's Law - at some point even throwing hardware at a scaling issues has its limits. Of course, there's also a truism that the team who implemented the first pass won't have to support (financially or as a developer) the software when it no longer scales.

dnautics 7y ago
No, amdahl's law is (roughly speaking) a limit to how parallel an algorithm can be. Applications (in the sense of web apps) generally have the potential to scale via Gustafson's law, but we are (IMO) largely held back by framework and old ways of programming. https://en.wikipedia.org/wiki/Gustafson's_law https://en.wikipedia.org/wiki/Gustafson's_law
- falcolas 7y ago
  So long as an application needs to share state between worker processes, (database, redis cluster, etc) then Amdahl’s law still applies. There’s very few modern applications that can truly scale linearly.
  - dnautics 7y ago
    Share consistent state. Eventually consistent models (most web apps) are often generally okay.
  - imtringued 7y ago
    You're confusing something. Fully consistent databases are primarily limited by the speed of communication because they need to replicate writes and queries to all nodes and wait for a response even if a node is on the other side of the planet. Unless your CPU is extremely slow (clock frequencies of a few kilo Herz) the speed of light is a significantly more important limit. This is actually a usecase where modern CPUs are more than fast enough and we don't need a significant improvement in processing speed. Faster storage and networks are welcome though.

deleted 7y ago

[deleted]

pchiusano 7y ago

I don’t know if you feel this way, but my complaint about the more hyped cloud services is that not only can they be expensive (fine) but the promised time-savings and simplicity of operating the system often doesn’t really materialize either, except in restricted circumstances that you don’t appreciate in advance and only find out later after you’ve already committed. If it really did save time and were simpler, some companies would (quite reasonably) be willing to pay a premium for that - time is money and all that. In reality it seems like people often end up with the worst of both worlds - it’s expensive, complicated, still needs a huge staff to maintain, and doesn’t even work that well.

tluyben2 7y ago
Well it is better than making even trivial architectures with actual hardware (I have some pictures of me hauling servers over xmas a long time ago; that was cheap (in monthlies and hardware, not in hours!) but I would and do pay a premium for that). Otherwise I do agree somewhat; most overbearing systems can be done much simpler but we are all preparing (and thus paying) for eventualities that most likely will never occur or will not influence the bottom line. Tech like AWS Lambda (of which I like the theoretical idea) are meant to remedy the issues with complexity for a premium. But that premium makes, personally, my eyes water. I cannot see any high volume operation justifying going live with it. Are there big examples of those? And how is it justified vs the alternatives (which are, besides some programmer+admin time and scalability) far more efficient?
- garethmcc 7y ago
  There are some significant high volume cases. We work with companies doing billions of Lambda invocations per month and realising large cost saving benefits. Lambda itself is usually the smallest part of the bill as one of the advantages of building serverless applications is you shift the responsibility of certain execution to specially designed managed services as opposed to code consuming CPU cycles; for example API Gateway takes over routing, S3 takes over file system calls, etc. A large portion of savings organisations see though is in time to production, as well as the overhead of managing servers and container clusters which is a lot more costly than you might think. Especially in the environment we are in now where qualified Dev Ops talent is hard to come by and at a premium. Sure, a developer can take some time to try and learn how to put together some infrastructure, but that's time taken away from adding direct value to business needs and not to mention the fall out when things go pear shaped later because it turns out a few hours Googling doesn't turn someone into a DevOps expert.