4 ms·

This started from him telling me that we need to keep in mind RAM usage for a language model we were planning to use. I said that we shouldn't worry about RAM r

This started from him telling me that we need to keep in mind RAM usage for a language model we were planning to use. I said that we shouldn't worry about RAM right now because RAM is cheap but apparently it isn't so cheap in the cloud so.. here we are :)

nuclearnice1 4y ago
Maybe that’s a another interesting question. “Hi, I have a big language model I’d like to run on AWS. What’s the best and cheapest way to run it?”
- that_guy_iain 4y ago
  Here is the thing, why isn't don't use cloud computing use dedicated servers often not considered the right answer. Also, it could be a case of they do hybrid hosting where they dedicated servers for the heavily lifting background work and cloud for the customer facing.
frenchman99 4y ago
Sounds too me like you're saying "Let's take Hetzner so we don't have to be smart" and your CTO is telling you to be smart. In a way, he trusts you're smart enough to figure out a better way then just throwing hardware at the problem. Also, while cloud pricing can be daunting, once you reach a point when Azure costs to much, you can look at optimizing things. There are a lot of techniques to keep cloud cost acceptable. Look up "FinOps" if you haven't already.
- that_guy_iain 4y ago
  Sometimes it's cheaper not to be smart. It's often the case. Because the reality is, most poeple aren't smart. They're average. If you're entire case depends on "we're smart" then more often than not you're going to end up with egg on your face. Your solution is to spend time and money on reducing costs. This only makes sense when the costs to move outweigh the costs to reduce costs.
alfor 4y ago
Make a 15min spreadsheet of $/month server cost and developer cost for differents usage scenario, include things that can speed up developpement. Instead of fighting. Don’t argue with him, provide numbers for the perspective he might not see. Tell him you want to help him make the best decision. He might loosen up on budget for cloud if he see that you are fumbling around to save a few Go of RAM.
- krageon 4y ago
  > provide numbers Even a spreadsheet with such numbers is primarily speculation and your upper management knows this (they make such things all the time). As such, if they did not want to agree with you before this spreadsheet won't make a difference. If they want to agree but don't want to take the time to present exactly that spreadsheet to their compatriots, it is worth making it. If you don't work for a very large company (essentially a bureaucracy), this situation probably will not occur.
enkrs 4y ago
His goal on keeping RAM low is probably for growing to thousands and millions of customers. And he’s right- right now it would look like RAM is cheap, but when you start having much users, it adds up. There is a line between premature optimization and total resource ignorance.
that_guy_iain 4y ago
Personally, if your CTO is already talking about worrying about resources, then I would be telling them they're moving to dedicated servers.
keikobadthebad 4y ago
You can get a dedicated xeon + 64GB DDR if that's enough for $50/mo (random google: https://www.binaryracks.com/dedicated-bargain/#plans https://www.binaryracks.com/dedicated-bargain/#plans ). Maybe just put the NLP part on a couple of those. But it sounds like your boss has the idea cloud is where it's at, it's hard to shift. I'd put numbers on it like how many less connections the cloud he wants to pay for can handle, vs the hosted solution for less $$$.
JamesSwift 4y ago
Its a game of tradeoffs, you are going to be spending money somewhere. Consider: - You ignore resource usage and move to dedicated hosting. You are now wasting money on inefficient processes. To remedy this you apply software engineering practices to understand the model and design it correctly. - You ignore server unit costs and stay on cloud, but improve your runtime characteristics. You are now wasting money on the hardware, but your software is efficient. To remedy this you have to learn how to manage infra and take it on as constant maintenance. It seems on your team it makes more sense to me to just pay for the cloud and stay in your software wheel-house.

nuclearnice1 4y ago

Maybe that’s a another interesting question. “Hi, I have a big language model I’d like to run on AWS. What’s the best and cheapest way to run it?”

that_guy_iain 4y ago
Here is the thing, why isn't don't use cloud computing use dedicated servers often not considered the right answer. Also, it could be a case of they do hybrid hosting where they dedicated servers for the heavily lifting background work and cloud for the customer facing.

frenchman99 4y ago

Sounds too me like you're saying "Let's take Hetzner so we don't have to be smart" and your CTO is telling you to be smart. In a way, he trusts you're smart enough to figure out a better way then just throwing hardware at the problem. Also, while cloud pricing can be daunting, once you reach a point when Azure costs to much, you can look at optimizing things. There are a lot of techniques to keep cloud cost acceptable. Look up "FinOps" if you haven't already.

that_guy_iain 4y ago
Sometimes it's cheaper not to be smart. It's often the case. Because the reality is, most poeple aren't smart. They're average. If you're entire case depends on "we're smart" then more often than not you're going to end up with egg on your face. Your solution is to spend time and money on reducing costs. This only makes sense when the costs to move outweigh the costs to reduce costs.

alfor 4y ago

Make a 15min spreadsheet of $/month server cost and developer cost for differents usage scenario, include things that can speed up developpement. Instead of fighting. Don’t argue with him, provide numbers for the perspective he might not see. Tell him you want to help him make the best decision. He might loosen up on budget for cloud if he see that you are fumbling around to save a few Go of RAM.

krageon 4y ago
> provide numbers Even a spreadsheet with such numbers is primarily speculation and your upper management knows this (they make such things all the time). As such, if they did not want to agree with you before this spreadsheet won't make a difference. If they want to agree but don't want to take the time to present exactly that spreadsheet to their compatriots, it is worth making it. If you don't work for a very large company (essentially a bureaucracy), this situation probably will not occur.

enkrs 4y ago

His goal on keeping RAM low is probably for growing to thousands and millions of customers. And he’s right- right now it would look like RAM is cheap, but when you start having much users, it adds up. There is a line between premature optimization and total resource ignorance.

that_guy_iain 4y ago

Personally, if your CTO is already talking about worrying about resources, then I would be telling them they're moving to dedicated servers.

keikobadthebad 4y ago

You can get a dedicated xeon + 64GB DDR if that's enough for $50/mo (random google: https://www.binaryracks.com/dedicated-bargain/#plans https://www.binaryracks.com/dedicated-bargain/#plans ). Maybe just put the NLP part on a couple of those. But it sounds like your boss has the idea cloud is where it's at, it's hard to shift. I'd put numbers on it like how many less connections the cloud he wants to pay for can handle, vs the hosted solution for less $$$.

JamesSwift 4y ago

Its a game of tradeoffs, you are going to be spending money somewhere. Consider: - You ignore resource usage and move to dedicated hosting. You are now wasting money on inefficient processes. To remedy this you apply software engineering practices to understand the model and design it correctly. - You ignore server unit costs and stay on cloud, but improve your runtime characteristics. You are now wasting money on the hardware, but your software is efficient. To remedy this you have to learn how to manage infra and take it on as constant maintenance. It seems on your team it makes more sense to me to just pay for the cloud and stay in your software wheel-house.