5 ms·

Is boring technology even an option in high-velocity fields like ML/AI? I can pick boring databases and boring servers but when it comes to model architectures

Is boring technology even an option in high-velocity fields like ML/AI? I can pick boring databases and boring servers but when it comes to model architectures, model weights, GPUs, serving frameworks, etc. there's been such massive improvements lately that it doesn't feel like anything has settled enough to be "boring". If I need a language model, or an image classifier, or an ASR model, there are older options which might be "boring", but they're not nearly as powerful as the latest options. Does choosing boring ML tech mean sitting out on massive recent advancements?

dgrin91 3y ago
99.999...% of us don't work or seriously touch in AI in any way. At best you make an API call to something that might do some AI, but at that point its just another service. So for the vast majority of people doing dev work, boring choices are correct. I've had a lot of junior devs come to me with proposals for microservice architectures that can autoscale, be orchestrated, have 0 downtime with red/blue deployments, etc, etc, when at the end of the day it could be done with a boring, single server (plus second for resiliency, but not needed for scaling) with just a few endpoints. It will be done much faster & simpler, which means it will be cheaper and less likely to fail. My guess is that the same is true in a lot of AI work. Yes there is core stuff that needs to be performant, scalable, and modern, but I'm sure there is also a ton of boring old regular internal corporate infra that can be done 'boring'.
- threeseed 3y ago
  The first approach will be significantly cheaper if using the cloud. It allows you to use spot/ephemeral instances and scale down capacity to almost zero cost without any loss in availability. Single server in the cloud is great for dev environments and if you don't care about uptime. But given that poor uptime is toxic for customer retention only incompetent businesses would deliberately not choose to use something better suited. Especially when it's so much cheaper.
  - lelanthran 3y ago
    > The first approach will be significantly cheaper if using the cloud. Only if you're regularly scaling down to zero (AKA too few paying customers to stay in business). > only incompetent businesses would deliberately not choose to use something better suited. There's a time and place for both approaches. If you're serving anywhere near the number of users needed to show a profit, spot instances are extremely expensive.
    - threeseed 3y ago
      Ignoring managed services: Spot instances are the cheapest types of instances you can get. Auto-scaling the smallest compute instances for your use case is the most efficient architecture you can get. It doesn't matter what time or place it is. These are facts that you can check for yourself on AWS/GCP etc cost calculators.
ryanschneider 3y ago
I think the whole point is that you only have a limited “cutting edge budget” so if you are somewhat forced to spend that budget on cutting edge ML models then don’t overload your team _also_ using cutting edge languages or infrastructure or databases. So save your budget for what will really make your product unique instead of “wasting” it on things that in the end won’t give you a real competitive advantage.
jvans 3y ago
Yes. Use GBTs and off the shelf language models you don't train yourself.
- threeseed 3y ago
  But that choice has absolutely nothing to do with being boring or not. It's a business/product decision. ChatGPT costs serious money, is easy to integrate with and is great for specific, general tasks. But it hasn't been trained on recent datasets, can't take advantage of internal or custom datasets and has restrictions on how it can be used. Also most companies will never allow exfiltration of sensitive data to a third party.
riku_iki 3y ago
> Does choosing boring ML tech mean sitting out on massive recent advancements? that's the point that for new tech it is hard to see where is the hype and where are real advancements which can be useful, and author's choice not to take the risk. Say you bought the hype of GPT, and integrated it to your product to unlock new cases, but it also can add lots of confusion and quality issues, which author prefers to avoid.
jpgvm 3y ago
The point is to constrain innovation to the areas that you vitally need it. Say you are building some sort of visual AI startup, say something that does generative AI things with videos. You would a) use a boring AF web stack b) use ffmpeg and c) don't go and try do a bunch of silly serverless things, d) try use one of the relatively settled model serving platforms if it's viable for you. Then you can use all of your innovation budget on your model itself, your training infrastructure etc, things that will provide you with a competitive advantage over people trying to do the same thing as you.

dgrin91 3y ago

99.999...% of us don't work or seriously touch in AI in any way. At best you make an API call to something that might do some AI, but at that point its just another service. So for the vast majority of people doing dev work, boring choices are correct. I've had a lot of junior devs come to me with proposals for microservice architectures that can autoscale, be orchestrated, have 0 downtime with red/blue deployments, etc, etc, when at the end of the day it could be done with a boring, single server (plus second for resiliency, but not needed for scaling) with just a few endpoints. It will be done much faster & simpler, which means it will be cheaper and less likely to fail. My guess is that the same is true in a lot of AI work. Yes there is core stuff that needs to be performant, scalable, and modern, but I'm sure there is also a ton of boring old regular internal corporate infra that can be done 'boring'.

threeseed 3y ago
The first approach will be significantly cheaper if using the cloud. It allows you to use spot/ephemeral instances and scale down capacity to almost zero cost without any loss in availability. Single server in the cloud is great for dev environments and if you don't care about uptime. But given that poor uptime is toxic for customer retention only incompetent businesses would deliberately not choose to use something better suited. Especially when it's so much cheaper.
- lelanthran 3y ago
  > The first approach will be significantly cheaper if using the cloud. Only if you're regularly scaling down to zero (AKA too few paying customers to stay in business). > only incompetent businesses would deliberately not choose to use something better suited. There's a time and place for both approaches. If you're serving anywhere near the number of users needed to show a profit, spot instances are extremely expensive.
  - threeseed 3y ago
    Ignoring managed services: Spot instances are the cheapest types of instances you can get. Auto-scaling the smallest compute instances for your use case is the most efficient architecture you can get. It doesn't matter what time or place it is. These are facts that you can check for yourself on AWS/GCP etc cost calculators.

ryanschneider 3y ago

I think the whole point is that you only have a limited “cutting edge budget” so if you are somewhat forced to spend that budget on cutting edge ML models then don’t overload your team _also_ using cutting edge languages or infrastructure or databases. So save your budget for what will really make your product unique instead of “wasting” it on things that in the end won’t give you a real competitive advantage.

jvans 3y ago

Yes. Use GBTs and off the shelf language models you don't train yourself.

threeseed 3y ago
But that choice has absolutely nothing to do with being boring or not. It's a business/product decision. ChatGPT costs serious money, is easy to integrate with and is great for specific, general tasks. But it hasn't been trained on recent datasets, can't take advantage of internal or custom datasets and has restrictions on how it can be used. Also most companies will never allow exfiltration of sensitive data to a third party.

riku_iki 3y ago

> Does choosing boring ML tech mean sitting out on massive recent advancements? that's the point that for new tech it is hard to see where is the hype and where are real advancements which can be useful, and author's choice not to take the risk. Say you bought the hype of GPT, and integrated it to your product to unlock new cases, but it also can add lots of confusion and quality issues, which author prefers to avoid.

jpgvm 3y ago

The point is to constrain innovation to the areas that you vitally need it. Say you are building some sort of visual AI startup, say something that does generative AI things with videos. You would a) use a boring AF web stack b) use ffmpeg and c) don't go and try do a bunch of silly serverless things, d) try use one of the relatively settled model serving platforms if it's viable for you. Then you can use all of your innovation budget on your model itself, your training infrastructure etc, things that will provide you with a competitive advantage over people trying to do the same thing as you.