10 ms·

TBQH I'd bet the following would yield pretty good results: 1. Take your existing incident reporting / review docs (you have those, right?) that cover everythi

TBQH I'd bet the following would yield pretty good results: 1. Take your existing incident reporting / review docs (you have those, right?) that cover everything including 5-why incident reporting and analysis. 2. Fine-tune a Llama-3.1-70b [1] LoRA on the data associated with the outage as input, and the root cause analysis as the output 3. Tada! You have a state-of-the-art custom LLM that is good at analyzing your outages and guessing what the root causes might be. It's a little shocking sometimes to me how underutilized fine-tuning is. Most of the "learning" happening in "machine learning" is in training — yes, it's definitely true that LLMs can exhibit a surprising amount of "in-context learning" via prompts, but it's a surprising because learning during training is so much more powerful, and it's surprising that in-context works at all. Honestly even just fine-tuning an 8b model — which is totally doable on a 3090/4090 — can yield SOTA results on task-specific performance. It's so much better than prompting! 1: I mean you could also finetune 405b, but you'll need a lot of GPUs both to train it and to run it. In my experience (public benchmarks be damned; the models all tend to saturate the public benchmarks, even though everyone claims not to train on them), 70b on internal evals tends to perform similarly to gpt-4o, and with OSS models you have somewhat more ownership and control.

gbasin 2y ago
For large models that are well-trained with large context support, ICL seems to work about as well as FT
- reissbaker 2y ago
  It's totally task dependent. On some tasks, large well-trained models are already great; if the large model already exhibits human-level performance, a small-model finetune is unlikely to beat it. Similarly, on very general tasks (e.g. "coding" as a general task, as opposed to "writing idiomatic NextJS" being a specific task), a small-model finetune will be unlikely to beat a large model. But there are plenty of tasks that even large, well-trained models struggle with. If the OP is struggling to get useful root-cause analysis for cloud service incidents out of an existing large model, that seems exactly like a use case where a finetune would shine. Also, finetunes don't have to be just for small models! Medium-sized models like Llama-3.1-70b can be finetuned, and if you want to burn a lot of GPUs you can finetune 405b as well.
  - gbasin 2y ago
    Yes I was specifically thinking of the latter with larger models, I think many many shot ICL still tends to outperform but you're right it's worth trying both for your use case
Infinity315 2y ago
I take issue with this. Typically issues arise because they are novel and are unforeseen. If we did see these issues beforehand they'd be fixed! LLMs by definition are trained by example, so I fail to see how finetuning LLMs on things that have already happened to be helpful for determining the root cause of a novel issue. LLMs seem to lack systemic modelling that humans do. I can see LLMs being practical for this if it is shown that LLMs are capable of modelling scenarios outside of their dataset, but thus far none such examples exist.
- reissbaker 2y ago
  During my time taking on-call pager rotations at fairly large engineering organizations, I'd say that a distressingly large number of incidents happened for fairly standard reasons ("didn't write tests for this case" + "deployed" + "insufficient monitoring," with things like incorrect concurrency assumptions around DB access, or memory leaks in application code, or insufficient rate limiting / circuitbreaking causing cascading failures being fairly common passengers). In fact, it's actually pretty hard for me to come up with an issue that wasn't basically a fairly common programming error combined with some relatively common infrastructure, build, test, and/or observability problem. Sometimes with some common bureaucratic human problems thrown into the mix too, i.e. "no one owns this service's uptime."
  - Infinity315 2y ago
    I think we're at an epistemic impasse here. At what point would/could you be convinced that LLMs are incapable or unsuited here? If LLMs were successfully deployed in a production environment is the day I bite my tongue. What about you? I'm not even sure that LLMs are even capable of solving standard bugs see: [1]. Hallucination seems to be a significant hurdle and any time spent validating the fixes of an LLM is wasted when it could be spent tackling the bug head on. The amount of energy spent espousing garbage requires an order of magnitude more effort to invalidate. [1]. https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/ https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-f...
    - 2y ago
williamcotton 2y ago
It's a little shocking sometimes to me how underutilized fine-tuning is. I imagine it is because of the costs of building a dataset.
lkrubner 2y ago
This strategy amounts to building a RAG for your company. I have a friend who is CTO at a startup and he is doing exactly this. He has uploaded every document he can to Anthropic/Claude, using their service which allows easy construction of RAGs. He has uploaded thousands of documents. And now he can ask it questions like, "Which will increase profits more, hiring another engineer, or hiring another sales person?" I think we will see more of this. There are some big privacy concerns, but "an internal RAG for every business" will probably find some customers.
- reissbaker 2y ago
  RAG is different! RAG is entirely in-context learning. You can combine finetuning and RAG, though, and often get better results than either alone.
bradfox2 2y ago
Having done this for domain specific engineering paperwork that looks similar to cause analysis, it does work well at param sizes << 70B. lora does not work though, you need full parameter training if the knowledge isnt already present in pre training set.

gbasin 2y ago

For large models that are well-trained with large context support, ICL seems to work about as well as FT

reissbaker 2y ago
It's totally task dependent. On some tasks, large well-trained models are already great; if the large model already exhibits human-level performance, a small-model finetune is unlikely to beat it. Similarly, on very general tasks (e.g. "coding" as a general task, as opposed to "writing idiomatic NextJS" being a specific task), a small-model finetune will be unlikely to beat a large model. But there are plenty of tasks that even large, well-trained models struggle with. If the OP is struggling to get useful root-cause analysis for cloud service incidents out of an existing large model, that seems exactly like a use case where a finetune would shine. Also, finetunes don't have to be just for small models! Medium-sized models like Llama-3.1-70b can be finetuned, and if you want to burn a lot of GPUs you can finetune 405b as well.
- gbasin 2y ago
  Yes I was specifically thinking of the latter with larger models, I think many many shot ICL still tends to outperform but you're right it's worth trying both for your use case

Infinity315 2y ago

I take issue with this. Typically issues arise because they are novel and are unforeseen. If we did see these issues beforehand they'd be fixed! LLMs by definition are trained by example, so I fail to see how finetuning LLMs on things that have already happened to be helpful for determining the root cause of a novel issue. LLMs seem to lack systemic modelling that humans do. I can see LLMs being practical for this if it is shown that LLMs are capable of modelling scenarios outside of their dataset, but thus far none such examples exist.

reissbaker 2y ago
During my time taking on-call pager rotations at fairly large engineering organizations, I'd say that a distressingly large number of incidents happened for fairly standard reasons ("didn't write tests for this case" + "deployed" + "insufficient monitoring," with things like incorrect concurrency assumptions around DB access, or memory leaks in application code, or insufficient rate limiting / circuitbreaking causing cascading failures being fairly common passengers). In fact, it's actually pretty hard for me to come up with an issue that wasn't basically a fairly common programming error combined with some relatively common infrastructure, build, test, and/or observability problem. Sometimes with some common bureaucratic human problems thrown into the mix too, i.e. "no one owns this service's uptime."
- Infinity315 2y ago
  I think we're at an epistemic impasse here. At what point would/could you be convinced that LLMs are incapable or unsuited here? If LLMs were successfully deployed in a production environment is the day I bite my tongue. What about you? I'm not even sure that LLMs are even capable of solving standard bugs see: [1]. Hallucination seems to be a significant hurdle and any time spent validating the fixes of an LLM is wasted when it could be spent tackling the bug head on. The amount of energy spent espousing garbage requires an order of magnitude more effort to invalidate. [1]. https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/ https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-f...
  - 2y ago

williamcotton 2y ago

It's a little shocking sometimes to me how underutilized fine-tuning is. I imagine it is because of the costs of building a dataset.

lkrubner 2y ago

This strategy amounts to building a RAG for your company. I have a friend who is CTO at a startup and he is doing exactly this. He has uploaded every document he can to Anthropic/Claude, using their service which allows easy construction of RAGs. He has uploaded thousands of documents. And now he can ask it questions like, "Which will increase profits more, hiring another engineer, or hiring another sales person?" I think we will see more of this. There are some big privacy concerns, but "an internal RAG for every business" will probably find some customers.

reissbaker 2y ago
RAG is different! RAG is entirely in-context learning. You can combine finetuning and RAG, though, and often get better results than either alone.

bradfox2 2y ago

Having done this for domain specific engineering paperwork that looks similar to cause analysis, it does work well at param sizes << 70B. lora does not work though, you need full parameter training if the knowledge isnt already present in pre training set.