2 ms·
Given section A7 in your paper: https://arxiv.org/pdf/2505.20266 https://arxiv.org/pdf/2505.20266 ...would it be accurate to say that syftr finds Pareto-optima
by djoldman 1y ago
Given section A7 in your paper: https://arxiv.org/pdf/2505.20266 https://arxiv.org/pdf/2505.20266
...would it be accurate to say that syftr finds Pareto-optimal choices across cost, accuracy, and latency, where accuracy is decided by an LLM whose assessments are 90% correlated to that of human labelers?
Are there 3 objectives: cost, accuracy, and latency or 2: cost and accuracy?
- physicalrobot 1y agoWe did two separate studies, one of accuracy and cost the other of accuracy and latency. Most of our studies used cost as the definition is pretty clear and less sensitive to environmental conditions (e.g. LLM provider changes) or definition (e.g. typical vs. worst case). But latency is a concern for many usecases, so we did want to investigate that problem as well. Note we used Random LLM which had a 0.84 correlation with human labeling rather than 0.90 from only using gpt-4o-mini.