2 ms·

I am suspicious of the use of MAE as a metric. I would like to see a range of metrics, especially RMSE that penalizes large differences more. If we can pick the

I am suspicious of the use of MAE as a metric. I would like to see a range of metrics, especially RMSE that penalizes large differences more. If we can pick the metric to judge the models by surely some metrics make the results look better than others.

rvacareanu 2y ago
If you're interested in exploring other metrics, the output of the models and code examples for how to read them are available at: https://github.com/robertvacareanu/llm4regression/tree/main/data/outputs https://github.com/robertvacareanu/llm4regression/tree/main/... For example, slightly adapting the code from the `README.md` for re-creating a barplot similar to Figure 1: ``` print(cdf.groupby(by=['dataset', 'model']).apply(lambda x: np.sqrt(((x['pred'] - x['gold']) ** 2).mean()))) ``` The code above prints (the dataset is Original #3): ``` dataset model original3 Claude 3 Opus 11.862656 DBRX 32.186706 GPT-4 31.672733 Gradient Boosting 21.985396 KNN 55.043003 Linear Regression 29.987077 Random Forest 31.688074 ```