3 ms·

Wait... judging by their own press release, this thing looks terrible. They didn't benchmark it on finance-specific tasks against GPT-4, or even GPT-3.5, just a

• mdredze 4y ago
  Hi! I'm a co-author on the paper (Mark Dredze). We didn't compare to GPT-4, or any instruction tuned model. We're comparing a causal LM pre-trained only (BloombergGPT) to other causal LM pre-trained only models. You need to compare like to like. Would an instruction tuned model work better? Sure! I would hope so. That seems like a good next step to explore. It's also difficult to make scientific comparisons to models that we know nothing about. I don't know what data GPT-4 or GPT-3-5-turbo used for training, how many parameters they have, etc. We can't make any scientific conclusions when comparing two systems if we don't know anything about one of the systems. We compared to GPT-3 results when they were available. Does GPT-3 do better on general purpose datasets? Yes, as we expected it would. It's a much larger model (~3.5 times the size). No surprise there. If anything, the surprise was how well bloombergGPT did against these larger models, beating them on some datasets.