> The evaluation tasks are sourced from experienced industry professionals (avg. 14 years' experience), 30 tasks per occupation for a total of 1320 tasks. Gradi

> The evaluation tasks are sourced from experienced industry professionals (avg. 14 years' experience), 30 tasks per occupation for a total of 1320 tasks. Grading is performed by blinded comparison of human and model-generated solutions, allowing for both clear preferences and ties. It's important to carefully scrutinize the tasks to understand they actually reflect tasks that are unique to industry professionals. I just looked quickly at the nursing ones (my wife is a nurse) and half of them were creating presentations, drafting reports, and the like, which is the primary strength of LLMs but a very small portion of nursing duties. The computer programming tests are more straightforward. I'd take the other ones with a grain of salt for now.