It is voluntary. The convention is used in BIG-bench [1] and mentioned in their README here: > All task files (including README.md and task.json) contain a "c

It is voluntary. The convention is used in BIG-bench [1] and mentioned in their README here: > All task files (including README.md and task.json) contain a "canary" string, which should not be edited. This is to prevent benchmark tasks from leaking into web-scraped training data. It's not universally adopted and of course anyone is free to ignore it. [1] https://github.com/google/BIG-bench#creating-a-json-task https://github.com/google/BIG-bench#creating-a-json-task