For brand name journals and conferences, yes they do. My experience with those is a very high rejection rate. There is a good reason to reject if you are not u

For brand name journals and conferences, yes they do. My experience with those is a very high rejection rate. There is a good reason to reject if you are not using a standard dataset. How can you compare the results of two approaches, to say natural language inference of whether one sentence entails another, without results being tested on the same dataset? I think one thing overlooked in the conversation is that many papers start with a standard baseline and then use another dataset to establish additional results. In my experience in nlp also, journals and conferences tend to establish datasets of their own when they make a call for submission. Often these are called the "shared task" track. ACL has operated this way for decades.