In practice, it seems to create a black hole where a school that serves a population that isn't good at tests (e.g disabled kids, non English speakers) gets low

In practice, it seems to create a black hole where a school that serves a population that isn't good at tests (e.g disabled kids, non English speakers) gets low scores so everyone who could get a good score leaves and then the school does even worse and gets less funding and then the school is failing and the "fix" is to fire the administration, spend money bringing in new staff who don't know the area or the students, and expect that to solve the systemic issues revealed by poor test scores. To avoid this, schools are indeed incentivized to improve - for instance, by directing low performing students to stay home on the day of the test, or by discouraging students with learning difficulties from attending at all.