While they could let it slide, they won't, because that could trigger adverse possession from him. He could wait ten years, then sue to gain ownership of the l

While they could let it slide, they won't, because that could trigger adverse possession from him. He could wait ten years, then sue to gain ownership of the land, arguing that because they didn't kick him off when they found out, they were implicitly granting him ownership. He'd win, too. You have to make a positive defense against adverse possession, meaning if you know about it you have to do something about it.