Every one of the neighbors holding the right under the covenants could sue. They could either seek damages for however much they could show that harms them (goo

Every one of the neighbors holding the right under the covenants could sue. They could either seek damages for however much they could show that harms them (good luck, but sometimes more modern covenants will provide a minimum damages amount), or they could seek an equitable remedy and have a court order that anything built past the setback be torn down. If none of the neighbors care, theoretically you'd be fine to ignore since no one else has standing to sue, but that doesn't preclude the possibility that a neighbor would sell it someday to someone who wants to shake you down. If everyone's fine with it, you really should get them to give up their interests in the covenants.