Ask HN: Do you have experience running an open source web app?

As many of you may know, HipHopGoblin is now an open source project. And due to popular demand, it is going to be back up and running soon. What's the best way to make sure that changes pushed to the Github repository don't break the app? A suite of tests? A staging server? What kind of process should we use? Thanks!