The way F-Droid (android app distribution service) does this is all builds are done server-side (devs only update the publicly available source) and then anyone

The way F-Droid (android app distribution service) does this is all builds are done server-side (devs only update the publicly available source) and then anyone can quickly setup a verification server that builds+diffs the packages https://verification.f-droid.org https://verification.f-droid.org