This is a very cool idea, I think this could potentially be more powerful by (at least initially) limiting it to small sections of the UI (Like you've done in y

This is a very cool idea, I think this could potentially be more powerful by (at least initially) limiting it to small sections of the UI (Like you've done in your example) You could take it a step further and, instead of adding/removing buttons, simply updating the copy of (a) button(s) based on the question, which I imagine you could manage with a very small model.