If everyone's goals were pure, you're correct. But if I continued to farm and then caused the people around me to become indebted to me in exchange for the food

If everyone's goals were pure, you're correct. But if I continued to farm and then caused the people around me to become indebted to me in exchange for the food, then I could perhaps force them to farm for me in order to pay their debts; of course, eating any food would put them farther into debt so they would never get out, but I wouldn't have to farm any more yet would still own everything. I could enforce this by picking the meanest, biggest people in my town and feeding them in exchange for their protection. You should remember that prior to the invention of capitalism and profit as a calling in the 19th century, many people in the western world lived like this.