It seems to be a corruption of an older phrase comparing apples to oysters. Citation from 1670: https://archive.org/details/bim_early-english-books-1641-1700_a

It seems to be a corruption of an older phrase comparing apples to oysters. Citation from 1670: https://archive.org/details/bim_early-english-books-1641-1700_a-collection-of-english-_ray-john_1670/page/205/mode/1up?q=Apple+ https://archive.org/details/bim_early-english-books-1641-170... Apples and oysters are still things you can eat, but they're at least further apart than two fruits. The book also cites the similar expression "chalk and cheese".