Metaphorical Fruit

Whenever I hear people say "that's like comparing apples to oranges" it never makes sense to me.

In fact, there is no two fruits that are more alike than apples and oranges. I mean, they're both fruit, they're both roughly spherical and they serve the same social purpose. Like, if I was having lunch with someone eating and apple, and I looked away for a moment and he suddenly decided to switch it with an orange, I don't think I would have noticed. I can't think of anything can be more similar to and orange than and apple.

So how can someone use this phrase as a metaphor for making the distinguish of a difference. I could better understand if you said something like "that's like comparing apples and vitamin C pills", or "that's like comparing apples and JRR Tolkein", or "that's like comparing apples and the internet", or "that's like comparing apples to the steady decline in our society's ability to communicate a simple concept". Those are much better examples of comparison. But not apples and oranges.