I agree that removing the points may cause the issue where someone is upvoted tremendously without merit. After all, I often decide whether to upvote a comment

I agree that removing the points may cause the issue where someone is upvoted tremendously without merit. After all, I often decide whether to upvote a comment based on its current score. If I see someone with a score of 20 for a reasonable comment yet someone with a score of 1 for a brilliant comment, I would be more likely to upvote the lower score. I think this would cause a bit of Karma inflation if the points were not visible.