You need some familiarity with the world, though, to get many of the jokes ("In what universe would someone like Hermione get sorted anywhere but Ravenclaw?" -

You need some familiarity with the world, though, to get many of the jokes ("In what universe would someone like Hermione get sorted anywhere but Ravenclaw?" - "Ron's just..." "You just don't see any good reason for him to exist?" "Yeah.") and also knowing the story allows you to anticipate certain realities before Harry gets to grip with them. For example, we all know that Voldemort is Professor Quirrell, but Harry doesn't (at least not as far as I've read). Background knowledge definitely helps with the enjoyment - and imho is the only reason why you'd want to read the JK Rowling books or watch the movies! I mean, I don't dislike the books (I've read them), but they're not much more than decent entertainment (with some of them dragging on, and on, and on).