It's better to let everyone flag stories (or better yet - downvote). But the weight of every flag should depend on how well user's downvoting correlate with yo

It's better to let everyone flag stories (or better yet - downvote). But the weight of every flag should depend on how well user's downvoting correlate with your own downvoting. If there is no correlation between user and you - then downvoting should not affect ranking. That way you would get types of stories you don't like on HN to be quickly downvoted.