52% correct seems awfully high - I am guessing it's inflated because of the easiness of the questions on StackOverflow. In my experience, at least in my domain,

52% correct seems awfully high - I am guessing it's inflated because of the easiness of the questions on StackOverflow. In my experience, at least in my domain, I will be lucky it it can generate an answer that is more than 5 lines of code without a single error. Still incredibly useful - just have to handhold it and watch it very carefully when going through a solution.