There are a lot of direct technical reason this might not work (not all edge cases are sufficiently sampled). But there is also a "fundamental" issue of it bei

There are a lot of direct technical reason this might not work (not all edge cases are sufficiently sampled). But there is also a "fundamental" issue of it being difficult/impossible to enumerate "bad behaviors". This is an issue related to a lot of AI safety, including AGI safety as discussed by for example in Nick Bostrom's "Superintelligence" (https://www.amazon.com/dp/B00LOOCGB2 https://www.amazon.com/dp/B00LOOCGB2)