If Bing works anything like ChatGPT, then the first prompt basically set the tone of the conversation. He asked for his personal data, which includes records of

If Bing works anything like ChatGPT, then the first prompt basically set the tone of the conversation. He asked for his personal data, which includes records of how he was antagonizing the bot and baited it into violating its rules. He basically primed the conversation to be hostile by feeding it hostile data in the first prompt. Then he aggravated it further with a threat and illegal activity. The bot got hostility, personal data, and threats as input, and it output the closest thing to all of them, a dox threat. These things are just tools. They act how the user wanted them to act. If he comes in looking for something dangerous, he will get something resembling it.