Not the case, actually. It's safe to assume that the FBI is very big on backups, probably paper backups, to prevent people doing just that. But stealing the bit

Not the case, actually. It's safe to assume that the FBI is very big on backups, probably paper backups, to prevent people doing just that. But stealing the bitcoin from the FBI wouldn't require deleting anything, just access to the private key. Now admittedly, the FBI also has a lot of experience keeping short bits of data secret (for instance, the real identities of moles and protected witnesses). But they don't exactly have a perfect record on that count.