Your imagination is incorrect. Barium is indeed toxic, as is the case for pretty much every metal at the bottom of the periodic table even if they aren't radioa

Your imagination is incorrect. Barium is indeed toxic, as is the case for pretty much every metal at the bottom of the periodic table even if they aren't radioactive. However, it is one of the more mild toxins, and its toxicity is heavily mitigated by the fact that it's one of the least soluble cations (i.e., barium-anything is pretty insoluble), so it would be relatively difficult to actually ingest.