Are there any multimodal models where you can annotate your time series with events in natural language, and have it identify other occurrences of similar event

Are there any multimodal models where you can annotate your time series with events in natural language, and have it identify other occurrences of similar events? Or where you can ask it to do a forecast assuming some new event, also in natural language? I feel like this is what I'd want out of a DL time series NN. For just prediction it seems like overkill.