Perhaps not a perfect fit, but what I've done in the past, and found to work quite well, is to turn those big datasets you mention into an event stream. You ca

Perhaps not a perfect fit, but what I've done in the past, and found to work quite well, is to turn those big datasets you mention into an event stream. You can do it quite trivially by simply raising an event on every line read, or you can add some kind of transform to the data beforehand, etc. Whatever you need. You can obviously take this further and turn it into a service or something, but that's up to you. Here's a simple example in C# - https://ghostbin.com/paste/bzgez https://ghostbin.com/paste/bzgez