3 ms·

Some very detailed advice in this response - thank you for taking the time to write it all up. > What is the scale of data that you're working with? It's not

Some very detailed advice in this response - thank you for taking the time to write it all up. > What is the scale of data that you're working with? It's not all that large. Mostly text data. Around 2 TB all combined. > How many analysts will be querying this data? Again, not that large. Around 4 to 5 analysts building reports, but potentially several thousand users in the field using said reports. That said we can easily scale out reads if need be. I was more just wondering about the ingest side of things (hope I'm using the right terminology here, forgive me if not). Any good books on this topic you'd recommend? Would love to hear your thoughts. Thanks again.

gigatexal 4y ago
There’s an O’Reilly book on data engineering. I’d recommend that. And then the data engineering podcast from Tobias Macey is really good, too. I’d stand up a set of replicas in a sort of datamart for those thousands of report end users. If the reports are read only and they are then a replica should be just fine. And an even beefier replica for the analysts to create the reports. You might then graduate to something like looker or tableau. Happy to chat more about this. This is what I do all day. I love this stuff.