I‘m a Chief Data Officer in the beverage industry. My day job is creating layers to make the incoming information meaningful. Over the past years we went from G

I‘m a Chief Data Officer in the beverage industry. My day job is creating layers to make the incoming information meaningful. Over the past years we went from Google BigQuery to self hosted solutions using SQLite and Postgres. Recently even using JSONL for major parts of our data pipeline. I discovered the raw performance of languages like C and C++ compared to web scripting languages. I try to follow UNIX principles which means we create simple tools to work with our data that can be piped. What I learned over the years: it‘s easy to fell quickly in love with great SaaS services. At a given point in time you realize the business is (data in = super easy, data out = tough) Human readable, append only formats like JSONL over flexibility. The strategy is that you can reprocess the data again if necessary using the tools available that year. I believe we will have tools in two years that are working superior with structured text files like JSONL.