3 ms·
We love this aspect of SQLite and leverage it to great advantage. One of our projects is a tool for customers to validate that data files submitted at regular i
by vector_spaces 5y ago
We love this aspect of SQLite and leverage it to great advantage. One of our projects is a tool for customers to validate that data files submitted at regular intervals comply with hundreds of data and business rules before submitting them to a lengthy ETL process, which will need to be repeated if there are data issues
The tool loads the data into a SQLite database, and runs a battery of SQL queries against it which represent the rule checks. The queries return data that fail our checks. Since SQLite is dynamically typed, we can load the data in just fine even if it is incorrectly typed, but still leverage the awesomeness of a relational database along with the particular awesomeness of SQLite which allows us to define ephemeral, connection-specific UDFs with Python code.