3 ms·
This sounds unintuitive, but the most lightweight API for analytics work I've come across is a SQL view. It provides one level of indirection from the underlyi
by wenc 8y ago
This sounds unintuitive, but the most lightweight API for analytics work I've come across is a SQL view.
It provides one level of indirection from the underlying data structures (if the data changes, you can just rewrite the view), and most db's provide very good access control. The view's schema is your interface contract, so users can rely on its output.
I was all about building REST interfaces to our backend databases until I realized that SQL views were a simpler, more efficient abstraction for analytics/machine learning work.
You may be like, "SQL views aren't REST APIs! It doesn't seem kosher!"
But if you think about it, REST APIs were designed for retrieving small amounts of data quickly using simple parameters, which is useful for low-latency web applications.
For analytics/machine learning work, we generally don't want that. We usually want to retrieve large chunks of pre-computed tabular data. And there's already a language for that... SQL. And SQL databases are naturally fast for large chunks of data (with optimizations like indexing, compression, etc. your results can get even faster... plus no JSON serialization cost).
If you build a REST API to SQL datasets, you have to first (inefficiently) parse the returned JSON, and reconstruct the dataframe for downstream analytics/machine learning work. Why do that, when you can use SQL (with all of its attendant data aggregation/manipulation keywords) to efficiently retrieve data via a binary wire protocol (native drivers or ODBC/JDBC) into a dataframe-like object?
- Serow225 8y agoCan you please explain what the interface looks like a little more concretely?
- wenc 8y agoIt's exactly as it sounds: a SQL view on a database server. It manifests as a SQL table backed by a query. It's mind-bendingly simple. You can connect to it via a native db connector or ODBC. It's also accessible to a wide range of tools including Excel, Tableau, or any number of programming languages. In Python you can pull data from it into a Pandas dataframe via TurbODBC (which recently added Apache Arrow in-memory support). In R, you can connect to it via dplyr. Apache Spark can get data from it via JDBC. You can experiment with it via Jupyter Notebooks. SQL support is ubiquitous.
- learc83 8y agoI've started using a rails model backed by a SQL view anytime I need to do something more complicated than book.author.name with active record. Then I can treat it just like any other rails model. If you add a controller and a route you can expose it just like you would any other read restful resource with the caveat that it's read only. As an added bonus for analytics that don't need to be real-time I can turn the view into a materialized view that updates periodically for easy caching. I'm using the Scenic gem to handle view versioning, which all let's me write all of my complicated queries in a dedicated SQL file instead of an embedded string without syntax highlighting.