4 ms·
Sqlite for data science. You can do so much with a few lines of SQL, where people usually write error prone, slow python scripts. Importing CSV or TSV data is t
by manx 5y ago
Sqlite for data science. You can do so much with a few lines of SQL, where people usually write error prone, slow python scripts. Importing CSV or TSV data is trivial.
- psadri 5y agoI can imagine a notebook like environment where the results of each step are put into a temp table to use in subsequent steps. Why do you think people are not using it enough? Because they don’t know SQL? Or because of tooling?
- kingkongjaffa 5y agoWell because usually you pull some data from a service API as csv or Json and you’re not directly pulling from a SQL connection since that isn’t exposed. Even internally across different teams. With all the other technologies falling back to json, modern web dev only thinks about SQL as a store for CRUD operations, and not as an integral part of the stack. At least that’s how it seems to me.
- jstx1 5y agoIt makes only if you're doing a few operations and all of them are on tables. As soon as people need to mix up table operations and other code (custom functions and calculations, REST API calls, linear algebra etc) in the same script, it makes a lot more sense to use a dataframe and have all of your code in the same language. If there's a way in which sqlite is underrated by data scientists, it's as a storage format - it's a lot more reliable than passing around csv files.
- manx 5y agoAt least for preprocessong csv files it makes a lot of sense. You can pipe data into it, run a query and output the result in many formats.