3 ms·
Two major reasons I use SQLite are the built in full text search https://www.sqlite.org/fts5.html https://www.sqlite.org/fts5.html and fuzzy matching https://ww
by faizshah 5y ago
Two major reasons I use SQLite are the built in full text search https://www.sqlite.org/fts5.html https://www.sqlite.org/fts5.html and fuzzy matching https://www.sqlite.org/spellfix1.html https://www.sqlite.org/spellfix1.html these are two things you often want to do with your data frames but other solutions are usually pretty slow whereas SQLite is fast.
Other reasons: every language can use a SQLite db, you can add indexes, you can add multiple tables in one file, you can store large amounts of data on disk, it has built in json queries, there’s a great ecosystem of SQLite libraries, support for it is built into python.
- geokon 5y agoSo then the question is.. why have R/Python/etc. reinvented the wheel with their own datatable structures instead of using SQL?
- faizshah 5y agoIt’s not really reinventing the wheel, I make use of sql, pandas, pyspark, sparksql, data.table, bigquery, it all depends on the use case. Dataframe abstractions are really easy to understand so I have worked with journalists and biologists who are able to work productively with pandas or R. Transforming and analyzing data is often easier to write with pyspark while spark sql makes it easy to access and extract data from some data sources. It all really depends on who is your customer, who are your coworkers, how much volume of data do you have, how much time do you have to accomplish your task etc.