3 ms·

Is this a managed hadoop/hive cluster with some sort of field heuristics baked in? Looks cool and very useful for a certain set of users.

• dboreham 9y ago
  No.
• simonw 9y ago
  Nope, it's much more simple than that. Each CSV upload is converted into a SQLite database. Then I deploy a static, read-only copy of that database to a Zeit Now hosting account and fire up my "datasette" command-line tool (written in Python 3) which opens the SQLite database and starts serving up queries from it. https://github.com/simonw/datasette https://github.com/simonw/datasette The magic here is that the SQLite database is opened in read-only mode, which means I don't have to worry about concurrent access or write activity. SQLite is screamingly fast if you use it in this context. Because the data is read-only, if you ever DO need to scale to handle more traffic you can do so by firing up additional instances, each with their own copy of the SQLite database file.
  • brandonlipman 9y ago
    This is awesome. Could this be extended so that new CSV uploads can be added at a later date and deduplicate?
    • simonw 9y ago
      Zeit Now deployments are immutable, but I can have an "edit" feature which retrieves the data used for a deployment, lets you modify it (add new CSV files for example) and then deploys a brand new instance with the new changes. Zeit supports URL aliases (which I need to expose in Datasette Publish) so you can point an alias at the fresh deployment. Stuff like CSV de-deplication is out of scope for Datasette Publish - it's something you would need to do with other tools (e.g. Excel) before uploading the data.