6 ms·

Welcome to Datasette Cloud

317 points by swyx 3y ago

OnlyMortal 3y ago
I thought this was a reference to the C64 C2N Datasette tape drive. Unfortunately not. https://en.m.wikipedia.org/wiki/Commodore_Datasette https://en.m.wikipedia.org/wiki/Commodore_Datasette
- syslog 3y ago
  I was hoping for the same. Still have the 1531 lying in the attic (next to the C16).
- Findecanor 3y ago
  I had been expecting something in the spirit of the "Floppy RAID" [1], but something even more ridiculous on even older technology. 1: https://youtu.be/1hc52_PWeU8 https://youtu.be/1hc52_PWeU8
  - OnlyMortal 3y ago
    That’s funny. His comedy reminds me of the man who did “Roadworthy Rescues”. https://youtu.be/-A8cvrTgqGk https://youtu.be/-A8cvrTgqGk
- simonw 3y ago
  I wrote my first "database" program on a C64 with a Datasette (when I was about 7 years old I think, it didn't really do much!), the name is absolutely an homage to that.
lifeisstillgood 3y ago
Datasette is a open project from simon willison (a fairly well known HNer), and this looks like his monetisation project - good luck to you, hope Softbank buys you out soon :-) (It's a sort of wrapper around sqlite files so it's fairly easy to publish a file, think maybe Tableau for sqlite?) Anyway all the best
- dunk010 3y ago
  Bit cynical, no?
  - yaseer 3y ago
    I interpreted the tone as friendly, lighthearted humour, rather than cynical.
  - adg33 3y ago
    The cynical thing is to interpret something that talks about monetization as negative - it's a good thing Datasette is finding a way to make money. Great tool that fills out the SQLite ecosystem well - I always have a server running when I'm working with applications with SQLite databases.
    - dunk010 3y ago
      > hope Softbank buys you out soon :-) The cynical thing is to say this is about monetisation and an exit, rather than about making an excellent tool more easily available to journalists.
      - jakelazaroff 3y ago
        The Softbank line was definitely a joke.
  - thomascountz 3y ago
    [flagged]
    - bertman 3y ago
      No offense, but was this comment written by ChatGPT or something? Please blink if not.
      - lifeisstillgood 3y ago
loveparade 3y ago
[flagged]
skowalak 3y ago
This looks really great Simon, best of luck! I have been able to use Datasette for so much cool stuff in the last few years, I can't recommend it enough and will definitely try out datasette.cloud!
- d__a__n___g 3y ago
  [flagged]
andrewpakers 3y ago
I caught Simon on the Latent Space podcast and have spent the last few weeks going through his blogs and various YT videos. As a former journalist, I’ve been wanting to try to learn some data journalism. Maybe I’ll try it when this is available.
- swyx 3y ago
  whoa, not often that i get to introduce Simon to people, usually its the other way around. thanks for listening! he's been on our pod 3x: - https://latent.space/p/llama2 https://latent.space/p/llama2 - https://latent.space/p/code-interpreter https://latent.space/p/code-interpreter - https://latent.space/p/no-moat https://latent.space/p/no-moat which is great bc we never had to schedule him at our studio, he just dials in from home or from the park or whatever haha
- swyx 3y ago
  btw you can try it right now - thats the joy of open source - simon recently did a 2hr tutorial here https://www.youtube.com/watch?v=5TdIxxBPUSI https://www.youtube.com/watch?v=5TdIxxBPUSI
hiAndrewQuinn 3y ago
Datasette is fantastic. While working my last job I jury-rigged a way to publish Datasette internally to our Azure cloud so I could quickly share the results of other complicated SQL queries we were running. Glad to see Simon has got the dot cloud up himself.
ridruejo 3y ago
Simon is a great technologist. I have learnt quite a bit from his videos and articles. I hope this works out for him!
jamietanna 3y ago
I've been a big fan of Datasette - as Simon recently featured in https://simonwillison.net/2023/Aug/11/dependency-management-data/ https://simonwillison.net/2023/Aug/11/dependency-management-... - and think this looks really cool, both as an approach and a service. Deffo considering using it!
qwertox 3y ago
I've heard the name Datasette a couple of times but never spent time informing myself about what it can be used for. The video on the landing page does a really great job at explaining it, which doesn't happen all too often. Simon can be congratulated for it and his entire project. I really hope he has the best of success with his cloud offering.
- leetrout 3y ago
  Where is the video? I dont see it in the blog post or the home page when I click through.
  - simonw 3y ago
    The https://datasette.io/ https://datasette.io/ homepage has that video. I just edited this blog post to link to that.
    - leetrout 3y ago
      Thank you, sir!
brap 3y ago
I don't mean to be a partypooter, but I watched the full Datasette pitch video and there's something I probably just don't get. Isn't this really just a SQL GUI? Like basically any other SQL admin panel out there (minus the the writes)? What's the distinctive feature here? The extensions?
- qbasic_forever 3y ago
  I think of it more like MS Access but a sane backend of sqlite and python. There are thousands and thousands of critical business processes cludged together in Excel and Access--datasette could be a much better choice for those use cases. Something both devs and business people can use.
  - simonw 3y ago
    Yeah Access is a really interesting comparison (Datasette has quite a way to go on that front). I find it baffling that Microsoft haven't invested more in Access. The world needs a truly great desktop/mobile database solution! Excel isn't enough. Regular human beings should be able to point a full database at their own problems.
    - qbasic_forever 3y ago
      Totally agree, so many things people get strong feelings about customizing workflows--note taking, todo lists, personal document management, inventory of goods, etc.--are really just a sqlite database with some nice custom views and interfaces. I could definitely see a future where datasette or similar tools can replace some of that stuff. Access is probably caught in a weird spot internally at MS. If they put effort into it then it just removes some of the need to sell proper SQL server or azure cloud database tech. Better to just limp it along then start internal wars with bigger organizations/products.
      - simonw 3y ago
        And the great thing about those tools is that Datasette doesn't need to replace them - SQLite becomes the integration layer, so you can use any tool you like that provides a neat UI to storing data in SQLite, then use Datasette itself directly against that same database when you need to run your own SQL or integrate with other JSON apps or run custom plugins.
reaperducer 3y ago
Interesting to see how data journalism has evolved into uploading CSVs into the cloud. I remember when the hottest thing at the I.R.E.† conventions was learning how to extract and decipher the data from 9-track tapes. In the early days of FOIA, governments would try to stymie your reporting by "complying" with data requests by dumping massive amounts of information on you in giant 9-track data reels. Almost no newsroom had the equipment or technical ability to read them, so we had to figure things out by ourselves, or find friendly businesses and institutions that would help us out. † https://www.ire.org/ https://www.ire.org/
- simonw 3y ago
  I love reminding people that NICAR (National Institute for Computer-Assisted Reporting, part of IRE) was founded in the 1980s and involved working with mainframes. Data journalism is not a new thing!
pbiswal 3y ago
Has there been any interest in using Datasette for bioinformatics? I didn’t see any plugins for that space, but I could see a lot of potential for scientists to publish their datasets in an interactive form. Better-equipped or tech savvy groups do this using custom websites today, and some people upload raw data to central “depositories.” A suitably-priced offering of Datasette Cloud could open this up to many more scientists. Python already has a fantastic ecosystem of biology-related libraries (arguably R’s is better but Python is definitely a contender). One potential risk is that “omics” datasets are often much bigger than is typical for SQLite.
- simonw 3y ago
  I've heard from a couple of people who are using it for bioinformatics. It's not an area I know anything about myself but I'm excited to hear it's being applied there. How big are we talking here? My rule of thumb for SQLite and Datasette is that anything up to 1GB will Just Work. Up to 10GB works OK too but you need to start thinking a little bit about your indexes. Beyond 10GB works in theory, but you need to start throwing more hardware at the problem (mainly RAM) if you're going to get decent response times. The theoretical maximum for a single SQLite database file is 280TB - it used to be 140TB but someone out there in the world ran up against that limit and the SQLite developers doubled it for them!
  - pletnes 3y ago
    Lots of science is «big data, small but important metadata». Also «big raw data, small result data» use cases are out there. (I used to do hyperspectral stuff for a while, which lets you record tons of sensor data to get a small and neat result, think TB -> kB). So GB might not be the best or only metric, as such.
    - simonw 3y ago
      My story for Datasette and Big Data at the moment is that you can use Big Data tooling - BigQuery, Parquet, etc, but then run aggregate queries against that which produce an interesting ~10MB/~100MB/~1GB summary that you then pipe into Datasette for people to explore. I've used that trick myself a few times. Most people don't need to be able to interactively query TBs of data, they need to be able to quickly filter against a useful summary of it.
fbdab103 3y ago
Love Datasette. I have successfully deployed it internally to host many studies which were too big for sharing or consumption by normal users (100MB-20GB). Historical options have been to: distribute very high-level summary information (with a few data call outs), build up a minimal Django app, or use a much heavier weight solution (eg Metabase). Once you get into this medium data space, just distributing the data becomes a challenge as you can no longer email results around. Maybe there are limits on network shares, Sharepoint, whatever your customer is accustomed to using. Then, you run into the problem that your typical user can only use Excel which will similarly barf on too much data. Previously, I would make a few data call-outs for the most interesting results: whatever could fit in an email or a Powerpoint. Maybe an attached Excel file with the top N interesting data points. Tell the customers too reach out if they have questions or need to know anything else. Questions rarely came. Now, you can give them ~everything (usually I would only include data after some amount of processing, raw signals are not useful without extreme domain knowledge or the software to process), build up a few views to show different data highlights, and a <5 minute tutorial (“this is Super Excel, this is how you filter data”), and away they go. My first deployment, I thought it was a cute trick, that was just satisfying my nerd curiosity. However, when I checked in on the logs, I saw they were hammering the system. For a routine study, they were looking up all sorts of things. Which made me wonder: how many times in the past had they wanted more detail, but did not want to bother me? They were now empowered by data, and they could now do their own sleuthing. When I did receive questions post-Datasette, they were more sophisticated because they were able to answer the routine ones on their own.
- simonw 3y ago
  Wow, this is a fantastic success story. I really need to start gathering case studies like this! Absolutely love "this is Super Excel, this is how you filter data".
dmje 3y ago
Amazing. Simon, do you know of any museums using this (I know of your niche museum site!) - but thinking more museum collections? Would love a conversation.
- simonw 3y ago
  There are a bunch of people using it in the cultural / heritage space now, but I've not seen an official museum collection published online yet. Really looking forward to the first time that happens! Always interested in talking - swillison @ Google's mail service.
  - dmje 3y ago
    Ace, thanks, will see if there's stuff we can do with our client base - will be in touch :-)
cubes 3y ago
Are launch congratulations in order? If so, congratulations! I'm super excited to see where you take this, and I hope you're able to find a solid business model to support you and your work.
geertj 3y ago
My interest is now piqued.. How does this look on the backend? Does this store Parquet files and if so where? What's the compute model over those files (pyarrow, Spark, Trino)? Most trying to understand how far this will scale.
- simonw 3y ago
  It's using SQLite files (all of Datasette is built around SQLite at the moment) which are stored on Fly Volumes (Datasette Cloud provides a dedicated Fly Machines Firecracker container for each team account) and backed up to S3 using Litestream. The initial goal was to provide a private collaboration space, where scaling isn't as much of a challenge - at least until you get companies with thousands of employees all using it at once, though even then I would expect SQLite to be able to keep up. I've since realized that the "publishing" access of Datasette is crucially important to support. For that I have a few approaches I'm exploring: 1. Published data sits behind a Varnish cache, which should then handle huge spikes of traffic as long as it's to the same set of URLs. 2. Datasette has a great scalability story already for read-only data: you publish to something like Cloud Run or Vercel which can spin up new copies of the data on-demand to handle increased traffic. So I could let Datasette Cloud users say "publish this subset of data once every X minutes" and use that. 3. Fly are working on https://fly.io/docs/litefs/ https://fly.io/docs/litefs/ which is a perfect match for Datasette Cloud - it would allow me to run read-replicas of SQLite databases in multiple regions around the world. Part of Datasette/Datasette Cloud development is sponsored by Fly at the moment, in return for which we'll be publishing detailed notes on what we learn about building and scaling on their platform. In terms of scaling volume storage itself... the technical size limit for SQLite is 280TB, but I'm not planning on getting anywhere near that! I expect the sweet spot for Datasette Cloud will be more around the 100MB to 100GB range, probably mostly <10GB.
skybrian 3y ago
Perhaps I missed it while skimming, but one thing that’s not really explained is Datasette’s take on versioning. If you edit some rows in a table and change your mind later, is there an undo for just that change? For small amounts of data, sharing files on GitHub is a default choice and I wonder what I’d be giving up. (There is also DoltHub but it didn’t quite do what I wanted when I kicked the tires a bit.) DoltHub is explicitly trying to be a “GitHub for data” and it seems like Datasette could become that, though maybe with a different take on versions.
- simonw 3y ago
  I've been thinking about this quite a bit recently. I want to start adding features where LLMS can help with data cleanup, but for that to be useful it will need VERY robust "undo" for if they make mistakes. I wrote up one of my explorations here: sqlite-history https://simonwillison.net/2023/Apr/15/sqlite-history/ https://simonwillison.net/2023/Apr/15/sqlite-history/ I've also had a lot of success using GitHub itself for versioned data. If your data is less than a GB (and each file is under 50MB) you can dump it out to a GitHub repo and use that to track changes over time. One example of that is my personal blog, here: https://github.com/simonw/simonwillisonblog-backup/tree/main/simonwillisonblog https://github.com/simonw/simonwillisonblog-backup/tree/main... That's using this tool: https://datasette.io/tools/sqlite-diffable https://datasette.io/tools/sqlite-diffable I imagine Datasette Cloud will end up with some sort of hybrid of those approaches.
theboat 3y ago
How does datasette work with unstructured data? I work with large text datasets, and I typically have to go through hundreds of samples to evaluate a dataset's quality and determine if any cleaning or processing needs to be done. A tool that lets me sample and explore a dataset living in cloud storage, and then share it with others, would be incredibly valuable, but I haven't seen any tools that support long-form non-tabular text data well.
- simonw 3y ago
  There are a few things you can do here. SQLite is great at JSON - so I often dump JSON structures in a TEXT column and query them using https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html I also have plugins for running jq() functions directly in SQL queries - https://datasette.io/plugins/datasette-jq https://datasette.io/plugins/datasette-jq and https://github.com/simonw/sqlite-utils-jq https://github.com/simonw/sqlite-utils-jq SQLite's FTS search is surprisingly decent, and I have tools for quickly turning that on both from a CLI: https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search https://sqlite-utils.datasette.io/en/stable/cli.html#configu... and as a Datasette Plugin (available in Datasette Cloud): https://datasette.io/plugins/datasette-configure-fts https://datasette.io/plugins/datasette-configure-fts I've been trying to drive the cost of turning semi-structured data into structured SQL queries down as much as possible with https://sqlite-utils.datasette.io https://sqlite-utils.datasette.io - see this tutorial for more: https://datasette.io/tutorials/clean-data https://datasette.io/tutorials/clean-data This is also an area that I'm starting to explore with LLMs. I love the idea that you could take a bunch of messy data, tell Datasette Cloud "I want this imported into a table with this schema"... and it does that. I have a prototype of this working now, I hope to turn it into an open source plugin (and Datasette Cloud feature) pretty soon. It's using this trick: https://til.simonwillison.net/gpt3/openai-python-functions-data-extraction https://til.simonwillison.net/gpt3/openai-python-functions-d...
yevpats 3y ago
Congrats!! How does it compare to the ELT space and the modern data stack where you have ingestion/storage/visualization layers decoupled? Asking as the founder of CloudQuery (https://github.com/cloudquery/cloudquery https://github.com/cloudquery/cloudquery), Saw Datasette quite a few times around data exploration but curious to hear about the most popular use-cases of Datasette!
- simonw 3y ago
  This is a great question, and touches on one of the challenges I've been having positioning Datasette. Is Datasette an ELT tool? That's part of the ground it covers, but it's not a primary focus. Is it a visualization tool? Same problem. I worry that picking a specific vertical for it instantly limits me, in terms of how people think about the product, how pricing can work and suchlike. But the alternative is trying to define a new category entirely, which is absurdly difficult.
dgudkov 3y ago
@simonw, Datasette Cloud is a great idea! I wonder if it can benefit from additional ETL capabilities. How can I reach you? Thanks

317 points by swyx 3y ago

I thought this was a reference to the C64 C2N Datasette tape drive. Unfortunately not. https://en.m.wikipedia.org/wiki/Commodore_Datasette https://en.m.wikipedia.org/wiki/Commodore_Datasette

syslog 3y ago
I was hoping for the same. Still have the 1531 lying in the attic (next to the C16).
Findecanor 3y ago
I had been expecting something in the spirit of the "Floppy RAID" [1], but something even more ridiculous on even older technology. 1: https://youtu.be/1hc52_PWeU8 https://youtu.be/1hc52_PWeU8
- OnlyMortal 3y ago
  That’s funny. His comedy reminds me of the man who did “Roadworthy Rescues”. https://youtu.be/-A8cvrTgqGk https://youtu.be/-A8cvrTgqGk
simonw 3y ago
I wrote my first "database" program on a C64 with a Datasette (when I was about 7 years old I think, it didn't really do much!), the name is absolutely an homage to that.

lifeisstillgood 3y ago

Datasette is a open project from simon willison (a fairly well known HNer), and this looks like his monetisation project - good luck to you, hope Softbank buys you out soon :-) (It's a sort of wrapper around sqlite files so it's fairly easy to publish a file, think maybe Tableau for sqlite?) Anyway all the best

dunk010 3y ago
Bit cynical, no?
- yaseer 3y ago
  I interpreted the tone as friendly, lighthearted humour, rather than cynical.
- adg33 3y ago
  The cynical thing is to interpret something that talks about monetization as negative - it's a good thing Datasette is finding a way to make money. Great tool that fills out the SQLite ecosystem well - I always have a server running when I'm working with applications with SQLite databases.
  - dunk010 3y ago
    > hope Softbank buys you out soon :-) The cynical thing is to say this is about monetisation and an exit, rather than about making an excellent tool more easily available to journalists.
    - jakelazaroff 3y ago
      The Softbank line was definitely a joke.
- thomascountz 3y ago
  [flagged]
  - bertman 3y ago
    No offense, but was this comment written by ChatGPT or something? Please blink if not.
    - lifeisstillgood 3y ago

loveparade 3y ago

[flagged]

skowalak 3y ago

This looks really great Simon, best of luck! I have been able to use Datasette for so much cool stuff in the last few years, I can't recommend it enough and will definitely try out datasette.cloud!

d__a__n___g 3y ago
[flagged]

andrewpakers 3y ago

I caught Simon on the Latent Space podcast and have spent the last few weeks going through his blogs and various YT videos. As a former journalist, I’ve been wanting to try to learn some data journalism. Maybe I’ll try it when this is available.

swyx 3y ago
whoa, not often that i get to introduce Simon to people, usually its the other way around. thanks for listening! he's been on our pod 3x: - https://latent.space/p/llama2 https://latent.space/p/llama2 - https://latent.space/p/code-interpreter https://latent.space/p/code-interpreter - https://latent.space/p/no-moat https://latent.space/p/no-moat which is great bc we never had to schedule him at our studio, he just dials in from home or from the park or whatever haha
swyx 3y ago
btw you can try it right now - thats the joy of open source - simon recently did a 2hr tutorial here https://www.youtube.com/watch?v=5TdIxxBPUSI https://www.youtube.com/watch?v=5TdIxxBPUSI

hiAndrewQuinn 3y ago

Datasette is fantastic. While working my last job I jury-rigged a way to publish Datasette internally to our Azure cloud so I could quickly share the results of other complicated SQL queries we were running. Glad to see Simon has got the dot cloud up himself.

ridruejo 3y ago

Simon is a great technologist. I have learnt quite a bit from his videos and articles. I hope this works out for him!

jamietanna 3y ago

I've been a big fan of Datasette - as Simon recently featured in https://simonwillison.net/2023/Aug/11/dependency-management-data/ https://simonwillison.net/2023/Aug/11/dependency-management-... - and think this looks really cool, both as an approach and a service. Deffo considering using it!

qwertox 3y ago

I've heard the name Datasette a couple of times but never spent time informing myself about what it can be used for. The video on the landing page does a really great job at explaining it, which doesn't happen all too often. Simon can be congratulated for it and his entire project. I really hope he has the best of success with his cloud offering.

leetrout 3y ago
Where is the video? I dont see it in the blog post or the home page when I click through.
- simonw 3y ago
  The https://datasette.io/ https://datasette.io/ homepage has that video. I just edited this blog post to link to that.
  - leetrout 3y ago
    Thank you, sir!

brap 3y ago

I don't mean to be a partypooter, but I watched the full Datasette pitch video and there's something I probably just don't get. Isn't this really just a SQL GUI? Like basically any other SQL admin panel out there (minus the the writes)? What's the distinctive feature here? The extensions?

qbasic_forever 3y ago
I think of it more like MS Access but a sane backend of sqlite and python. There are thousands and thousands of critical business processes cludged together in Excel and Access--datasette could be a much better choice for those use cases. Something both devs and business people can use.
- simonw 3y ago
  Yeah Access is a really interesting comparison (Datasette has quite a way to go on that front). I find it baffling that Microsoft haven't invested more in Access. The world needs a truly great desktop/mobile database solution! Excel isn't enough. Regular human beings should be able to point a full database at their own problems.
  - qbasic_forever 3y ago
    Totally agree, so many things people get strong feelings about customizing workflows--note taking, todo lists, personal document management, inventory of goods, etc.--are really just a sqlite database with some nice custom views and interfaces. I could definitely see a future where datasette or similar tools can replace some of that stuff. Access is probably caught in a weird spot internally at MS. If they put effort into it then it just removes some of the need to sell proper SQL server or azure cloud database tech. Better to just limp it along then start internal wars with bigger organizations/products.
    - simonw 3y ago
      And the great thing about those tools is that Datasette doesn't need to replace them - SQLite becomes the integration layer, so you can use any tool you like that provides a neat UI to storing data in SQLite, then use Datasette itself directly against that same database when you need to run your own SQL or integrate with other JSON apps or run custom plugins.

reaperducer 3y ago

Interesting to see how data journalism has evolved into uploading CSVs into the cloud. I remember when the hottest thing at the I.R.E.† conventions was learning how to extract and decipher the data from 9-track tapes. In the early days of FOIA, governments would try to stymie your reporting by "complying" with data requests by dumping massive amounts of information on you in giant 9-track data reels. Almost no newsroom had the equipment or technical ability to read them, so we had to figure things out by ourselves, or find friendly businesses and institutions that would help us out. † https://www.ire.org/ https://www.ire.org/

simonw 3y ago
I love reminding people that NICAR (National Institute for Computer-Assisted Reporting, part of IRE) was founded in the 1980s and involved working with mainframes. Data journalism is not a new thing!

pbiswal 3y ago

Has there been any interest in using Datasette for bioinformatics? I didn’t see any plugins for that space, but I could see a lot of potential for scientists to publish their datasets in an interactive form. Better-equipped or tech savvy groups do this using custom websites today, and some people upload raw data to central “depositories.” A suitably-priced offering of Datasette Cloud could open this up to many more scientists. Python already has a fantastic ecosystem of biology-related libraries (arguably R’s is better but Python is definitely a contender). One potential risk is that “omics” datasets are often much bigger than is typical for SQLite.

simonw 3y ago
I've heard from a couple of people who are using it for bioinformatics. It's not an area I know anything about myself but I'm excited to hear it's being applied there. How big are we talking here? My rule of thumb for SQLite and Datasette is that anything up to 1GB will Just Work. Up to 10GB works OK too but you need to start thinking a little bit about your indexes. Beyond 10GB works in theory, but you need to start throwing more hardware at the problem (mainly RAM) if you're going to get decent response times. The theoretical maximum for a single SQLite database file is 280TB - it used to be 140TB but someone out there in the world ran up against that limit and the SQLite developers doubled it for them!
- pletnes 3y ago
  Lots of science is «big data, small but important metadata». Also «big raw data, small result data» use cases are out there. (I used to do hyperspectral stuff for a while, which lets you record tons of sensor data to get a small and neat result, think TB -> kB). So GB might not be the best or only metric, as such.
  - simonw 3y ago
    My story for Datasette and Big Data at the moment is that you can use Big Data tooling - BigQuery, Parquet, etc, but then run aggregate queries against that which produce an interesting ~10MB/~100MB/~1GB summary that you then pipe into Datasette for people to explore. I've used that trick myself a few times. Most people don't need to be able to interactively query TBs of data, they need to be able to quickly filter against a useful summary of it.

fbdab103 3y ago

Love Datasette. I have successfully deployed it internally to host many studies which were too big for sharing or consumption by normal users (100MB-20GB). Historical options have been to: distribute very high-level summary information (with a few data call outs), build up a minimal Django app, or use a much heavier weight solution (eg Metabase). Once you get into this medium data space, just distributing the data becomes a challenge as you can no longer email results around. Maybe there are limits on network shares, Sharepoint, whatever your customer is accustomed to using. Then, you run into the problem that your typical user can only use Excel which will similarly barf on too much data. Previously, I would make a few data call-outs for the most interesting results: whatever could fit in an email or a Powerpoint. Maybe an attached Excel file with the top N interesting data points. Tell the customers too reach out if they have questions or need to know anything else. Questions rarely came. Now, you can give them ~everything (usually I would only include data after some amount of processing, raw signals are not useful without extreme domain knowledge or the software to process), build up a few views to show different data highlights, and a <5 minute tutorial (“this is Super Excel, this is how you filter data”), and away they go. My first deployment, I thought it was a cute trick, that was just satisfying my nerd curiosity. However, when I checked in on the logs, I saw they were hammering the system. For a routine study, they were looking up all sorts of things. Which made me wonder: how many times in the past had they wanted more detail, but did not want to bother me? They were now empowered by data, and they could now do their own sleuthing. When I did receive questions post-Datasette, they were more sophisticated because they were able to answer the routine ones on their own.

simonw 3y ago
Wow, this is a fantastic success story. I really need to start gathering case studies like this! Absolutely love "this is Super Excel, this is how you filter data".

dmje 3y ago

Amazing. Simon, do you know of any museums using this (I know of your niche museum site!) - but thinking more museum collections? Would love a conversation.

simonw 3y ago
There are a bunch of people using it in the cultural / heritage space now, but I've not seen an official museum collection published online yet. Really looking forward to the first time that happens! Always interested in talking - swillison @ Google's mail service.
- dmje 3y ago
  Ace, thanks, will see if there's stuff we can do with our client base - will be in touch :-)

cubes 3y ago

Are launch congratulations in order? If so, congratulations! I'm super excited to see where you take this, and I hope you're able to find a solid business model to support you and your work.

geertj 3y ago

My interest is now piqued.. How does this look on the backend? Does this store Parquet files and if so where? What's the compute model over those files (pyarrow, Spark, Trino)? Most trying to understand how far this will scale.

simonw 3y ago
It's using SQLite files (all of Datasette is built around SQLite at the moment) which are stored on Fly Volumes (Datasette Cloud provides a dedicated Fly Machines Firecracker container for each team account) and backed up to S3 using Litestream. The initial goal was to provide a private collaboration space, where scaling isn't as much of a challenge - at least until you get companies with thousands of employees all using it at once, though even then I would expect SQLite to be able to keep up. I've since realized that the "publishing" access of Datasette is crucially important to support. For that I have a few approaches I'm exploring: 1. Published data sits behind a Varnish cache, which should then handle huge spikes of traffic as long as it's to the same set of URLs. 2. Datasette has a great scalability story already for read-only data: you publish to something like Cloud Run or Vercel which can spin up new copies of the data on-demand to handle increased traffic. So I could let Datasette Cloud users say "publish this subset of data once every X minutes" and use that. 3. Fly are working on https://fly.io/docs/litefs/ https://fly.io/docs/litefs/ which is a perfect match for Datasette Cloud - it would allow me to run read-replicas of SQLite databases in multiple regions around the world. Part of Datasette/Datasette Cloud development is sponsored by Fly at the moment, in return for which we'll be publishing detailed notes on what we learn about building and scaling on their platform. In terms of scaling volume storage itself... the technical size limit for SQLite is 280TB, but I'm not planning on getting anywhere near that! I expect the sweet spot for Datasette Cloud will be more around the 100MB to 100GB range, probably mostly <10GB.

skybrian 3y ago

Perhaps I missed it while skimming, but one thing that’s not really explained is Datasette’s take on versioning. If you edit some rows in a table and change your mind later, is there an undo for just that change? For small amounts of data, sharing files on GitHub is a default choice and I wonder what I’d be giving up. (There is also DoltHub but it didn’t quite do what I wanted when I kicked the tires a bit.) DoltHub is explicitly trying to be a “GitHub for data” and it seems like Datasette could become that, though maybe with a different take on versions.

simonw 3y ago
I've been thinking about this quite a bit recently. I want to start adding features where LLMS can help with data cleanup, but for that to be useful it will need VERY robust "undo" for if they make mistakes. I wrote up one of my explorations here: sqlite-history https://simonwillison.net/2023/Apr/15/sqlite-history/ https://simonwillison.net/2023/Apr/15/sqlite-history/ I've also had a lot of success using GitHub itself for versioned data. If your data is less than a GB (and each file is under 50MB) you can dump it out to a GitHub repo and use that to track changes over time. One example of that is my personal blog, here: https://github.com/simonw/simonwillisonblog-backup/tree/main/simonwillisonblog https://github.com/simonw/simonwillisonblog-backup/tree/main... That's using this tool: https://datasette.io/tools/sqlite-diffable https://datasette.io/tools/sqlite-diffable I imagine Datasette Cloud will end up with some sort of hybrid of those approaches.

theboat 3y ago

How does datasette work with unstructured data? I work with large text datasets, and I typically have to go through hundreds of samples to evaluate a dataset's quality and determine if any cleaning or processing needs to be done. A tool that lets me sample and explore a dataset living in cloud storage, and then share it with others, would be incredibly valuable, but I haven't seen any tools that support long-form non-tabular text data well.

simonw 3y ago
There are a few things you can do here. SQLite is great at JSON - so I often dump JSON structures in a TEXT column and query them using https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html I also have plugins for running jq() functions directly in SQL queries - https://datasette.io/plugins/datasette-jq https://datasette.io/plugins/datasette-jq and https://github.com/simonw/sqlite-utils-jq https://github.com/simonw/sqlite-utils-jq SQLite's FTS search is surprisingly decent, and I have tools for quickly turning that on both from a CLI: https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search https://sqlite-utils.datasette.io/en/stable/cli.html#configu... and as a Datasette Plugin (available in Datasette Cloud): https://datasette.io/plugins/datasette-configure-fts https://datasette.io/plugins/datasette-configure-fts I've been trying to drive the cost of turning semi-structured data into structured SQL queries down as much as possible with https://sqlite-utils.datasette.io https://sqlite-utils.datasette.io - see this tutorial for more: https://datasette.io/tutorials/clean-data https://datasette.io/tutorials/clean-data This is also an area that I'm starting to explore with LLMs. I love the idea that you could take a bunch of messy data, tell Datasette Cloud "I want this imported into a table with this schema"... and it does that. I have a prototype of this working now, I hope to turn it into an open source plugin (and Datasette Cloud feature) pretty soon. It's using this trick: https://til.simonwillison.net/gpt3/openai-python-functions-data-extraction https://til.simonwillison.net/gpt3/openai-python-functions-d...

yevpats 3y ago

Congrats!! How does it compare to the ELT space and the modern data stack where you have ingestion/storage/visualization layers decoupled? Asking as the founder of CloudQuery (https://github.com/cloudquery/cloudquery https://github.com/cloudquery/cloudquery), Saw Datasette quite a few times around data exploration but curious to hear about the most popular use-cases of Datasette!

simonw 3y ago
This is a great question, and touches on one of the challenges I've been having positioning Datasette. Is Datasette an ELT tool? That's part of the ground it covers, but it's not a primary focus. Is it a visualization tool? Same problem. I worry that picking a specific vertical for it instantly limits me, in terms of how people think about the product, how pricing can work and suchlike. But the alternative is trying to define a new category entirely, which is absurdly difficult.

dgudkov 3y ago

@simonw, Datasette Cloud is a great idea! I wonder if it can benefit from additional ETL capabilities. How can I reach you? Thanks