4 ms·

This is the dataset on Kaggle - https://www.kaggle.com/simiotic/github-code-snippets https://www.kaggle.com/simiotic/github-code-snippets

justinclift 6y ago
Yeah, that sized database is likely to be a challenge unless the computer system it's running on has scads of memory. One of my projects (DBHub.io) is putting effort towards working through the problem with larger sized SQLite databases (~10GB), and that's mainly through using bare metal hosts with lots of memory. eg 64GB, 128GB, etc. Putting the same data into PostgreSQL, or even MySQL, would likely be much more efficient memory wise. :)
- zomglings 6y ago
  Can't beat SQLite for distribution as a public dataset, though.
  - zachmu 6y ago
    We think dolt can :)
    - justinclift 6y ago
      How do you send someone a dolt database as a file?
      - acidbaseextract 6y ago
        How do you send someone a git repository as a file? Why would a tarball not work?
        touisteur 6y ago
        Git bundles. Amazing simple tool. Doing it all the time via sneaker net, or mail/chat-with-attachments, better that sending patches around IME.
      - 411111111111111 6y ago
        Probably by first pushing it into a file. This command is in the readme. dolt remote add <remote> file:///Users/xyz/abs/path/
      - zachmu 6y ago
        Push it to DoltHub, tell them to clone it. Just like with source code.
- zachmu 6y ago
  We have 200 GB databases in dolt format that are totally queryable. They don't work well querying on the web though - you need a local copy to query it effectively. Making web query as fast as local is an ongoing project.
  - justinclift 6y ago
    Yeah the "on the web" piece is the thing we're talking about. :) 200GB databases for PostgreSQL (etc) isn't any kind of amazing.
- tomcam 6y ago
  Uh, personal question here. Where does your ~10G number come from? I pretty much run my life on the Apple Notes app. My Notes database is about 12G and now I’m scared.
  - justinclift 6y ago
    Oh, it's just the vast majority of SQLite databases we see are pretty tiny. eg a few MB's at max So, resource usage is pretty much not a consideration in any way when doing things with them. Once people start doing things with multi-GB databases though, if the system they're running on has a small amount of memory (say 4GB, 8GB) then things can start going poorly. Not like "crash and burn" poorly (so far). More like "tries to read 10GB of data into 4GB of ram" poorly. eg Dog slow, not lightning quick. If your machine has a bunch of ram in it, you're likely safe . Though, I'd be making damn sure there are tested and working backups of it, just to be safe. :)
    - tomcam 6y ago
      I missed this answer the first time around. Thanks so much for the reply. I always get machines with the maximum amount of RAM precisely for this reason.

justinclift 6y ago

Yeah, that sized database is likely to be a challenge unless the computer system it's running on has scads of memory. One of my projects (DBHub.io) is putting effort towards working through the problem with larger sized SQLite databases (~10GB), and that's mainly through using bare metal hosts with lots of memory. eg 64GB, 128GB, etc. Putting the same data into PostgreSQL, or even MySQL, would likely be much more efficient memory wise. :)

zomglings 6y ago
Can't beat SQLite for distribution as a public dataset, though.
- zachmu 6y ago
  We think dolt can :)
  - justinclift 6y ago
    How do you send someone a dolt database as a file?
    - acidbaseextract 6y ago
      How do you send someone a git repository as a file? Why would a tarball not work?
      - touisteur 6y ago
        Git bundles. Amazing simple tool. Doing it all the time via sneaker net, or mail/chat-with-attachments, better that sending patches around IME.
    - 411111111111111 6y ago
      Probably by first pushing it into a file. This command is in the readme. dolt remote add <remote> file:///Users/xyz/abs/path/
    - zachmu 6y ago
      Push it to DoltHub, tell them to clone it. Just like with source code.
zachmu 6y ago
We have 200 GB databases in dolt format that are totally queryable. They don't work well querying on the web though - you need a local copy to query it effectively. Making web query as fast as local is an ongoing project.
- justinclift 6y ago
  Yeah the "on the web" piece is the thing we're talking about. :) 200GB databases for PostgreSQL (etc) isn't any kind of amazing.
tomcam 6y ago
Uh, personal question here. Where does your ~10G number come from? I pretty much run my life on the Apple Notes app. My Notes database is about 12G and now I’m scared.
- justinclift 6y ago
  Oh, it's just the vast majority of SQLite databases we see are pretty tiny. eg a few MB's at max So, resource usage is pretty much not a consideration in any way when doing things with them. Once people start doing things with multi-GB databases though, if the system they're running on has a small amount of memory (say 4GB, 8GB) then things can start going poorly. Not like "crash and burn" poorly (so far). More like "tries to read 10GB of data into 4GB of ram" poorly. eg Dog slow, not lightning quick. If your machine has a bunch of ram in it, you're likely safe . Though, I'd be making damn sure there are tested and working backups of it, just to be safe. :)
  - tomcam 6y ago
    I missed this answer the first time around. Thanks so much for the reply. I always get machines with the maximum amount of RAM precisely for this reason.