2 ms·
I have built my own database engine on top of a general-purpose data management system I invented. It has an application that will let you create a DB table fro
by didgetmaster 4y ago
I have built my own database engine on top of a general-purpose data management system I invented. It has an application that will let you create a DB table from a file (Json, JsonL, CSV, etc.). The application uses the Qt windowing framework that also includes a Json parser so I just used it.
It seems slower than I would like, so I wonder if your code could be used as a stand alone library for parsing the files? My current implementation is able to parse about 25K Json documents per second when each document has 20 KV pairs. I used the Chicago crime data available for download on their open data portal.
How fast will your code parse it? Here is an example of three lines:
[
{ "ID" : "10224738", "Case Number" : "HY411648", "Date" : "2015-09-05T13:30:00", "Block" : "043XX S WOOD ST", "IUCR" : "0486", "Primary Type" : "BATTERY", "Description" : "DOMESTIC BATTERY SIMPLE", "Location Description" : "RESIDENCE", "Arrest" : "false", "Domestic" : "true", "Beat" : 924, "District" : 9, "Ward" : 12, "Community Area" : 61, "FBI Code" : "08B", "X Coordinate" : "1165074", "Y Coordinate" : "1875917", "Year" : 2015, "Latitude" : 41.815117, "Longitude" : -87.670000 },
{ "ID" : "10224742", "Case Number" : "HY411435", "Date" : "2015-09-05T10:55:00", "Block" : "082XX S LOOMIS BLVD", "IUCR" : "0610", "Primary Type" : "BURGLARY", "Description" : "FORCIBLE ENTRY", "Location Description" : "RESIDENCE", "Arrest" : "false", "Domestic" : "false", "Beat" : 614, "District" : 6, "Ward" : 21, "Community Area" : 71, "FBI Code" : "05", "X Coordinate" : "1168430", "Y Coordinate" : "1850165", "Year" : 2015, "Latitude" : 41.744379, "Longitude" : -87.658431 },
{ "ID" : "10224745", "Case Number" : "HY411654", "Date" : "2015-09-05T11:30:00", "Block" : "031XX W WASHINGTON BLVD", "IUCR" : "0320", "Primary Type" : "ROBBERY", "Description" : "STRONGARM - NO WEAPON", "Location Description" : "STREET", "Arrest" : "false", "Domestic" : "true", "Beat" : 1222, "District" : 12, "Ward" : 27, "Community Area" : 27, "FBI Code" : "03", "X Coordinate" : "1155536", "Y Coordinate" : "1900515", "Year" : 2015, "Latitude" : 41.882814, "Longitude" : -87.704326 }
]
- anitil 4y agoBecause you have small files, you may not benefit as much from this extension as you'll probably be limited by OS-related issues of how fast you can open files. If you could cat them together then you might benefit from this. Also, this is for newline-deliminated json whereas yours looks like a json blob. But ultimately, it'd only take a couple of hours to test this module with the sqlite command line application and get a rough benchmark, which is more reliable than my guess. On a seperate note, I'm not sure if that data is public, but it makes me a bit uncomfortable to see lat/long on crime data on a public forum.
- didgetmaster 4y agoThe files are not that small. They can have many millions of rows in each. The example I posted was to show what each row would look like. I used it in a Json array for the example, but the file itself can be in Json Lines format. The data set blocks off the last two digits of the street address for anonymity but it doesn't look like they did the same for lat/long. If you drill down to a single block (e.g. 055XX S ROCKWELL ST), you will see 37 different latitudes and 37 different longitudes.