4 ms·

Looks like ~7,100 in the dataset; its definitely not "Big Data", but its definitely sizable. The USDA .xls file is 16 megabytes.

by yahelc 15y ago

cousin_it 15y ago
Here's a single datapoint: { "type": "Feature", "id": "b52fc9c8940ff3885830dc3de305e5ac", "geometry": { "type":"Point", "coordinates":[-122.393,37.7955] }, "properties": { "_id":"b52fc9c8940ff3885830dc3de305e5ac", "_rev":"1-1b07544c7620ff6af55550eb2e3ed66a", "Jams":"1", "WIC":"1", "Vegetables":"1", "Street":"One Ferry Building, Suite 50", "Credit":"1", "Prepared":"1", "Soap":"0", "Nuts":"1", "MarketName":"Ferry Plaza Farmers' Market", "lng":"-122.393365", "Cheese":"1", "SFMNP":"1", "WICcash":"0", "Zip":"94111", "Seafood":"1", "Fruit":"1", "Herbs":"1", "Honey":"1", "lat":"37.7955195", "Description": "The Ferry Plaza Farmers' Market is located at \ the foot of Market Street and on the Embarcadero \ at the historic San Francisco Ferry Building.", "Flowers":"1", "updateTime":"4/14/11 12:02", "Website":"http://www.cuesa.org", "Bakedgoods":"1", "full_address": "One Ferry Building, Suite 50, San Francisco, California 94111", "Location":"City/ township public building grounds", "City":"San Francisco", "Crafts":"0", "Meat":"1", "Maple":"0", "County":"San Francisco", "State":"California", "SNAP":"1", "Plants":"1", "y":"37.7955", "x":"-122.393" } } About 1 kilobyte of text. Throw away the GeoJSON boilerplate and the long IDs, compress all the 0/1 fields into a single int (all entries have the same set of them), remove 2 of the 3 redundant copies of coordinates, basically strip away the cruft and you can make it 2x-3x smaller. So optimistically that would be about 3 megs of JSON. And the user can throw away unneeded data even further. I think the usability advantages of having the whole file available locally (compared to an API) outweigh the disadvantages here.

Looks like ~7,100 in the dataset; its definitely not "Big Data", but its definitely sizable. The USDA .xls file is 16 megabytes.

by yahelc 15y ago

Looks like ~7,100 in the dataset; its definitely not "Big Data", but its definitely sizable. The USDA .xls file is 16 megabytes.

cousin_it 15y ago

Here's a single datapoint: { "type": "Feature", "id": "b52fc9c8940ff3885830dc3de305e5ac", "geometry": { "type":"Point", "coordinates":[-122.393,37.7955] }, "properties": { "_id":"b52fc9c8940ff3885830dc3de305e5ac", "_rev":"1-1b07544c7620ff6af55550eb2e3ed66a", "Jams":"1", "WIC":"1", "Vegetables":"1", "Street":"One Ferry Building, Suite 50", "Credit":"1", "Prepared":"1", "Soap":"0", "Nuts":"1", "MarketName":"Ferry Plaza Farmers' Market", "lng":"-122.393365", "Cheese":"1", "SFMNP":"1", "WICcash":"0", "Zip":"94111", "Seafood":"1", "Fruit":"1", "Herbs":"1", "Honey":"1", "lat":"37.7955195", "Description": "The Ferry Plaza Farmers' Market is located at \ the foot of Market Street and on the Embarcadero \ at the historic San Francisco Ferry Building.", "Flowers":"1", "updateTime":"4/14/11 12:02", "Website":"http://www.cuesa.org", "Bakedgoods":"1", "full_address": "One Ferry Building, Suite 50, San Francisco, California 94111", "Location":"City/ township public building grounds", "City":"San Francisco", "Crafts":"0", "Meat":"1", "Maple":"0", "County":"San Francisco", "State":"California", "SNAP":"1", "Plants":"1", "y":"37.7955", "x":"-122.393" } } About 1 kilobyte of text. Throw away the GeoJSON boilerplate and the long IDs, compress all the 0/1 fields into a single int (all entries have the same set of them), remove 2 of the 3 redundant copies of coordinates, basically strip away the cruft and you can make it 2x-3x smaller. So optimistically that would be about 3 megs of JSON. And the user can throw away unneeded data even further. I think the usability advantages of having the whole file available locally (compared to an API) outweigh the disadvantages here.