3 ms·

Why an API? The dataset is pretty small. Converting it from Excel to JSON and putting it up for download as one file would be more useful for any task I can thi

Why an API? The dataset is pretty small. Converting it from Excel to JSON and putting it up for download as one file would be more useful for any task I can think of.

yahelc 15y ago
Looks like ~7,100 in the dataset; its definitely not "Big Data", but its definitely sizable. The USDA .xls file is 16 megabytes.
- cousin_it 15y ago
  Here's a single datapoint: { "type": "Feature", "id": "b52fc9c8940ff3885830dc3de305e5ac", "geometry": { "type":"Point", "coordinates":[-122.393,37.7955] }, "properties": { "_id":"b52fc9c8940ff3885830dc3de305e5ac", "_rev":"1-1b07544c7620ff6af55550eb2e3ed66a", "Jams":"1", "WIC":"1", "Vegetables":"1", "Street":"One Ferry Building, Suite 50", "Credit":"1", "Prepared":"1", "Soap":"0", "Nuts":"1", "MarketName":"Ferry Plaza Farmers' Market", "lng":"-122.393365", "Cheese":"1", "SFMNP":"1", "WICcash":"0", "Zip":"94111", "Seafood":"1", "Fruit":"1", "Herbs":"1", "Honey":"1", "lat":"37.7955195", "Description": "The Ferry Plaza Farmers' Market is located at \ the foot of Market Street and on the Embarcadero \ at the historic San Francisco Ferry Building.", "Flowers":"1", "updateTime":"4/14/11 12:02", "Website":"http://www.cuesa.org", "Bakedgoods":"1", "full_address": "One Ferry Building, Suite 50, San Francisco, California 94111", "Location":"City/ township public building grounds", "City":"San Francisco", "Crafts":"0", "Meat":"1", "Maple":"0", "County":"San Francisco", "State":"California", "SNAP":"1", "Plants":"1", "y":"37.7955", "x":"-122.393" } } About 1 kilobyte of text. Throw away the GeoJSON boilerplate and the long IDs, compress all the 0/1 fields into a single int (all entries have the same set of them), remove 2 of the 3 redundant copies of coordinates, basically strip away the cruft and you can make it 2x-3x smaller. So optimistically that would be about 3 megs of JSON. And the user can throw away unneeded data even further. I think the usability advantages of having the whole file available locally (compared to an API) outweigh the disadvantages here.
imperialWicket 15y ago
JSON (slightly scrubbed): https://github.com/imperialwicket/usdaFarmersMarkets https://github.com/imperialwicket/usdaFarmersMarkets

Why an API? The dataset is pretty small. Converting it from Excel to JSON and putting it up for download as one file would be more useful for any task I can think of.

yahelc 15y ago

Looks like ~7,100 in the dataset; its definitely not "Big Data", but its definitely sizable. The USDA .xls file is 16 megabytes.

cousin_it 15y ago
Here's a single datapoint: { "type": "Feature", "id": "b52fc9c8940ff3885830dc3de305e5ac", "geometry": { "type":"Point", "coordinates":[-122.393,37.7955] }, "properties": { "_id":"b52fc9c8940ff3885830dc3de305e5ac", "_rev":"1-1b07544c7620ff6af55550eb2e3ed66a", "Jams":"1", "WIC":"1", "Vegetables":"1", "Street":"One Ferry Building, Suite 50", "Credit":"1", "Prepared":"1", "Soap":"0", "Nuts":"1", "MarketName":"Ferry Plaza Farmers' Market", "lng":"-122.393365", "Cheese":"1", "SFMNP":"1", "WICcash":"0", "Zip":"94111", "Seafood":"1", "Fruit":"1", "Herbs":"1", "Honey":"1", "lat":"37.7955195", "Description": "The Ferry Plaza Farmers' Market is located at \ the foot of Market Street and on the Embarcadero \ at the historic San Francisco Ferry Building.", "Flowers":"1", "updateTime":"4/14/11 12:02", "Website":"http://www.cuesa.org", "Bakedgoods":"1", "full_address": "One Ferry Building, Suite 50, San Francisco, California 94111", "Location":"City/ township public building grounds", "City":"San Francisco", "Crafts":"0", "Meat":"1", "Maple":"0", "County":"San Francisco", "State":"California", "SNAP":"1", "Plants":"1", "y":"37.7955", "x":"-122.393" } } About 1 kilobyte of text. Throw away the GeoJSON boilerplate and the long IDs, compress all the 0/1 fields into a single int (all entries have the same set of them), remove 2 of the 3 redundant copies of coordinates, basically strip away the cruft and you can make it 2x-3x smaller. So optimistically that would be about 3 megs of JSON. And the user can throw away unneeded data even further. I think the usability advantages of having the whole file available locally (compared to an API) outweigh the disadvantages here.

imperialWicket 15y ago

JSON (slightly scrubbed): https://github.com/imperialwicket/usdaFarmersMarkets https://github.com/imperialwicket/usdaFarmersMarkets