8 ms·
Data.sparkfun.com: A place to push your data
- augbot 12y agoGreat work SparkFun! Any more info available on what other libraries that are in stages of completion...? » http://phant.io/libraries/ http://phant.io/libraries/
- brennen 12y agoThere's some simple example code in the core phant repo: https://github.com/sparkfun/phant/tree/master/examples https://github.com/sparkfun/phant/tree/master/examples And a tiny PHP wrapper here: https://github.com/sparkfun/SparkLib/blob/master/lib/SparkLib/Phant.php https://github.com/sparkfun/SparkLib/blob/master/lib/SparkLi...
- augbot 12y agoVery cool, thanks!
- bckmn 12y agoI built something similar a while ago: http://getontrack.herokuapp.com/about http://getontrack.herokuapp.com/about
- bnejad 12y agoThis is awesome. Can't wait to see a bunch of public data from random sources. Should open up some unique project opportunities.
- nacnud 12y agoXively (formerly Cosm) is a similar service: https://xively.com/dev/ https://xively.com/dev/
- Zuph 12y agoUnfortunately, Xively is neither free, nor open source.
- Frencil 12y agoXively was one of the services we used as inspiration for Phant / data.sparkfun.com. Before they were Xively they were Pachube and things were easier to work with and free. Then Xively gobbled up Pachube and things got all business-y. Thus Xively has been an example of how we didn't want to build it.
- Zuph 12y agoGlad to hear it! I've been looking for a viable Pachube alternative since Pachube got gobbled up.
- nothans 12y agoTry ThingSpeak on GitHub... Lots of active development and growing community.
- noise 12y agoVery nice and thanks! Writing a replacement for Xively/Cosm/Pachube was on my TODO list, but now I don't have to. I'll take this for a spin later, and hope to contribute if there is anything I have to offer.
- bjpirt 12y agoIt makes me very happy to hear this. I've always been sad that the initial promise of Pachube being an open place to store and exchange data was subsumed by corporate overlords. This is as close to the mindset of the original Pachube as I've seen in a long while. Nice work!
- outside1234 12y agoNitrogen is a good alternative in this space that is free and open source: http://nitrogen.io http://nitrogen.io
- chomp 12y agoLooks like some people are abusing it: https://data.sparkfun.com/streams/Jxyjr7DmxwTD5dG1D1Kv https://data.sparkfun.com/streams/Jxyjr7DmxwTD5dG1D1Kv
- 2bluesc 12y agoLooks like a few people have already hit their quota for space on seemingly legitimate streams: https://data.sparkfun.com/streams/NJKOopLR4VC6yN59jE5L https://data.sparkfun.com/streams/NJKOopLR4VC6yN59jE5L https://data.sparkfun.com/streams/pwG8Lw6jNDT5VwlnbE45 https://data.sparkfun.com/streams/pwG8Lw6jNDT5VwlnbE45
- uniontownlabs 12y agoThat's how much space is remaining: "100% (50.00 of 50 MB) remaining" The bar along the top will turn red when the stream is getting close to full.
- 2bluesc 12y agoGood call, I misread it and was shocked. Makes sense now.
- voronoff 12y agoThis is cool, but it really needs a better metadata section than just tags. Weather station data becomes a lot more interesting with it's location.
- uniontownlabs 12y agoThanks for the feedback. That's a planned feature: https://github.com/sparkfun/phant/issues/13 https://github.com/sparkfun/phant/issues/13 Right now the web UI is fairly simple, but we will start adding features soon.
- 83yg49 12y agoNotice how nowhere on that page does it actually say what the thing is. “A place to push your data” describes nothing specific.
- brennen 12y agoThis is certainly true for values of "that page" excluding everything after the title. :)
- IanCal 12y agoMaybe it was different half an hour ago, but the page has an example of how to use it and a list of projects that could use it (as well as answers to a few other questions). I'm not sure what else you'd want on a first page.
- jrullmann 12y agoOut of curiosity, what database are you using to store the data?
- uniontownlabs 12y agoBy default it writes metadata about the stream (title, description, etc) using a file based db called nedb, and it appends the actual logged data to CSV files that are split into 500k chunks. When the user requests their logged data, all of the files are stitched back together, converted into the requested format (JSON, CSV, etc), and streamed to the user’s web client. For the production server, we are currently using MongoDb for metadata storage and the same CSV module for logged data storage.
- jrullmann 12y agoThat's a pretty unusual setup :) I'd be interested in a blog post about how you choose this architecture.
- uniontownlabs 12y agoSounds good. I'll work on one once the traffic stabilizes.
- deleted 12y ago[deleted]
- ejr 12y agoIs this the same Nedb? https://github.com/louischatriot/nedb/ https://github.com/louischatriot/nedb/ Looks like a pretty nice project.
- deleted 12y ago[deleted]
- dreaken667 12y agoI'll be installing this on my own server to tinker with and will probably find the answer to my questions that way, but has anybody thought to use this to transfer binary data (i.e. zip, exe, etc.)?
- tdicola 12y agoDoes it support SSL? I ask because the examples just show normal non-SSL usage and I'd be very careful of sending the public and private key in a request as authentication without SSL. How easy is it to revoke your access keys and get new ones if someone does steal the keys?
- brennen 12y agoSSL is supported, but not mandatory because a lot of the hardware we're targeting just isn't up to the task. We're aware this isn't exactly optimal. Suggestions welcome. Re: Key revocation, that's a good question and we should probably think hard about it. It's trivial to create a new stream, but obviously that could be a pain for various reasons.
- tdicola 12y agoYou might check out stuff like Amazon's web services (DynamoDB, etc.) for inspiration. They sign requests with a hash of the request data and private access key (and date, etc.) so the private key doesn't need to be sent in the request. The data is still visible if it's over non-SSL but it keeps the key from getting out. Yeah no SSL support on Arduino-capable hardware is a bummer. I have been able to do Amazon's signature generation on an Arduino without too much trouble though (just need to run SHA256 hash algorithm a few times). I would definitely think about key revocation and renewal since folks will need it if someone steals their keys.
- simcop2387 12y agoSigning might be doable like that but it all depends on the size of the data that you're pushing. if it's a long string of readings that might not be so easy since you might not be able to fit everything in memory still (These devices routinely have <4k of ram). Might be doable to go byte by byte as you build the message and output it. It's certainly a challenging problem.
- tdicola 12y ago
- deleted 12y ago[deleted]
- wlindner 12y agoAny plans for a REST api? Would be useful to query for updates since a timestamp parameter.
- pit 12y agoThis is pretty cool -- it's like a mini-ElasticSearch (and of course I'm overstating): $ npm install -g phant $ phant $ telnet localhost 8081 phant> create Enter a title> library Enter a description> a bunch of books Enter fields (comma separated)> title,author Enter tags (comma separated)> paper,cloth Stream created! PUBLIC KEY: yrDmwO3XZEF6vMajz9oVTwPYVNM PRIVATE KEY: NYzxmKOdl8tEQ0Bn2d3WhXLbn2O DELETE KEY: WxpPb29VEkuyAemL8za2INraDPb $ curl "http://localhost:8080/input/yrDmwO3XZEF6vMajz9oVTwPYVNM.json" -H 'Content-Type: application/json' -d '{"title": "The Book of Virtues", "author": "William J. Bennett"}' -H 'Phant-Private-Key: NYzxmKOdl8tEQ0Bn2d3WhXLbn2O' $ curl "http://localhost:8080/output/yrDmwO3XZEF6vMajz9oVTwPYVNM.json" [{"author":"William J. Bennett","timestamp":"2014-07-10T18:07:13.964Z","title":"The Book of Virtues"}]
- tdicola 12y agoYou should edit out those public and private keys.
- pit 12y agoWhy? I belieb that those keys are unique to the stream (which I don't really care about).
- tdicola 12y agoSorry didn't see it was just a local instance you were running so it probably doesn't matter. In general I wouldn't let your public and private keys get out because anyone could start writing, deleting, etc. with those keys.
- mrkrwtsn 12y agoObviously.
- stronglikedan 12y agoNot obvious to everyone. HN is a place for people to learn, as well as discuss.
- xt508 12y agoThingSpeak ( https://thingspeak.com/ https://thingspeak.com/ ) is another open source data store for the Internet of Things.
- ccallebs 12y agoAny plans to allow the data to push back? Specifically, would it be possible to add webhooks after data was pushed to a stream? This could be amazing for a decentralized notification service.
- uniontownlabs 12y agoWe currently push data via websockets when you are viewing a stream, and we will probably offer live TCP output soon. Feel free to file a bug here if you would like to see webhooks added: https://github.com/sparkfun/phant/issues https://github.com/sparkfun/phant/issues
- platz 12y agocool. btw, lack of ability to side-scroll on the public views: https://data.sparkfun.com/streams/dZ4EVmE8yGCRGx5XRX1W https://data.sparkfun.com/streams/dZ4EVmE8yGCRGx5XRX1W
- ansimionescu 12y agoWorks fine on Chrome 35.0.1916.153 on OS X 10.9.4
- tokenizerrr 12y agoNo luck on Chrome 35.0.1916.153 on Windows 8. edit: Actually, using the middle mouse button and arrows work once the focus is on the right element, but no scroll bar in sight. edit edit: The scrollbar is actually at the bottom of the page, but it does behave a bit weird.
- platz 12y agoOn Chrome 38.0.2085.0 Windows, the bottom scroll bar appears briefly when the page loads, and then promptly dissapears.
- bliti 12y agoThere is also dweet[0]. I wrote a simple Python client[1] for dweet and its been fun playing with it. I like the approach taken here. Giving it a readable web GUI is a nice touch. Any future plans with push data? [0]http://dweet.io http://dweet.io [1]https://github.com/bliti/pydweet https://github.com/bliti/pydweet
- matthiasv 12y agoHere is a Python client library [1]. It's really amazing how fast I could put this together. Thanks Guido and Kenneth ;-) [1] https://github.com/matze/python-phant https://github.com/matze/python-phant
- boomlinde 12y agoInteresting! It will be fun to see what people end up using it for. I made a simple Python client library for it: https://bitbucket.org/boomlinde/pyphant https://bitbucket.org/boomlinde/pyphant
- tokenizerrr 12y agoTried the example on their front page, but it just tells me "0 rate limit exceeded. try again in 131 seconds."
- dlhavema 12y agothat's the one that anyone can just paste and use right? it seems like it must be rate limited as well..
- uniontownlabs 12y agoall feeds (including the example) are rate limited on data.sparkfun.com. Here are the details: http://phant.io/docs/input/limit http://phant.io/docs/input/limit
- noonespecial 12y agoThat's the most honest IoT effort I've seen yet. The only thing better would be a VM I could spin up that let me self host an instance of this. Edit: I know you can install phant yourself, but I'm not a node guy so I don't have an enviro just at the ready. I do use VMware player all the time thought to try stuff out. An aws image wouldn't be terrible either. Doesn't sound like a long hike to make a nice polished "IoT" backend server.
- rco8786 12y agoI bet a Docker image will pop up in the next 48 hours, if there's not 10 already.
- brennen 12y ago> The only thing better would be a VM I could spin up that let me self host an instance of this. Yeah, we'll get on that.
- fit2rule 12y agoThe one thing that would make it better would be if it posted the data to the public blockchain, so we didn't need to depend on sparkfuns' servers for retrieval purposes ..
- fiatjaf 12y agoThe page listing the public data sources makes clear no one has a thing connected to the internet yet.
- ndesaulniers 12y agoI really don't know much about crypto, but should you be sending your private key over HTTP GET requests?
- mrpoptart 12y agoI've been building a little project called guppy it's at http://gpy.me http://gpy.me and it requires less registration and is still being tested right now, but's a fun idea.
- streametry 12y agoLooks great, I've been looking for a solution like this. What would be really cool is to be able to query public data streams and combine them in way similar to Yahoo Pipes. I suspect people will also want to run aggregations/rollups of their data and that's something we enable in Streametry [1] in addition to other analytics. I might try to build a a bridge to Sparkfun. [1] http://www.streametry.com http://www.streametry.com
- listic 12y agoHow does phant compare to using Redis or RRDtool with node.js?
- dschep 12y agoAny chance of jsonp or CORS for reading the data. It'd allow for statically hosted frontends that consume a feed. Edit: nm. Added ?callback=foo and saw it's already supported!
- girishso 12y agoQuickly developed a Ruby client... PhantRb. https://github.com/girishso/phant_rb https://github.com/girishso/phant_rb