3 ms·
I thought about it for a while, and this is the best I could come up with: - All data is just a stream of "events" from different "sources" for any given "data
by thejash 13y ago
I thought about it for a while, and this is the best I could come up with:
- All data is just a stream of "events" from different "sources" for any given "data type"
- Sources are things like my desktop, my laptop, my phone, my camera, etc
- Data Types are things like global position, cpu usage, webcam snapshots, webpage text, etc. Anything really.
- Events are just a timestamp + zero or more other "lightly typed" "fields".
- By "lightly typed" I mean the field is either: binary data, searchable text, a float, or an integer
- Fields give the actual data for the event. If there are no fields, it's a countable event (ex: imagine recording every heartbeat, all you need is a timestamp). For something like global position, you would have latitude and longitude fields. For another datatype called "semantic location" you have just a searchable text string with things like "car", "work", "home" or whatever other labels you have. Fields can also be binary data (like photos, videos, etc). I'll probably try to make lots of data types, each with very few fields, to make it easier to work with the data (ie, it's nice if there is only one numeric field in any given data type, then it's easy to graph over time)
I'm currently collecting all of the events via some scripts that are specific to each data type, and then emailing those events (serialized as json) to my own server, which just inserts the events into a sqlite database specific to that source/data type pair.
The nice thing about this approach is that anyone can easily send an email from any language, and if you don't want to use my backend, you can just send the events to your own email server and then do whatever you want with them once they're there.
Another benefit is that you and I don't have to agree about exactly what fields and data types there are. If you call your gps data "gps" and I call mine "global position", well, who cares? It's our own personal data. If this takes off, I'll make guidelines about what to call what information and how to format it for easier interoperability.