5 ms·

Selfspy - Tool for Personal Data Analytics

154 points by krat0sprakhar 14y ago

spindritf 14y ago
That looks absolutely amazing. You(?) even prepared packages. > In general, try to set your programs (editors, terminals, web apps, ...) to include information on what you are doing in the window title. This will make it easier to search for later. So if I'm in the terminal, Selfspy can only tell what I'm doing by the window title?
- natep 14y ago
  No, it's just for easier data retrieval with e.g. selfstats.py. It has a -T option that filters by window title, if I understand the docs correctly.
ninetax 14y ago
This looks awesome, but I wish there were some built in graphs and charts for my usage, most visited tabs, etc. Otherwise, awesome stuff!
_neil 14y ago
This is awesome. I just started building an app to track misc personal stats (a la Daytum, but with an API) and this will work great with it.
monk_the_dog 14y ago
I recently wrote a similar utility. I'll definitively take a closer look at this. I don't know if you have these features, but here's couple of features I have in mine that you might find useful: 1) I can upload a timeline into Google calendar. I do this by chunking my day into half-hour increments, and whatever I spent the most time on in that half-hour get assigned to the half-hour. (Hey, Google, please let me set the task color through the API - thanks) 2) I also sample the title window of the foreground app, but I can override this. For example, if I go running, or do the dishes, that time will be properly tracked. 3) I assign tasks to categories. For example, emacs, shell, my IDE, are all "Development". Dishes, cooking, etc are all "Chores". I'd rather track time by these larger categories. I wrote this utility a week ago, so I haven't been using it that long. But I am liking it. One more thing: Python absolutely ROCKS! This utility is less than 700 LOC and was a pleasure to write.
- rsolis 14y ago
  Any chance you're willing to share this code?
  - deleted 14y ago
    [deleted]
- deleted 14y ago
  [deleted]
momoro 14y ago
If anyone is interested in collaborating on similar stuff for a commercial project (an OS X app) let me know. I'm working on an objective-c app that does some similar things. I'm f.mischa at gmail
Matt_Cutts 14y ago
This is very cool. I've wanted something like this for a while.
- amirmc 14y ago
  Something like this, which can also string together what I do across phone, tablet and computer would be even more interesting.
  - alexatkeplar 14y ago
    I was just thinking the same thing - potentially you could mashup Selfspy with our SnowPlow (https://github.com/snowplow/snowplow https://github.com/snowplow/snowplow) project - so all your behaviour events get logged to S3 and then analysed...
- pgbovine 14y ago
  as part of my dissertation work, i built a related self-monitoring tool, except with a focus on helping scientists/researchers record and annotate their workflow: http://pgbovine.net/burrito.html http://pgbovine.net/burrito.html
dysoco 14y ago
This is amazing, I'm worried about the possible security flaws, but I don't think that could be much of a problem. I'll take a look tomorrow and see how it works.
nswanberg 14y ago
Glad someone found and posted this. I'd been using PyKeyLogger for the past week (http://sourceforge.net/projects/pykeylogger/ http://sourceforge.net/projects/pykeylogger/), but this one has built-in analytics. Hopefully someone beats me to a Windows version. Suppose I want to combine this with other personal time-series data--can anyone suggest a good tool and workflow?
proksoup 14y ago
When I first heard the title, I thought it might be an idea I had heard about for providing analytics/funnels/charts for your personal browsing history.
hnreader123 14y ago
This is an extremely interesting tool but the thought of collecting everything that I have ever typed and storing it gives me the chills, even if encrypted....
- Zenst 14y ago
  You sound like somebody who links there history file to /dev/null. In that I agree, but early days. Though the ability to have everything you typed stored in a controlled way does enable the posibility to see if anything else if collecting that same data in any way and with that opening up the possibility to identify unknown key logging. At the very least it would be another approach and opertunity to spot the bad amongst the good.
eitland 14y ago
I've been using * Karm on KDE/Linux (tracking by virtual desktop, one for each project) * Manic Time Tracker on Windows (very much like this but with a full GUI. Only works on Windows though) * For emergencies on Windows there is also Timesnapper which in facts documents in screenshots what happened (in case you have to "prove" it .) this utility seems like a real improvement on the current situation though. Thanks in advance!
- miahi 14y ago
  I'm using Timesnapper and it's great. Not only it keeps screenshots for everything, but you can also filter by application while searching for an old website you saw or data that you forgot to save. It needs a lot of storage though. In 50GB you can keep 2-3 months of screenshots.
Zenst 14y ago
This is very interesting and the first aspect what stood out to me is the possibility of such a tool being used to enable a form of ongoing user sanitation with regards to security. By that I mean opens up the possibility to collect statistics upon how a user does various tasks and qith that finger print approach enable the ability to highlight acceptions to how the user works. This could in essence provide a form of ongoing verification that the user is who they say they are. That is certainly one possibility this type of approach opens up. The ability to do ongoing user biometrics.
donniezazen 14y ago
I have tried tracking but data becomes too clumpsy to be of any use.
idm 14y ago
Here is a fork by github user ljos that includes OSX support: https://github.com/ljos/selfspy https://github.com/ljos/selfspy I am actively using it, and it seems to be pretty much like the original linux version.
- idm 14y ago
  Actually, now I've forked this myself to streamline the installation process: git clone git://github.com/iandennismiller/selfspy.git cd selfspy && make requirements && sudo make install
  - tixzdk 14y ago
    I keep getting stuck at: Traceback (most recent call last): File "/usr/bin/selfspy", line 19, in <module> from activity_store import ActivityStore File "/private/var/lib/selfspy/activity_store.py", line 9, in <module> import sniff_cocoa File "/private/var/lib/selfspy/sniff_cocoa.py", line 1, in <module> from Foundation import NSObject, NSLog ImportError: No module named Foundation
    - idm 14y ago
      I think Foundation provided by pyobjc. I suppose it's possibly you don't have a suitable version installed in site-packages, in which case you might try running the following command globally (i.e. outside the virtual environment): sudo pip install pyobjc
wazari972 14y ago
It seems a bit violent for the HDD, doesn't it? every time I scroll the mouse wheel, I can hear that selfspy writes data! Is it really safe to use for a long time?

154 points by krat0sprakhar 14y ago

That looks absolutely amazing. You(?) even prepared packages. > In general, try to set your programs (editors, terminals, web apps, ...) to include information on what you are doing in the window title. This will make it easier to search for later. So if I'm in the terminal, Selfspy can only tell what I'm doing by the window title?

natep 14y ago
No, it's just for easier data retrieval with e.g. selfstats.py. It has a -T option that filters by window title, if I understand the docs correctly.

ninetax 14y ago

This looks awesome, but I wish there were some built in graphs and charts for my usage, most visited tabs, etc. Otherwise, awesome stuff!

_neil 14y ago

This is awesome. I just started building an app to track misc personal stats (a la Daytum, but with an API) and this will work great with it.

monk_the_dog 14y ago

I recently wrote a similar utility. I'll definitively take a closer look at this. I don't know if you have these features, but here's couple of features I have in mine that you might find useful: 1) I can upload a timeline into Google calendar. I do this by chunking my day into half-hour increments, and whatever I spent the most time on in that half-hour get assigned to the half-hour. (Hey, Google, please let me set the task color through the API - thanks) 2) I also sample the title window of the foreground app, but I can override this. For example, if I go running, or do the dishes, that time will be properly tracked. 3) I assign tasks to categories. For example, emacs, shell, my IDE, are all "Development". Dishes, cooking, etc are all "Chores". I'd rather track time by these larger categories. I wrote this utility a week ago, so I haven't been using it that long. But I am liking it. One more thing: Python absolutely ROCKS! This utility is less than 700 LOC and was a pleasure to write.

rsolis 14y ago
Any chance you're willing to share this code?
- deleted 14y ago
  [deleted]
deleted 14y ago
[deleted]

momoro 14y ago

If anyone is interested in collaborating on similar stuff for a commercial project (an OS X app) let me know. I'm working on an objective-c app that does some similar things. I'm f.mischa at gmail

Matt_Cutts 14y ago

This is very cool. I've wanted something like this for a while.

amirmc 14y ago
Something like this, which can also string together what I do across phone, tablet and computer would be even more interesting.
- alexatkeplar 14y ago
  I was just thinking the same thing - potentially you could mashup Selfspy with our SnowPlow (https://github.com/snowplow/snowplow https://github.com/snowplow/snowplow) project - so all your behaviour events get logged to S3 and then analysed...
pgbovine 14y ago
as part of my dissertation work, i built a related self-monitoring tool, except with a focus on helping scientists/researchers record and annotate their workflow: http://pgbovine.net/burrito.html http://pgbovine.net/burrito.html

dysoco 14y ago

This is amazing, I'm worried about the possible security flaws, but I don't think that could be much of a problem. I'll take a look tomorrow and see how it works.

nswanberg 14y ago

Glad someone found and posted this. I'd been using PyKeyLogger for the past week (http://sourceforge.net/projects/pykeylogger/ http://sourceforge.net/projects/pykeylogger/), but this one has built-in analytics. Hopefully someone beats me to a Windows version. Suppose I want to combine this with other personal time-series data--can anyone suggest a good tool and workflow?

proksoup 14y ago

When I first heard the title, I thought it might be an idea I had heard about for providing analytics/funnels/charts for your personal browsing history.

hnreader123 14y ago

This is an extremely interesting tool but the thought of collecting everything that I have ever typed and storing it gives me the chills, even if encrypted....

Zenst 14y ago
You sound like somebody who links there history file to /dev/null. In that I agree, but early days. Though the ability to have everything you typed stored in a controlled way does enable the posibility to see if anything else if collecting that same data in any way and with that opening up the possibility to identify unknown key logging. At the very least it would be another approach and opertunity to spot the bad amongst the good.

eitland 14y ago

I've been using * Karm on KDE/Linux (tracking by virtual desktop, one for each project) * Manic Time Tracker on Windows (very much like this but with a full GUI. Only works on Windows though) * For emergencies on Windows there is also Timesnapper which in facts documents in screenshots what happened (in case you have to "prove" it .) this utility seems like a real improvement on the current situation though. Thanks in advance!

miahi 14y ago
I'm using Timesnapper and it's great. Not only it keeps screenshots for everything, but you can also filter by application while searching for an old website you saw or data that you forgot to save. It needs a lot of storage though. In 50GB you can keep 2-3 months of screenshots.

Zenst 14y ago

This is very interesting and the first aspect what stood out to me is the possibility of such a tool being used to enable a form of ongoing user sanitation with regards to security. By that I mean opens up the possibility to collect statistics upon how a user does various tasks and qith that finger print approach enable the ability to highlight acceptions to how the user works. This could in essence provide a form of ongoing verification that the user is who they say they are. That is certainly one possibility this type of approach opens up. The ability to do ongoing user biometrics.

donniezazen 14y ago

I have tried tracking but data becomes too clumpsy to be of any use.

idm 14y ago

Here is a fork by github user ljos that includes OSX support: https://github.com/ljos/selfspy https://github.com/ljos/selfspy I am actively using it, and it seems to be pretty much like the original linux version.

idm 14y ago
Actually, now I've forked this myself to streamline the installation process: git clone git://github.com/iandennismiller/selfspy.git cd selfspy && make requirements && sudo make install
- tixzdk 14y ago
  I keep getting stuck at: Traceback (most recent call last): File "/usr/bin/selfspy", line 19, in <module> from activity_store import ActivityStore File "/private/var/lib/selfspy/activity_store.py", line 9, in <module> import sniff_cocoa File "/private/var/lib/selfspy/sniff_cocoa.py", line 1, in <module> from Foundation import NSObject, NSLog ImportError: No module named Foundation
  - idm 14y ago
    I think Foundation provided by pyobjc. I suppose it's possibly you don't have a suitable version installed in site-packages, in which case you might try running the following command globally (i.e. outside the virtual environment): sudo pip install pyobjc

wazari972 14y ago

It seems a bit violent for the HDD, doesn't it? every time I scroll the mouse wheel, I can hear that selfspy writes data! Is it really safe to use for a long time?