6 ms·

Map of my personal data infrastructure

299 points by nreece 6y ago

generalizations 6y ago
The way I see it, if you need a map then it's too complicated.
- rektide 6y ago
  I am unbelievably tempted to dive into personal insult at such a repulsive, abbrasive, uncaring, unperceptive comment! How horrible! It is us who live too complicated lives. It is us who do not try to bring about consistency. It is us all who have our data scattered across endless different planes, who leave ourselves behind to wander between firewall & firewall, cloud & cloud, each time permitted to carry nothing with us, naked. Each time we begin anew, afresh, no bearings, no perspective, no timeline on ourselves. Nothing consistent. It is us who are too complicated. Karlicoss is one of the only rare few among us who are putting in the work to uncomplicate, to bring together, the endless isolated streams of data into something manageable, consistent, uncomplicated. Searching to no end to bring about simplicity, coherency, where the varied digital affords most low mortals zilch, nothing, chaos. What a disappointing sad view of things. Unwilling to try, easy to criticize. Raise your eyes up, off what you see as complexity, and try, for a second, to see what is trying to be achieved! The irony of it, that a user named generalizations, can see only the individual trees of this circumstance, and not the general forrest. This effort is exalted, more pure and uncomplicated than what any other digitally-connected mortal on earth has access to. It brings order, consistency, controllability, agency, all the exalted aspects of the amazing human spirit, out from the digital prisons it is trapped in. Exalted work, this.
  - readflaggedcomm 6y ago
    What the chart describes strikes me more as a prison than transcendence. Not all complexity is order, and not all chaos isn't simple. However, much of the wrangling could relate to contractual data preservation (like client communications), which means we don't have enough information to decide whether it's "too complex" for remuneration received. Doesn't mean the grandparent comment is wrong, though.
    - rektide 6y ago
      Having the capabilities, to me, implies nothing. What you do with gathered information is up to you. No one else has any capacity, capability, to gather, to understand, to begin to consider. We all live in eternal, scattered, not just unconnected, but unconnectable, un-understandable forms. The possibility should exist to try to harness our many forms. The digital should be a channel to explore, view, gather, from which understanding might possibly begin. But we have only ephemera, our thought & action owned inside other people's clouds.
      - readflaggedcomm 6y ago
        Data is what you make of it, and you can pile it up so high that you spend all your effort managing avalanches. Me, I like sand art.
  - deleted 6y ago
    [deleted]
  - elseweather 6y ago
    I mean, you can also just let the data go. "nothing, chaos" is pretty simple. I don't believe being able to search my chatlogs from 2005 would make my life simpler, or better. Some people are natural archivists and love that second-brain stuff though, and this is a pretty sweet example :). One area I strongly relate to Karlicoss is how bad it is getting data out of my bank; as a regular non-institutional customer I don't get API access, so I have to download a PDF manually every month and run it through some script to parse out the line items.
    - sokoloff 6y ago
      I have friends who have died and for whom I’ve lost our chat logs (some back from the AIM days). I also carelessly lost my home dir from college with my thesis and thesis code/data in it. (Tar, gzip, ftp in ascii mode, don’t check for months.) Is my life meaningfully worse for those losses? No, not really. Have there been several times when it would have been better (even just temporary happiness/comfort) if I had access to the data? Definitely yes.
    - rektide 6y ago
      Oh sure, you can opt to be an animal. You can opt to not gather your intellect, intelligence, to avoid trying to understand who you are, what you do, what you are, as you go about your time online. You can easily- we all easily- avoid reflection, understanding, coherence. Like animals. It's uncivil, the very definition of unbecoming: eternally forgetful. I don't even mind this low path! Sure! Be nothing! Remember nothing! Scatter your essence everywhere, collect & distill nothing! Fine. I don't oppose your choice to eternally disperse yourselves, to harvest & focus & observe nothing. Exist only in the now, for eternity. It's your right, as much as I think it is below you, less than what you probably ought merit yourself. What is just repulsive as heck, just dejected & pathetic, is that there are no alternatives. Except this work. Except what karlicoss is doing. The entire world lets itself be nothing, be zero, like you advocate, just sands of time. Karlicoss is focused on being a human, one with conscientiousness. Seemingly, to me, the entire rest of humanity lets all their data be trapped, contained in hostile corporate networks, under phones that don't give us access to our own data, existing on networks where we have no rights. Let it go is the only option available to nine nines, 99.9999999%. There are probably 7 people in this 7.5 billion person world doing anything at all like this, not letting it go. I'm sorry humanity, it's not your fault: you keep getting dealt a shit, ultra-consumerist crap un-personal computing hand, just getting shorter & shorter ends. But at some point, revolt, working, taking the effort, becoming a Karlicoss, opting to care, is the only sane, sensible option. I keep thinking of Google's mission statement, how much radical potential it embodied, how radically far & further from the tree we fall, every single year: "to organize the world's information and make it universally accessible and useful." Even our own information is a mess, a chaos. Our own information leaks, worse & worse, every year. We retain less & less human potential of ourselves, each year, as mainframes & cloud dominate more & more of the public space, steal up the oxygen of computing & information technology that we could be using to invigorate ourselves, orient ourselves & each other, in the newly public, ever more shared world we cohabitate upon. We have made no progress. Karlicoss has.
- mcint 6y ago
  ...Great (bait)! It seems it's not for you. The green labels show the simplicity of the structure: - export from services (a serious and shifting challenge for some or many of the services individually) - cached raw export values - imported internal/processed values - query, use, and visualization tools
- deleted 6y ago
  [deleted]
- karlicoss 6y ago
  Hi, author here. Yes, it's complicated and this is exactly one of the points of this map, to illustrate how ridiculous it is -- that I have to jump over all these hoops to do something useful with the data that's already somewhere in my phone/was in my browser, but hidden and siloed (from myself!) for various reasons.
  - generalizations 6y ago
    Fair enough. The effort and quality of your work is apparent - well done. I guess for me, I would have opted for greater inconvenience and simply let some of what you graphed slide, rather than deal with the mental overhead of actually tracking it all. But I would also not have enjoyed building that graph. :)
    - karlicoss 6y ago
      Thanks :) The thing is -- often it's very hard to know in advance how much time you would spend on something. You think "ah this would be kind of a nice feature to have/question to answer... hmmm it'll probably take only an hour", and then you spend weeks doing that, and something else related, etc. Sometimes you do wish you hadn't bothered in the first place, but then it's also sunk cost so it's tempting to finish. You know what's crazy -- I started with simply trying to improve my sleep (so exorting data from Jawbone bracelet) and wanting to be a bit more efficient at learning physics (and other things) in my spare time. I did learn some physics (and other things) thanks to all these systems, but certainly went down a massive rabbit hole too :)
- Karawebnetwork 6y ago
  Is it really that complicated? The map is pretty much straight lines from top to bottom. This is like saying that the subway system is too complicated. It is simply a series of tubes with stops. The map simply allows someone to view the entire system at once.
- MarkSweep 6y ago
  If you are doing everything by hand, perhaps. But if you are working at a higher level of abstraction using something like Apache Airflow, it will take care of making the map for you.
aarchi 6y ago
I've exported my data from 35 services, most via GDPR/CCPA exports. I've written a couple scripts to automate some, but it's not painless. I hope to eventually have a local-first unified store of my data that's easy to analyze. As a part of this, I've been working on a Go library [0] for processing local Firefox- and Chromium-based browser data, including history, bookmarks, and extension settings. [0]: https://github.com/andrewarchi/browser https://github.com/andrewarchi/browser
rektide 6y ago
big "Technology without Industry"[1] resonance! big "Computer Liberation / Dream Machines"[2] energy. we can do & become anything. we can consider ourselves, gather ourselves, understand ourselves, review ourselves, try, strive, in our imperfect ("Everything is broken, and it's ok"[3]) forms to become more. there are a lot of questions here. a lot of doubts. but few embrace the challenge. that to me is the resounding, deep silence of this era: so many great, lovely, wonderful people all about. but so few who face, head on, the moral quandry of so much of ourselves being owned by corporate data centers, dispersed, un-usable. that's not even the point, the core. rather than highlight the negative, ask: what faculties, what capabilities are gathering for ourselves? Tim Orielly said back in 2012, "create more value than you capture"[4]. few are focused, as Karlicoss is, so clearly, on creating value. on making is usable, on freeing it from it's many, tight, narrow confines. genuine computing has so few advocates. i think wistfully of the Tim's opening quote, "the skill of writing is to create a context in which other people can think" - edwin schlossberg to which tim elaborates: "a lot of what i've done is to frame things in such a way that other people can see what's important about them" and none of our digital systems enable this. they all have an egocentric perspective, a monopolistic/platform/mainframe bent, a desire to own, control, grow prowess over the (limited domain of) data that they contain within, and to not let it free. not let it connect. they do not try to open the doors of perception, they do not try to empower and enable. they offer us all a fixed, finite set of powers, and hope to grow by out-manuevering every other monopoly of thought. no big player is out there, saying, let us organize all the information however the users want to see it. no big player is out there saying, we love you, how can we set you free. the world direly needs the energetic, the hopeful, those who see it all, want to drink it all up, take in a much broader perspective of life & the world & data & online-ness & existence. the digital, alas, has sapped us of the connected hopeful, rather than enable that spirit of infinite endless creativity. mapping ourselves, gaining some mastery over who we have in fact already become, gaining insight into our digital footprints: it's such a logical basic simple first step, such an untapped starting place, that these endless digital regimes have never given us any glimpse of. no one is freer. we all have to free ourselves. [1] https://news.ycombinator.com/item?id=26268484 https://news.ycombinator.com/item?id=26268484 [2] http://worrydream.com/refs/Nelson-ComputerLibDreamMachines1975.pdf http://worrydream.com/refs/Nelson-ComputerLibDreamMachines19... [3] https://news.ycombinator.com/item?id=26265907 https://news.ycombinator.com/item?id=26265907 [4] https://www.youtube.com/watch?v=njpGH_IHjFg https://www.youtube.com/watch?v=njpGH_IHjFg
karlicoss 6y ago
Hey author here. Happy to answer your questions!
- Laminary 6y ago
  I believe my app (launching soon) is close to solving your "what do I want" but currently wrapping up the API. Are there some other communities, links to other blogs or people that are doing what you are doing? (Tracking this much data, etc.)
  - karlicoss 6y ago
    For data tracking/analysis, you're looking for "quantified self" https://github.com/woop/awesome-quantified-self#awesome-quantified-self- https://github.com/woop/awesome-quantified-self#awesome-quan... For exporting and owning the data the keyword is "data liberation" Otherwise, the keywords are perhaps "personal knowledge management" and "memex". Have some links here, although they all have different components in scope https://beepb00p.xyz/exobrain/memex.html#cmmnts https://beepb00p.xyz/exobrain/memex.html#cmmnts I usually maintain related links at the end of articles, so perhaps you can find something there, for example https://beepb00p.xyz/sad-infra.html#links https://beepb00p.xyz/sad-infra.html#links
    - deleted 6y ago
      [deleted]
    - donblanco 6y ago
      I just noticed that the awesome-quantified-self on github has been abandoned with no updates for 3-5 years now. Looking for alternatives!
      - karlicoss 6y ago
        Was it? Seems that it had some updates (maybe not as active as it used to https://github.com/woop/awesome-quantified-self/commits/master/README.MD https://github.com/woop/awesome-quantified-self/commits/mast... ) Either way don't think I've seen anything better, so the best way would be to suggest them updates or help maintaining :)
crawancon 6y ago
How much time per week is spent administering or maintaining this system? if you were to start over with a new organizational methodology, what would you do differently knowing what you know now? I applaud the effort.
- karlicoss 6y ago
  Thanks! Hmm, it's hard to quantify it exactly, but overall there isn't that much maintenance required once it's all set up (except for the few manual bits). Let's say it could all run for months without needing any intervention. Maybe one or two data sources would stop exporting due to an expired token or some API weirdness, but the downstream services that use the data will carry on running against older data. It's possible due to the design: some general 'architectural' decisions [0]: separating the data retrieval from data processing, avoiding databases where it's not absolutely necessary [1], 'defensive enough' error handling [2]. (but naturally I fix small things here and there when I notice just like with other chores) I do spend quite a lot of time on it, it's sort of a hobby at this point (e.g. to add/connect new things, experiment etc.). But, lately most time was spent documenting, releasing different parts, and writing about it to share with people -- this is much more time consuming (and certainly less fun than tinkering). I'd like to reach a certain milestone and get some rest from it, switch to something else :) > if you were to start over with a new organizational methodology, what would you do differently knowing what you know now? Frankly, I'm not really regretting doing anything the way I did it (at least for now) -- most things make sense to me and what I implemented is a 'dependency' for more sophisticated methodologies anyway, so hopefully no effort wasted here. I wish I came up to the principles I described in the links quicker, for a while it was a 'random walk' without clear vision how to compose and connect things. [0] https://beepb00p.xyz/exports.html https://beepb00p.xyz/exports.html [1] https://beepb00p.xyz/unnecessary-db.html https://beepb00p.xyz/unnecessary-db.html [2] https://beepb00p.xyz/mypy-error-handling.html https://beepb00p.xyz/mypy-error-handling.html
  - rolandog 6y ago
    Your workflow and infrastructure is amazing. Great work!
3np 6y ago
Thanks a bunch for sharing this. I think the way we move forward from here is more posts like this. It's an uphill battle where companies are actively adding friction to people being in control of their own data, but together we can do it :) ...Personally I just decided to boycot services that are smartphone-only. My last mistake was Revolut. Even the (arguably non-existing) process of getting a yearly statement from them is a goddamn joke.
- karlicoss 6y ago
  I really hope my next phone to be a real Linux phone like Pinephone/Librem (or apparently you can even install Linux on my Oneplus 6T), so I could run android-only stuff in the VM and have a real computing device in my pocket.
systemvoltage 6y ago
Hello author, hope you don't mind the negative comments since this isn't for everyone including myself. I found your dedication to organization very impressive and only shows that you'd do the same professionally.
- karlicoss 6y ago
  Hi, thanks for kind words :) I don't really see them as negative (or at least don't get offended). It seems that people are on a spectrum of organization from not even using a calendar and never taking notes ("if it's important I'll remember") to what I do (and maybe even more extreme??). And it's okay! Some people suffer from information overload, some people suffer from inefficiencies and forgetting things, and you can be productive/unproductive, or happy/unhappy either way -- there are many other factors at play too.
  - mathgladiator 6y ago
    Here are some more kind words: I appreciate you. Thank you for sharing.
  - sodality2 6y ago
    This is incredible. If you enjoy it, more power to you. I hope I can manage my digital life with the efficiency and scale you've reached.
  - johnwyles 6y ago
    So I've been waiting patiently for something equivalent to Home Assistant but for QS and you pop up often. You're probably most expert to say what progress is being made for an integrated QS platform whereby you plop in a new integration and, even if there are a few manual steps (exporting, running a local software tool, etc.) you easily can import, slice on, and chart your data in a time series DB (similar to how with InfluxDB and Grafana you can do with Home Assistant). There were a few open source tools out there on GitHub but the ones I found were dated or didn't seem to have much love or enough of an audience to support them. OH! I just saw you are the author of HPI which I have starred - it certainly was as close as I found but I pined for more evaluating it about half a year ago. How is progress? What other wise words can you lend?
borisjoffe 6y ago
I love the emphasis on data export / analysis and ownership of your data. I also love the quantified self aspects of this. However as I've gotten older, I've learned that it's incredibly freeing to at least partially let go of some of your data and not worry if it gets lost or erased. Tracking and managing all of this sucks up a lot of time and mental effort, that you could be using to progress in other areas of your life. I prefer to focus on the present instead of letting the past occupy so much of my mind. On one hand, keeping backups of important files and automating that as much as possible is good. Tracking your top 2-10 most important habits (e.g. sleep, workouts) can also be very good. But for social media sites (like Reddit, HackerNews, Twitter, YouTube, etc), it's just too much of a pain to manage. And they can kick you off in a heartbeat for any reason. I personally treat all content on those sites as potentially being erased at any time. If I have anything important (e.g. a good post, comment, or image that I want to save), I make sure to save that locally, usually in a personal outliner/wiki that I use called zim-wiki (it's like Evernote/Notion/Obsidian, but it's all just flat wiki-formatted plain text files and folders that are easily greppable and scriptable). You can also post the best content on your own site. None of this requires regular social media exports, APIs, or any programming - you can just copy/paste/save this content as you create it into your own categorized section of your filesystem or PIM tool.
- xupybd 6y ago
  Everyone needs a hobby. If this is the authors, great! It's cool and meets exactly the nerdy cool factor I look for in articles on HN. Would I invest time into doing the same, nope but then I don't think many out there would spend the same time I spend on my hobbies.
- brokenmachine 6y ago
  The problem that I have is that three years down the track I will suddenly think of a topic and remember just a few details of a really good blog post or comment thread that I would like to read again, but I can never find it. I'm just not very good at knowing in advance which is the 0.001% of posts I will want to read again. Storage is so cheap that in my ideal world, I would like to store everything I read and post. But that's difficult as you mentioned.
- 6y ago
mikelago 6y ago
Thank you for sharing this! This genre of software tool is something that has been on my mind since at least 2010. Here* is a quick summary of my idea. I even applied to the first YC Fellowship in 2015 with the idea, but I didn’t make it past the application. I’ve come across many related ideas over years of reading HN (the general consensus is to just use org mode, so far I’ve been strictly a vim user xD), but this post is particularly inspirational! I got my first job in software in 2017 and to this day I’m growing constantly. Before this first job I struggled to keep the motivation to practice and work on projects. I finally have the abilities to make projects come to life, and I’ll definitely be studying your project carefully as I get going on building one of my own. *https://web.archive.org/web/20160806063444/http://mindengei.com/about https://web.archive.org/web/20160806063444/http://mindengei.... What's "MindEngei"? Engei (園芸) is the Japanese word for gardening. I plan to build applications that help us organize our thoughts, experiences, and knowledge in a pleasant way. Too much of what we learn and experience gets sucked up by the applications we used and maintaining that information is either a hassle or impossible. I want to make applications that log everything we want to remember in a simple and relaxing way, allowing the user to pull in data from services, take notes, and document knowledge and provide quick and pleasant ways of viewing and sharing their content.
- teraku 6y ago
  I find it weird to tell people which exact tools to use. I also really enjoy knowledge management and I think I would really love org mode once I figured it out, but I just never got around to it. I am a Sublime user and really enjoy it, so I found other tools for my use cases and so far I am happy with the results. Could org mode provide me more value? Maybe. But it's not the point. Somebody might be taking more notes than others, because they are used to it, or because they need it, and it's totally okay and people should do what works for them. If your data was truly yours (data affiliation vs data ownership) this would be even less of a problem.
chrisweekly 6y ago
This is amazing! Thanks for sharing. :)
gorgoiler 6y ago
I wish more people I know published sites like this.
myrrmyrrr 6y ago
Invitation to the Sеx Club - https://oo.lc/kERN2 https://oo.lc/kERN2
z77dj3kl 6y ago
Although there is a lot of data here, it seems all of it is roughly repetitive, relational data. Why not store it in a central database? I have a (much smaller project) that stores all data in a postgres datastore.
- elseweather 6y ago
  There's an explanation here: https://beepb00p.xyz/unnecessary-db.html https://beepb00p.xyz/unnecessary-db.html (the link behind "Against unnecessary databases")
  - z77dj3kl 6y ago
    To me, it just seems like the author wants to hoard their data, not make it useful. Making it useful (the first part of which is making it queriable) involves making some kind of interface to access it, and some code to parse it. In my own case, I find that just dumping JSON docs around the place gives me an unmanageable mess of unstructured data that I never come back to. If all you want to do is to hoard your data, then I guess that achieves the purpose... But if you want to do something with it, at some point you have to make sense of it. Doesn't mean you have to go and fully normalize everything and make a production-level database!
LindaGsd97 6y ago
I’m fгеe tоnight. Wanna hаvе а drinк, сliск thе linк to find mу рrofilе with thе phonе number -> https://news.ycombinator.com@oo.lc/kERN2 https://news.ycombinator.com@oo.lc/kERN2
randyrand 6y ago
I do this too! but am way less automated than you. Id love to pay for a product that made this easier.
randyrand 6y ago
There is a way to solve all your API issues, a camera recording your entire life! Anything you interact with will be recorded. Then the problem just becomes processing it all =P Unfortunately, it's not very ergonomic yet.
- juniperplant 6y ago
  Sounds like a Black Mirror episode
- rkwasny 6y ago
  This is an awesome idea for a startup! Camera integrated with glasses that you can wear for 8 hours without charging + home box that post-processes this into useful archive. I have a Vuzix Blade (https://www.vuzix.com/products/blade-smart-glasses-upgraded https://www.vuzix.com/products/blade-smart-glasses-upgraded) that could work for MVP.
- sdevonoes 6y ago
  I'm working on something similar: A dron the size of a baseball with camera that is floating around you and records your actions (it doesn't record every action, but it can recognize certain ones: e.g., reading, running, sleeping). The recorded actions go to a sqlite db. It should be extendable in terms of recognizing actions in 3D (perhaps some ML is needed to achieve better results).
- karlicoss 6y ago
  Yep. Even simpler (and less 'creepy'), e.g. OCR against your own screens, in theory, could provide quite a lot of data for free. But not very practical yet, unfortunately?
Brajeshwar 6y ago
This is awesome. Unfortunately, I will very unlikely be able to or inclined to go this deep. However, I've been collecting both digital and physical assets that I have used to built up a museum for the family. Digital is easier, physical items are much harder. It all started way back when I left my home-town for the bustling city of Bombay as the last century was ending. When I visited my home-town after almost a decade, I realized that my tapes, floppy discs, and other artifacts including some hand-written Bollywood actors/actress mails got lost. Ever since, I've tried to hold on to interesting objects. The floppy disc was particularly interesting as it had some of the first programs I wrote, including a stupid QBasic game that I wrote for my first girlfriend.
coolgeek 6y ago
This is pretty amazing from an analytical/engineering standpoint. OTOH, it serves to justify my natural inclination to limit where and how often I participate on sites and/or particular media (e.g. chat apps). It's not just a matter of data/privacy, either. It's simply a matter of attention, available time, and the best use of that time. I went from BBSs, to Usenet News, to (mostly) niche sites on the web. The tendency of all of these to focus conversations made them (and make them) more useful to me - better uses of my time and attention - than social media. I pretty much only comment on this site, and on reddit, and only sporadically on either. I pretty much ignore almost all social media. Deleted facebook. Haven't regretted it for a moment. I understand that social media provides real value - in certain situations - for people. I just don't need it, personally.

299 points by nreece 6y ago

generalizations 6y ago

The way I see it, if you need a map then it's too complicated.

rektide 6y ago
I am unbelievably tempted to dive into personal insult at such a repulsive, abbrasive, uncaring, unperceptive comment! How horrible! It is us who live too complicated lives. It is us who do not try to bring about consistency. It is us all who have our data scattered across endless different planes, who leave ourselves behind to wander between firewall & firewall, cloud & cloud, each time permitted to carry nothing with us, naked. Each time we begin anew, afresh, no bearings, no perspective, no timeline on ourselves. Nothing consistent. It is us who are too complicated. Karlicoss is one of the only rare few among us who are putting in the work to uncomplicate, to bring together, the endless isolated streams of data into something manageable, consistent, uncomplicated. Searching to no end to bring about simplicity, coherency, where the varied digital affords most low mortals zilch, nothing, chaos. What a disappointing sad view of things. Unwilling to try, easy to criticize. Raise your eyes up, off what you see as complexity, and try, for a second, to see what is trying to be achieved! The irony of it, that a user named generalizations, can see only the individual trees of this circumstance, and not the general forrest. This effort is exalted, more pure and uncomplicated than what any other digitally-connected mortal on earth has access to. It brings order, consistency, controllability, agency, all the exalted aspects of the amazing human spirit, out from the digital prisons it is trapped in. Exalted work, this.
- readflaggedcomm 6y ago
  What the chart describes strikes me more as a prison than transcendence. Not all complexity is order, and not all chaos isn't simple. However, much of the wrangling could relate to contractual data preservation (like client communications), which means we don't have enough information to decide whether it's "too complex" for remuneration received. Doesn't mean the grandparent comment is wrong, though.
  - rektide 6y ago
    Having the capabilities, to me, implies nothing. What you do with gathered information is up to you. No one else has any capacity, capability, to gather, to understand, to begin to consider. We all live in eternal, scattered, not just unconnected, but unconnectable, un-understandable forms. The possibility should exist to try to harness our many forms. The digital should be a channel to explore, view, gather, from which understanding might possibly begin. But we have only ephemera, our thought & action owned inside other people's clouds.
    - readflaggedcomm 6y ago
      Data is what you make of it, and you can pile it up so high that you spend all your effort managing avalanches. Me, I like sand art.
- deleted 6y ago
  [deleted]
- elseweather 6y ago
  I mean, you can also just let the data go. "nothing, chaos" is pretty simple. I don't believe being able to search my chatlogs from 2005 would make my life simpler, or better. Some people are natural archivists and love that second-brain stuff though, and this is a pretty sweet example :). One area I strongly relate to Karlicoss is how bad it is getting data out of my bank; as a regular non-institutional customer I don't get API access, so I have to download a PDF manually every month and run it through some script to parse out the line items.
  - sokoloff 6y ago
    I have friends who have died and for whom I’ve lost our chat logs (some back from the AIM days). I also carelessly lost my home dir from college with my thesis and thesis code/data in it. (Tar, gzip, ftp in ascii mode, don’t check for months.) Is my life meaningfully worse for those losses? No, not really. Have there been several times when it would have been better (even just temporary happiness/comfort) if I had access to the data? Definitely yes.
  - rektide 6y ago
    Oh sure, you can opt to be an animal. You can opt to not gather your intellect, intelligence, to avoid trying to understand who you are, what you do, what you are, as you go about your time online. You can easily- we all easily- avoid reflection, understanding, coherence. Like animals. It's uncivil, the very definition of unbecoming: eternally forgetful. I don't even mind this low path! Sure! Be nothing! Remember nothing! Scatter your essence everywhere, collect & distill nothing! Fine. I don't oppose your choice to eternally disperse yourselves, to harvest & focus & observe nothing. Exist only in the now, for eternity. It's your right, as much as I think it is below you, less than what you probably ought merit yourself. What is just repulsive as heck, just dejected & pathetic, is that there are no alternatives. Except this work. Except what karlicoss is doing. The entire world lets itself be nothing, be zero, like you advocate, just sands of time. Karlicoss is focused on being a human, one with conscientiousness. Seemingly, to me, the entire rest of humanity lets all their data be trapped, contained in hostile corporate networks, under phones that don't give us access to our own data, existing on networks where we have no rights. Let it go is the only option available to nine nines, 99.9999999%. There are probably 7 people in this 7.5 billion person world doing anything at all like this, not letting it go. I'm sorry humanity, it's not your fault: you keep getting dealt a shit, ultra-consumerist crap un-personal computing hand, just getting shorter & shorter ends. But at some point, revolt, working, taking the effort, becoming a Karlicoss, opting to care, is the only sane, sensible option. I keep thinking of Google's mission statement, how much radical potential it embodied, how radically far & further from the tree we fall, every single year: "to organize the world's information and make it universally accessible and useful." Even our own information is a mess, a chaos. Our own information leaks, worse & worse, every year. We retain less & less human potential of ourselves, each year, as mainframes & cloud dominate more & more of the public space, steal up the oxygen of computing & information technology that we could be using to invigorate ourselves, orient ourselves & each other, in the newly public, ever more shared world we cohabitate upon. We have made no progress. Karlicoss has.
mcint 6y ago
...Great (bait)! It seems it's not for you. The green labels show the simplicity of the structure: - export from services (a serious and shifting challenge for some or many of the services individually) - cached raw export values - imported internal/processed values - query, use, and visualization tools
deleted 6y ago
[deleted]
karlicoss 6y ago
Hi, author here. Yes, it's complicated and this is exactly one of the points of this map, to illustrate how ridiculous it is -- that I have to jump over all these hoops to do something useful with the data that's already somewhere in my phone/was in my browser, but hidden and siloed (from myself!) for various reasons.
- generalizations 6y ago
  Fair enough. The effort and quality of your work is apparent - well done. I guess for me, I would have opted for greater inconvenience and simply let some of what you graphed slide, rather than deal with the mental overhead of actually tracking it all. But I would also not have enjoyed building that graph. :)
  - karlicoss 6y ago
    Thanks :) The thing is -- often it's very hard to know in advance how much time you would spend on something. You think "ah this would be kind of a nice feature to have/question to answer... hmmm it'll probably take only an hour", and then you spend weeks doing that, and something else related, etc. Sometimes you do wish you hadn't bothered in the first place, but then it's also sunk cost so it's tempting to finish. You know what's crazy -- I started with simply trying to improve my sleep (so exorting data from Jawbone bracelet) and wanting to be a bit more efficient at learning physics (and other things) in my spare time. I did learn some physics (and other things) thanks to all these systems, but certainly went down a massive rabbit hole too :)
Karawebnetwork 6y ago
Is it really that complicated? The map is pretty much straight lines from top to bottom. This is like saying that the subway system is too complicated. It is simply a series of tubes with stops. The map simply allows someone to view the entire system at once.
MarkSweep 6y ago
If you are doing everything by hand, perhaps. But if you are working at a higher level of abstraction using something like Apache Airflow, it will take care of making the map for you.

aarchi 6y ago

I've exported my data from 35 services, most via GDPR/CCPA exports. I've written a couple scripts to automate some, but it's not painless. I hope to eventually have a local-first unified store of my data that's easy to analyze. As a part of this, I've been working on a Go library [0] for processing local Firefox- and Chromium-based browser data, including history, bookmarks, and extension settings. [0]: https://github.com/andrewarchi/browser https://github.com/andrewarchi/browser

rektide 6y ago

big "Technology without Industry"[1] resonance! big "Computer Liberation / Dream Machines"[2] energy. we can do & become anything. we can consider ourselves, gather ourselves, understand ourselves, review ourselves, try, strive, in our imperfect ("Everything is broken, and it's ok"[3]) forms to become more. there are a lot of questions here. a lot of doubts. but few embrace the challenge. that to me is the resounding, deep silence of this era: so many great, lovely, wonderful people all about. but so few who face, head on, the moral quandry of so much of ourselves being owned by corporate data centers, dispersed, un-usable. that's not even the point, the core. rather than highlight the negative, ask: what faculties, what capabilities are gathering for ourselves? Tim Orielly said back in 2012, "create more value than you capture"[4]. few are focused, as Karlicoss is, so clearly, on creating value. on making is usable, on freeing it from it's many, tight, narrow confines. genuine computing has so few advocates. i think wistfully of the Tim's opening quote, "the skill of writing is to create a context in which other people can think" - edwin schlossberg to which tim elaborates: "a lot of what i've done is to frame things in such a way that other people can see what's important about them" and none of our digital systems enable this. they all have an egocentric perspective, a monopolistic/platform/mainframe bent, a desire to own, control, grow prowess over the (limited domain of) data that they contain within, and to not let it free. not let it connect. they do not try to open the doors of perception, they do not try to empower and enable. they offer us all a fixed, finite set of powers, and hope to grow by out-manuevering every other monopoly of thought. no big player is out there, saying, let us organize all the information however the users want to see it. no big player is out there saying, we love you, how can we set you free. the world direly needs the energetic, the hopeful, those who see it all, want to drink it all up, take in a much broader perspective of life & the world & data & online-ness & existence. the digital, alas, has sapped us of the connected hopeful, rather than enable that spirit of infinite endless creativity. mapping ourselves, gaining some mastery over who we have in fact already become, gaining insight into our digital footprints: it's such a logical basic simple first step, such an untapped starting place, that these endless digital regimes have never given us any glimpse of. no one is freer. we all have to free ourselves. [1] https://news.ycombinator.com/item?id=26268484 https://news.ycombinator.com/item?id=26268484 [2] http://worrydream.com/refs/Nelson-ComputerLibDreamMachines1975.pdf http://worrydream.com/refs/Nelson-ComputerLibDreamMachines19... [3] https://news.ycombinator.com/item?id=26265907 https://news.ycombinator.com/item?id=26265907 [4] https://www.youtube.com/watch?v=njpGH_IHjFg https://www.youtube.com/watch?v=njpGH_IHjFg

karlicoss 6y ago

Hey author here. Happy to answer your questions!

Laminary 6y ago
I believe my app (launching soon) is close to solving your "what do I want" but currently wrapping up the API. Are there some other communities, links to other blogs or people that are doing what you are doing? (Tracking this much data, etc.)
- karlicoss 6y ago
  For data tracking/analysis, you're looking for "quantified self" https://github.com/woop/awesome-quantified-self#awesome-quantified-self- https://github.com/woop/awesome-quantified-self#awesome-quan... For exporting and owning the data the keyword is "data liberation" Otherwise, the keywords are perhaps "personal knowledge management" and "memex". Have some links here, although they all have different components in scope https://beepb00p.xyz/exobrain/memex.html#cmmnts https://beepb00p.xyz/exobrain/memex.html#cmmnts I usually maintain related links at the end of articles, so perhaps you can find something there, for example https://beepb00p.xyz/sad-infra.html#links https://beepb00p.xyz/sad-infra.html#links
  - deleted 6y ago
    [deleted]
  - donblanco 6y ago
    I just noticed that the awesome-quantified-self on github has been abandoned with no updates for 3-5 years now. Looking for alternatives!
    - karlicoss 6y ago
      Was it? Seems that it had some updates (maybe not as active as it used to https://github.com/woop/awesome-quantified-self/commits/master/README.MD https://github.com/woop/awesome-quantified-self/commits/mast... ) Either way don't think I've seen anything better, so the best way would be to suggest them updates or help maintaining :)

crawancon 6y ago

How much time per week is spent administering or maintaining this system? if you were to start over with a new organizational methodology, what would you do differently knowing what you know now? I applaud the effort.

karlicoss 6y ago
Thanks! Hmm, it's hard to quantify it exactly, but overall there isn't that much maintenance required once it's all set up (except for the few manual bits). Let's say it could all run for months without needing any intervention. Maybe one or two data sources would stop exporting due to an expired token or some API weirdness, but the downstream services that use the data will carry on running against older data. It's possible due to the design: some general 'architectural' decisions [0]: separating the data retrieval from data processing, avoiding databases where it's not absolutely necessary [1], 'defensive enough' error handling [2]. (but naturally I fix small things here and there when I notice just like with other chores) I do spend quite a lot of time on it, it's sort of a hobby at this point (e.g. to add/connect new things, experiment etc.). But, lately most time was spent documenting, releasing different parts, and writing about it to share with people -- this is much more time consuming (and certainly less fun than tinkering). I'd like to reach a certain milestone and get some rest from it, switch to something else :) > if you were to start over with a new organizational methodology, what would you do differently knowing what you know now? Frankly, I'm not really regretting doing anything the way I did it (at least for now) -- most things make sense to me and what I implemented is a 'dependency' for more sophisticated methodologies anyway, so hopefully no effort wasted here. I wish I came up to the principles I described in the links quicker, for a while it was a 'random walk' without clear vision how to compose and connect things. [0] https://beepb00p.xyz/exports.html https://beepb00p.xyz/exports.html [1] https://beepb00p.xyz/unnecessary-db.html https://beepb00p.xyz/unnecessary-db.html [2] https://beepb00p.xyz/mypy-error-handling.html https://beepb00p.xyz/mypy-error-handling.html
- rolandog 6y ago
  Your workflow and infrastructure is amazing. Great work!

3np 6y ago

Thanks a bunch for sharing this. I think the way we move forward from here is more posts like this. It's an uphill battle where companies are actively adding friction to people being in control of their own data, but together we can do it :) ...Personally I just decided to boycot services that are smartphone-only. My last mistake was Revolut. Even the (arguably non-existing) process of getting a yearly statement from them is a goddamn joke.

karlicoss 6y ago
I really hope my next phone to be a real Linux phone like Pinephone/Librem (or apparently you can even install Linux on my Oneplus 6T), so I could run android-only stuff in the VM and have a real computing device in my pocket.

systemvoltage 6y ago

Hello author, hope you don't mind the negative comments since this isn't for everyone including myself. I found your dedication to organization very impressive and only shows that you'd do the same professionally.

karlicoss 6y ago
Hi, thanks for kind words :) I don't really see them as negative (or at least don't get offended). It seems that people are on a spectrum of organization from not even using a calendar and never taking notes ("if it's important I'll remember") to what I do (and maybe even more extreme??). And it's okay! Some people suffer from information overload, some people suffer from inefficiencies and forgetting things, and you can be productive/unproductive, or happy/unhappy either way -- there are many other factors at play too.
- mathgladiator 6y ago
  Here are some more kind words: I appreciate you. Thank you for sharing.
- sodality2 6y ago
  This is incredible. If you enjoy it, more power to you. I hope I can manage my digital life with the efficiency and scale you've reached.
- johnwyles 6y ago
  So I've been waiting patiently for something equivalent to Home Assistant but for QS and you pop up often. You're probably most expert to say what progress is being made for an integrated QS platform whereby you plop in a new integration and, even if there are a few manual steps (exporting, running a local software tool, etc.) you easily can import, slice on, and chart your data in a time series DB (similar to how with InfluxDB and Grafana you can do with Home Assistant). There were a few open source tools out there on GitHub but the ones I found were dated or didn't seem to have much love or enough of an audience to support them. OH! I just saw you are the author of HPI which I have starred - it certainly was as close as I found but I pined for more evaluating it about half a year ago. How is progress? What other wise words can you lend?

borisjoffe 6y ago

I love the emphasis on data export / analysis and ownership of your data. I also love the quantified self aspects of this. However as I've gotten older, I've learned that it's incredibly freeing to at least partially let go of some of your data and not worry if it gets lost or erased. Tracking and managing all of this sucks up a lot of time and mental effort, that you could be using to progress in other areas of your life. I prefer to focus on the present instead of letting the past occupy so much of my mind. On one hand, keeping backups of important files and automating that as much as possible is good. Tracking your top 2-10 most important habits (e.g. sleep, workouts) can also be very good. But for social media sites (like Reddit, HackerNews, Twitter, YouTube, etc), it's just too much of a pain to manage. And they can kick you off in a heartbeat for any reason. I personally treat all content on those sites as potentially being erased at any time. If I have anything important (e.g. a good post, comment, or image that I want to save), I make sure to save that locally, usually in a personal outliner/wiki that I use called zim-wiki (it's like Evernote/Notion/Obsidian, but it's all just flat wiki-formatted plain text files and folders that are easily greppable and scriptable). You can also post the best content on your own site. None of this requires regular social media exports, APIs, or any programming - you can just copy/paste/save this content as you create it into your own categorized section of your filesystem or PIM tool.

xupybd 6y ago
Everyone needs a hobby. If this is the authors, great! It's cool and meets exactly the nerdy cool factor I look for in articles on HN. Would I invest time into doing the same, nope but then I don't think many out there would spend the same time I spend on my hobbies.
brokenmachine 6y ago
The problem that I have is that three years down the track I will suddenly think of a topic and remember just a few details of a really good blog post or comment thread that I would like to read again, but I can never find it. I'm just not very good at knowing in advance which is the 0.001% of posts I will want to read again. Storage is so cheap that in my ideal world, I would like to store everything I read and post. But that's difficult as you mentioned.
6y ago

mikelago 6y ago

Thank you for sharing this! This genre of software tool is something that has been on my mind since at least 2010. Here* is a quick summary of my idea. I even applied to the first YC Fellowship in 2015 with the idea, but I didn’t make it past the application. I’ve come across many related ideas over years of reading HN (the general consensus is to just use org mode, so far I’ve been strictly a vim user xD), but this post is particularly inspirational! I got my first job in software in 2017 and to this day I’m growing constantly. Before this first job I struggled to keep the motivation to practice and work on projects. I finally have the abilities to make projects come to life, and I’ll definitely be studying your project carefully as I get going on building one of my own. *https://web.archive.org/web/20160806063444/http://mindengei.com/about https://web.archive.org/web/20160806063444/http://mindengei.... What's "MindEngei"? Engei (園芸) is the Japanese word for gardening. I plan to build applications that help us organize our thoughts, experiences, and knowledge in a pleasant way. Too much of what we learn and experience gets sucked up by the applications we used and maintaining that information is either a hassle or impossible. I want to make applications that log everything we want to remember in a simple and relaxing way, allowing the user to pull in data from services, take notes, and document knowledge and provide quick and pleasant ways of viewing and sharing their content.

teraku 6y ago
I find it weird to tell people which exact tools to use. I also really enjoy knowledge management and I think I would really love org mode once I figured it out, but I just never got around to it. I am a Sublime user and really enjoy it, so I found other tools for my use cases and so far I am happy with the results. Could org mode provide me more value? Maybe. But it's not the point. Somebody might be taking more notes than others, because they are used to it, or because they need it, and it's totally okay and people should do what works for them. If your data was truly yours (data affiliation vs data ownership) this would be even less of a problem.

chrisweekly 6y ago

This is amazing! Thanks for sharing. :)

gorgoiler 6y ago

I wish more people I know published sites like this.

myrrmyrrr 6y ago

Invitation to the Sеx Club - https://oo.lc/kERN2 https://oo.lc/kERN2

z77dj3kl 6y ago

Although there is a lot of data here, it seems all of it is roughly repetitive, relational data. Why not store it in a central database? I have a (much smaller project) that stores all data in a postgres datastore.

elseweather 6y ago
There's an explanation here: https://beepb00p.xyz/unnecessary-db.html https://beepb00p.xyz/unnecessary-db.html (the link behind "Against unnecessary databases")
- z77dj3kl 6y ago
  To me, it just seems like the author wants to hoard their data, not make it useful. Making it useful (the first part of which is making it queriable) involves making some kind of interface to access it, and some code to parse it. In my own case, I find that just dumping JSON docs around the place gives me an unmanageable mess of unstructured data that I never come back to. If all you want to do is to hoard your data, then I guess that achieves the purpose... But if you want to do something with it, at some point you have to make sense of it. Doesn't mean you have to go and fully normalize everything and make a production-level database!

LindaGsd97 6y ago

I’m fгеe tоnight. Wanna hаvе а drinк, сliск thе linк to find mу рrofilе with thе phonе number -> https://news.ycombinator.com@oo.lc/kERN2 https://news.ycombinator.com@oo.lc/kERN2

randyrand 6y ago

I do this too! but am way less automated than you. Id love to pay for a product that made this easier.

randyrand 6y ago

There is a way to solve all your API issues, a camera recording your entire life! Anything you interact with will be recorded. Then the problem just becomes processing it all =P Unfortunately, it's not very ergonomic yet.

juniperplant 6y ago
Sounds like a Black Mirror episode
rkwasny 6y ago
This is an awesome idea for a startup! Camera integrated with glasses that you can wear for 8 hours without charging + home box that post-processes this into useful archive. I have a Vuzix Blade (https://www.vuzix.com/products/blade-smart-glasses-upgraded https://www.vuzix.com/products/blade-smart-glasses-upgraded) that could work for MVP.
sdevonoes 6y ago
I'm working on something similar: A dron the size of a baseball with camera that is floating around you and records your actions (it doesn't record every action, but it can recognize certain ones: e.g., reading, running, sleeping). The recorded actions go to a sqlite db. It should be extendable in terms of recognizing actions in 3D (perhaps some ML is needed to achieve better results).
karlicoss 6y ago
Yep. Even simpler (and less 'creepy'), e.g. OCR against your own screens, in theory, could provide quite a lot of data for free. But not very practical yet, unfortunately?

Brajeshwar 6y ago

This is awesome. Unfortunately, I will very unlikely be able to or inclined to go this deep. However, I've been collecting both digital and physical assets that I have used to built up a museum for the family. Digital is easier, physical items are much harder. It all started way back when I left my home-town for the bustling city of Bombay as the last century was ending. When I visited my home-town after almost a decade, I realized that my tapes, floppy discs, and other artifacts including some hand-written Bollywood actors/actress mails got lost. Ever since, I've tried to hold on to interesting objects. The floppy disc was particularly interesting as it had some of the first programs I wrote, including a stupid QBasic game that I wrote for my first girlfriend.

coolgeek 6y ago

This is pretty amazing from an analytical/engineering standpoint. OTOH, it serves to justify my natural inclination to limit where and how often I participate on sites and/or particular media (e.g. chat apps). It's not just a matter of data/privacy, either. It's simply a matter of attention, available time, and the best use of that time. I went from BBSs, to Usenet News, to (mostly) niche sites on the web. The tendency of all of these to focus conversations made them (and make them) more useful to me - better uses of my time and attention - than social media. I pretty much only comment on this site, and on reddit, and only sporadically on either. I pretty much ignore almost all social media. Deleted facebook. Haven't regretted it for a moment. I understand that social media provides real value - in certain situations - for people. I just don't need it, personally.