11 ms·

Show HN: Interface for all digital aspects of my life

478 points by karlicoss 6y ago

ybbond 6y ago
I just noticed about "own your data" movement ever since I know about IndieWeb. Your work is a great example for that and I like how you display them
srat17 6y ago
Cool! Offtopic: how did u create the blog? Did you use WordPress? If so, which theme? If not, what UI library did you use? I ask because I have a blog myself which looks like crap haha.
- TkTech 6y ago
  https://github.com/karlicoss/beepb00p https://github.com/karlicoss/beepb00p
- diffeomorphism 6y ago
  A right click, view source mentions hakyll https://jaspervdj.be/hakyll/ https://jaspervdj.be/hakyll/ Probably in combination with org-mode http://wikemacs.org/wiki/Blogging http://wikemacs.org/wiki/Blogging
  - karlicoss 6y ago
    Yep, most sources are org-mode, there are few ipython notebooks too. Hakyll in HTML sources is actually just an artifact left in my template, from the times when I did use Hakyll. Should probably remove it. I found it a bit overkill for my purposes and often overly restricting me, so I've switched to a Python script to generate everything: https://github.com/karlicoss/beepb00p/blob/master/src/build.py https://github.com/karlicoss/beepb00p/blob/master/src/build.... It's a bit ad-hoc, but ended up the same length as my old Hakyll code, and allows me to experiment much faster.
random_kris 6y ago
that looks awesome. I think this area is ready for a great new application. I am not sure how it could be monetized or what it would even do.
vbsteven 6y ago
Cool project. I've been toying with and thinking about similar ideas lately but more focused on my productivity as a freelance software dev/ops person. My approach is basically to view myself as a distributed system that receives various inputs (jira tickets, PR comments, slack messages, emails) and performs work that produces outputs (PRs, emails, PR reviews, deploys). I'm trying to model workflows for all types of work I do, and write plugins that process various inputs and outputs so tasks can enter and progress through these workflows.
ajflores1604 6y ago
This is incredible. I've been slowly mapping out a similar idea, and gaining the programming chops to build it, so I thoroughly appreciate all the work this has to have taken to make real. Heres an older post I made with thoughts on this topic of centralizing data and what could be done with it: https://news.ycombinator.com/item?id=21673846 https://news.ycombinator.com/item?id=21673846
alexmingoia 6y ago
What’s the coolest thing or most useful thing you’ve been able to do as a result of having all this information?
- karlicoss 6y ago
  I list my most common usecases here [0], but if I had to name just one thing, it would probably be promnesia [1]. It's an extension that I'm working on, with the purpose to actually use all this information and integrate it in my browser, like a remembrance agent. [0] https://beepb00p.xyz/hpi.html#usecases https://beepb00p.xyz/hpi.html#usecases [1] https://github.com/karlicoss/promnesia#demos https://github.com/karlicoss/promnesia#demos
rl3 6y ago
Fantastic job with this! Feels like I've stumbled upon a [vastly more productive] clone of myself. The best way I can describe this type of thinking is a very strong, overriding compulsion to think and live in an augmented, external, hivemind-like capacity. It's beautiful, but it certainly isn't normal. It makes one wonder what the etiology of such thinking is. Were I to guess, based on personal experience: hypomania, ADHD, obsessive tendencies, indignation at tech platforms sucking so bad, technology fracturing our fucking minds. Any combination thereof. >I'm not willing to wait till some vaporwave project reinvents the whole computing model from scratch Sorry, turns out it's kind of hard! :)
- metalliqaz 6y ago
  >Feels like I've stumbled upon a [vastly more productive] clone of myself. yep, same
amelius 6y ago
If only we had well-maintained open source projects for scraping every data type.
- slightwinder 6y ago
  Maybe something in the spirit of Home Assistant? But for general service-interaction, not only home automation. That would be so useful, as also kinda dangerous. It would be also a damn amount of work, just for managing it. Maybe building some general framework and API and let people build it decentral?
- the_duke 6y ago
  Yes please. I've been working on something quite similar to this project (in Rust), and I had the same need. I started tinkering with a library + DSL that describes data fetching and transformation, where all data is transformed into schema.org types. The DSL can then be either interpreted or client code generated for various languages. This would potentially make such a API collection library much more feasible. Sadly such a DSL turns out to be more of a full programming language if all requirements are covered.
samanator 6y ago
This has been on my development todo list for a year! What are your thoughts about allowing others to contribute new modules? I'm interested in a my.spotify and a my.film.traktv
- karlicoss 6y ago
  I don't mind merging new modules at the moment, so would be happy if you try it out and contribute! But it's also possible to use your module without merging in the upstream HPI package [0] In the long run, I'm not sure how sustainable it is, since there are many different data sources and ways of representing them. This is also particularly difficult when you don't use the service, e.g. as a maintaner I wouldn't have any means of testing Traktv. So far I've kept all HPI modules close, because a monolith is easier for prototyping and refactoring. But my fear is the fate of oh-my-zsh, or spacemacs, which are overwhelmed by the pull requests. Ideally I think it should be a simple core, only containing the common utility functions, extraction helpers, error handling, caching, logging, that sort of thing; and make the rest third-party. It's possible to achieve this in Python, thanks to the namespace packages. The only problem I see is managing these small individual packages, and declaring dependencies between them. This is possible with PIP and setuptools, but there is certain overhead involved, I feel like this step is ought to be simpler, especially to people who don't want to fully dive into Python. [0] https://github.com/karlicoss/HPI/blob/master/doc/DEVELOPMENT.org#modifyingadding-modules https://github.com/karlicoss/HPI/blob/master/doc/DEVELOPMENT...
  - the_duke 6y ago
    > since there are many different data sources and ways of representing them I've been working on something quite similar for a while (in Rust). I decided to normalize data to schema.org types where available and store JSON-LD documents in a key/value store. While the schema.org types are far from perfect, it at least morphs data into a standardized format.
  - simonw 6y ago
    Have you looked at Pluggy much? It's the Python plugin system that was spun out of pytest. I'm really impressed by it - it's a very clean design, and integrates great with Python packaging. I'm using it extensively for Datasette, which means myself or others can add new features to the core software without needing to ask permission and in a way which supports trying out crackpot new ideas without sullying the design of the core software. I've been thinking about usi it for my Dogsheep personal analytics suite too, which is currently split up into a bunch of separate tools in separate repos (since the only unifying interface is that they all spit out SQLite databases).
jasode 6y ago
>HPI is a Python package (named my), a collection of modules for: - social networks: posts, comments, favorites - reading: e-books and pdfs - annotations: highlights and comments - todos and notes - health data: sleep, exercise, weight, heart rate, and other body metrics - location - photos & videos - browser history - instant messaging I've been studying the concept of "life management software" for decades so thanks for sharing your project and I enjoyed reading your thought process. The concepts I always think of that the idealized software would manage: + timeline: [past] --> [present] --> [future] + The Big 2: time & money + reflection vs growth : mining the past data for patterns and metrics -- vs -- managing and prioritizing a future wishlist to deliberately design a future life I've come to the conclusion that it's very hard to come up with a good schema that unifies all aspects of life that's important to me. For example, looking at your list above, it is heavily populated by digital artifacts (things that happened in the past). One of the exceptions in your list that is "future" oriented would be "todos" and possibly "notes". I'm also interested in life "situational awareness". E.g. the other aspect of life data is money which means financial budgeting/planning. Another aspect of life data is time so a digital calendar of future events -- and goals -- is essential. Yes, I keep a list of books I've read (the "quantified self") -- but also books I plan to read in the future (designing a future version of myself a.k.a personal "growth"). There's nothing wrong with your project. I'm just explaining my observations after seeing various attempts at this over the decades. This includes 1980s software like Borland Sidekick, 1990s PIMs (Personal Information Managers) like Lotus Agenda, late 1990s PDA (Personal Digital Assistants) like Palm Pilot, 2000s software like Evernote, and a hundreds of 2010s SaaS "todo/calendar/notes" websites, or uber-geek tools like Emacs Org-mode. None of the above really do what I want so for now, I just split my digital life across various tools and files. My daily notes in "journal.txt". My financial planning (and bank & credit card data downloads) in "budget.xls". Saved webpages in ".mhtml" files. Planned programming projects in "projects.xls". Etc.
- karlicoss 6y ago
  Thanks! Yep, totally agree it's hard to incorporate the future, especially when it's in free-form like org-mode notes. Past data is much easier becuase it's at least somewhat structured and easy for the computer. I guess my take on this is that subjective metrics, like the one people use to define 'success', 'growth', etc., are orders of magnitude harder to grasp than objective metrics (e.g. hard data, like sleep or excercise logs). Yet, we don't even have good means of incorporating the hard data in our lives, so I chose to start with an easier problem. Yeah you're right, my future planning is all in my org-mode notes, I reflect on it with my brain, but so far not sure how I can use software (apart from organizer software) to aid me with it. It would certainly be an interesting area to explore.
nicbou 6y ago
I have been toying with a similar idea for a very long time. I want to have an all-encompassing personal history for the same reasons I want good access and error logs. I want to see when something wrong happens, and I want the means to diagnose the problem and fix the damage. A few good examples: - Did I forget to write down any tax-deductible expenses? - What is my net worth? Is my current lifestyle sustainable? - Which parts of my city have I yet to explore? - Can I search all my conversation with John across 3 messaging apps, email, SMS and phone? I took a few jabs at these problems. Invariably, the services I consumed data from shut down their API or hid my data better. In some cases, I wanted to use a different service, but didn't feel like updating my scripts. I'm currently looking into this again, but this time it's also a matter of privacy. I want to own all that data I create. I'd rather integrate tools under my control than services who actively try to hold my data hostage.
- chrisma0 6y ago
  > Which parts of my city have I yet to explore? > Can I search all my conversation with John across 3 messaging apps, email, SMS and phone? These are totally pain points for me / questions I'd love to have some answers for... Anyone tried to get the historic iOS location data to answer the first question?
  - eitland 6y ago
    If you live in Europe it should be possible..?
    - nicbou 6y ago
      The GDPR says you can get access to your data. It doesn't say it should be available in a convenient, computer-friendly format. Google Takeout gives me my location history, but getting it is a manual process. Banks must offer an API. I just haven't looked into it yet.
      - eitland 6y ago
        Bank APIs are kind of usable it seems, at least I can now see most of my accounts across all banks I ever used from my main bank.
pheme1 6y ago
I wish to see some kind of encryption and token access for each module. A quick glance of the setup.py, it seems to rely on appdirs to store user data unencrypted?
- karlicoss 6y ago
  appdirs is just a module to find the config directory in a portable way, the data can be stored anywhere you point the config to, but you're right that there is no encryption involved. Note that it's meant to be running on your own computer, and it's using the filesystem to access the data, no network interactions involved. Ultimately it's about how you're protecting the data on your disk and whether you trust youself with it. Of course, my code has to be trusted too, but it's at least possible to run it in a sandbox/use Docker, etc. The modules run untrusted Python code, so if you keep your token on the disk, they can potentially steal it token too, unless you use some elaborate authentication system.
GistNoesis 6y ago
Is there a myDigitalLifeInAGlimpse function that would generate a full synthetic report, ready to be shared on social networks ? If you can add some social scoring, it would be great too. Maybe if you have a way of certifying it too, it would be great indicator to put on resumes to show how much better you are than those other candidates. What about the next step ? Have you considered just putting screen recording (+ body cam if you want to extend it to all life monitoring and not just digital) and store all the footage. Then it's just some video indexing. You can always post-process the data as new deep learning algorithms become available. This way you don't have to mess up with specific api for each service. You handle the general case and you are done. (That's what smart people do https://writings.stephenwolfram.com/2019/02/seeking-the-productive-life-some-details-of-my-personal-infrastructure/ https://writings.stephenwolfram.com/2019/02/seeking-the-prod... ) Have you thought about plugging an optimizer into it so we could make the most of our lives ? I have got an Hedonic treadmill to climb. /s
- karlicoss 6y ago
  Hey, I've got a crude timeline of events (simply rendered as giant HTMLs), but it's private at the moment. I'll probably share some of it in the future, especially considering that some of the data is public anyway (e.g. online comments or tweets), it makes sense. Next step: yep, I throught about it a bit [0], agree that it would be great as a service-agnostic way of accessing the data, the same way we can see it with our eyes. But figured I should start with strictly easier problems for now, ones that I can solve at least to some extent. For now my goal is simply enjoying using it, making the setup simpler (to make it accessible to more people), making sure it's robust to the changes, flexible enough, etc. I want to inspire people to own their data. In many ways, I'm advocating an approach to owning it and working with it, rather than a specific set of tools, formats, etc. It would be great to have more people experimenting in the same area and trying to loosely integrate with existing efforts. Optimizer: to some extents, yes! Not sure about 'general happiness', but I'd happily let the computer dictate me what to eat, how to exercise and how to sleep. Maintenance is probably one of the most annoying things in my life. [0] https://beepb00p.xyz/pkm-search.html#future https://beepb00p.xyz/pkm-search.html#future
  - 6y ago
Communitivity 6y ago
Kudos - this looks like a great project and is fully usable now. I have a similar project, DL, that's unfinished. Mine revolves around using a custom API in both Rust and REST to aggregate all my digital life events using ActivityStreams 2.0 and extensions to that, in a manner that is decentralized and ranked/categorized through machine learning. I am still working on it and releasing it Open Source is one of this year's goals. My motivation is that the amount of information I receive from Twitter,Mastodon,Facebook,Reddit,HN,various Stack exchanges,blog postings,etc. has gotten to the point where it's too easy to miss things. Jeremie Miller, one of the creators of XMPP, had something similar revolving around the Telehash protocol. As far as I can tell, that effort is discontinued, or at least no longer Open Source.
- 21stio 6y ago
  Hey, we are building something similar. Please checkout https://metamate.io/blog/most_advanced_hackernews_api https://metamate.io/blog/most_advanced_hackernews_api. For now, we concentrate on reading public data from various social sources. We just released a HackerNews service and going to add more over the next couple of weeks. Here's a little application build on top of a MetaMate http://showcase.metamate.io/hackernews-activity http://showcase.metamate.io/hackernews-activity. Please let me know what you think :)
chrisma0 6y ago
The linked post and video (including a demo) of Andrew Louis's 'Memex' project is really neat as well! Really cool interface there: https://hyfen.net/memex/ https://hyfen.net/memex/
- hyfen 6y ago
  Thanks! Had a nice chat with karlicoss earlier this week to compare notes.
- nikisweeting 6y ago
  Memex is really awesome, I'm excited to see more of these solutions being developed with different approaches.
wakkaflokka 6y ago
Like a lot of other posters, I’ve been working on something like this off and on but never near to completion. Thank you for taking it all the way!
AJRF 6y ago
Polar opposite - Does anyone enjoy digital amnesia? I feel like I might be a little weird here, but every so often I'll just go in and delete all my notes, reminders, apps, photos, etc and start again. Wondering if anyone else does this? I like doing it because it helps clear my head, even though the files or data are digital they take up some space in my head. I do it every few months, just have a fresh slate. I wonder if that says something about me - I'm not really sure. It's probably just another form of procrastination. I once read Stephen Wolframs post about how he has a key logger that backs up every keystroke he types and its all indexed and he has a little front end over his enormous amount of data and it made me feel anxious just even reading it.
- graphpapa 6y ago
  I do something halfway in between - i routinely get frustrated with everything that’s not relevant to my present concerns, and just throw it into a folder called ‘archive’ and then move THAT into the big central archive folder. I never really look back through any of it but when I do it’s quite nostalgic.
  - AJRF 6y ago
    I should probably start doing this for the (very) few times over the years I've regretted deleting something.
  - wasureru 6y ago
    I do the same thing too. I’m not sure if I’ll ever look back through any of it but it’s nice to know that if for whatever reason I feel the need to review old data, my future self will be relieved that it’s not gone into the digital ether.
- brian_herman 6y ago
  I love deleting my accounts every year, It feels so liberating and cathartic to just click that button and have my information go to /dev/null.
- pjc50 6y ago
  I think I'm with the sibling commentators: deleting something when I could keep it feels like an obliteration, a destruction, a waste; on the other hand, realistically I don't need it and am almost certainly never going to look at it again. Almost. But if I did and wanted to and couldn't, I'd probably feel loss. So I chuck it in an archive. Gradually expanding storage at low prices means never having to think about deletion. I don't use any of the systems for sharing tabs or history across browsers, because each has a different set of almost abandoned tabs. Sometimes "tab bankruptcy" is forced on me by software failure, and it's frustrating.
Neph 6y ago
This is awesome, I'm working on an personal, self hosted, IoT System from scratch where I have several fetch agents in huggin to get data and store it in "The Archive" (DataStorage Server) Than I have different API's to that my IoT devices call to fetch the data from The Archive to trigger events and generate dashboards. This might be a great way to optimize it and save my some development time for things that I don't have yet implemented. (Twitter archive, messages, etc) Some examples of things already implemented: - Raspberry Pi Touchscreen on my desk with dashboards -> Calendar -> Financial Dashboard (Net Worth, Stocks, Expenses) -> Reminders (ongoing to do list, train delays etc) -> Dashboard with my indoor air quality/temp and plant sensor - Self made night light + alarm clock -> Calculates time I need to go to bed to get 8 hours of sleep and indicates it with a red light. -> In the morning triggers alarm and reads highlight, light shows weather forecast
- 21stio 6y ago
  Sounds interested. Do you have a link to share?
ozim 6y ago
One day you wake up and realize that you spent your life maintaining your system to improve your life instead of actually improving your life.
- keiferski 6y ago
  https://en.wikipedia.org/wiki/Map–territory_relation https://en.wikipedia.org/wiki/Map–territory_relation
  - notJim 6y ago
    Indeed. Today abstraction is no longer that of the map, the double, the mirror, or the concept. Simulation is no longer that of a territory, a referential being or substance. It is the generation by models of a real without origin or reality: A hyperreal[1]. The territory no longer precedes the map, nor does it survive it. *It is nevertheless the map that precedes the territory— precession of simulacra—that engenders the territory.* [1] https://en.wikipedia.org/wiki/Hyperreality https://en.wikipedia.org/wiki/Hyperreality
- jl6 6y ago
  Arnold Rimmer’s revision timetable.
- themodelplumber 6y ago
  Starting on a project to improve one's life by engaging latent gifts is one of the smartest things one can do. It is aligned with the best advice psychology can offer. If that means maintaining a measurement system works for you--great. It's not as if we lack data showing that measurement systems are beneficial to human development.
- twicetwice 6y ago
  Only if you think of "maintaining your system to improve your life" as work. I would think of it more as play. It's not time wasted if you enjoy it. Hm. I actually don't agree with that sentiment as written. It's not time wasted if you enjoy it and have something to show for it? That's not it either. Maybe this. I view the work we're discussing like a programmer's equivalent of woodworking in your garage. Sure, it's not necessary. But it's creative, fulfilling, and enjoyable. There's something immensely satisfying about using something you made with your own two hands, even if it's not perfect. As long as all these things remain true, I wouldn't call it time wasted. To me your critism reads something like, "if you wanted to canoe, why did you spend a year making your own? you could have bought one and spent that year doing what you really wanted to do!" Well, I wanted do that, but I also enjoyed making something to do that too. I feel my life is better for having made it.
op00to 6y ago
I just need a simple way to make sure I am doing the things that are important to me and make my life better.
A4ET8a8uTh0 6y ago
I am of two minds about it. First off, there is something deeply satisfying about seeing your stuff properly categorized and easily accessible when needed so I definitely understand the impulse. And the result looks like it does exactly that. Kudos. There is a part of me that dislikes this level of insight someone can have into my life and habits ( yes, it is intended for owner use, but the data is there, neatly organized for someone to access ). It is a weird comment for me, because I absolutely see the benefit of this project.
- karlicoss 6y ago
  I can see your concern! I guess I've got several thoughts on it: - One is along the lines of post-privacy: Lots of the stuff that I'm collecting one can find online anyway if they deliberately search for it. Tweets/reddit comments/instagram photos, etc. Some of it isn't public, let's say, Amazon purchases. But would I mind sharing it? Maybe not, who cares what I buy? Some ML algorithm that would show me ads? Whatever, I don't click on them anyway. The more stuff I don't mind to be public in the first place, the less is the chance someone can use it against me. But I understand it's ultimately something very personal. - Second is that I agree that the security professionals would handle the data much better than me. I don't even mind Google keeping my data, if only it was easier for me to access it when I need it! So generally I would happily pay professionals for a service to keep my data safe, being able to access it and integrate together, and for the infrastructure around it. Same way I'm using a bank to keep my money safe. But such infrastructure for data simply doesn't exist at the moment, because there is no demand for it from people. And silos only benefit the companies that keep this data, so the change isn't going to be come from them. E.g. if you can easily access your tweets through a nice and fast local app, why would you even visit bloated twitter.com, which is trying to 'entertain' and 'engage' you? I'm hoping that with projects like this, I can inspire people to think differently about their services and tools, and demand (from professionals, i.e. engineers/designers/product managers) for better means of using their own data.
- YayamiOmate 6y ago
  > There is a part of me that dislikes this level of insight someone can have into my life and habits ( yes, it is intended for owner use, but the data is there, neatly organized for someone to access ). Well this is already happening. Google and facebook for example earns basically all of their money from this. They replaced advertisment middle men because they can track and quantize human behaviours and identities. They do it because they are able to aggregate the data and provide API for marketing people. This is a tool that brings it closer to you as the creator of the data. it doesn't do anything new just exposes what's going on. If you are able to open pages through EU check out data processing partners from GDPR consent you have to accept. Lists on every professionally built services are long. There are literally thousands of companies accessing, aggregating, processing and reselling the data and their analysis. The tools you are afraid of already exist and are commercially available.
simonw 6y ago
My version of this is called Dogsheep, and involves building tools that suck my data from various sources into SQLite databases so I can query it all using Datasette. https://dogsheep.github.io/ https://dogsheep.github.io/ I enjoyed reading this author's essay on why he doesn't think loading everything into relational databases is the best approach: https://beepb00p.xyz/unnecessary-db.html https://beepb00p.xyz/unnecessary-db.html I think I have a workarounds for most of the issues he describes there. One example: I mostly create my base table schema automatically based on the shape of the incoming JSON, and automatically add new columns if that JSON has keys I haven't seen before. Very thought-provoking writing (and software) here.
- karlicoss 6y ago
  Ooooh, I totally forgot to add your link, I remember you commenting, sorry! Will amend now. Ah I see, so your approach is using databases as an 'intermediate' layer, rather than the main source of 'truth'. My main objection to databases is the fricton with the maintenance and the speed of prototyping, but if it's automatic and used as a cache, I agree it's reasonable. Although I'd still be worried about normalising something wrong, but this is my personal sort of anxiety :) I've got something similar [0]: if you have a sequence of NamedTuples/dataclasses, you can get a database 'for free'. [0] https://beepb00p.xyz/unnecessary-db.html#cachew_cachew https://beepb00p.xyz/unnecessary-db.html#cachew_cachew
- mholt 6y ago
  Cool, that's similar to my project Timeliner: https://github.com/mholt/timeliner https://github.com/mholt/timeliner It downloads your data from social media (or other online sites) into a single sqlite database. It can even draw relations between various accounts/people. I use it to back up my Google Photos.
  - mathnmusic 6y ago
    There's also heedy: https://heedy.org/ https://heedy.org/
    - 6y ago
edraferi 6y ago
Very cool! Intake [0] is another package that might help here. It organizes a set of data sources into (1) plugins that actually connect to the data source and map the data to standard Python data structures like Data Frames (2) catalogs that reference the plugins you want to use alongside project specific metadata like usernames/passwords/source URIs (3) convenience functions that persistence, concatenation, etc (4) a GUI for browsing data sources Looks like OP has already written a lot of data access logic... I might fork them into Intake plugins for my personal use. [0] https://github.com/intake/intake/blob/master/README.md https://github.com/intake/intake/blob/master/README.md
- karlicoss 6y ago
  Wow, looks interesting, thanks for the link. Please share if you integrate it!
chrisweekly 6y ago
Fantastic! Thank you for this. Agreed on most of your points, and stoked to give your scripts a try soon. :)
jackklika 6y ago
> What it isn't? > It's not vaporwave I look forward to the day we replace vaporware with vaporwave, ala segue>segway
nico_h 6y ago
Reminds me of the Feltron report[1], in the self-quantization realm. The person did it for 10 years and stopped in 2014. The accent was put on the visualization. http://feltron.com/FAR14.html http://feltron.com/FAR14.html
andrey_utkin 6y ago
This is amazing. I am putting as much as i can into my TODO+archive database (see https://github.com/andrey-utkin/taskdb/wiki/Live-demo#workout-dashboard https://github.com/andrey-utkin/taskdb/wiki/Live-demo#workou... ), and it is pretty neat already for analysis with querying and visualization. But your stuff is orders of magnitude bigger. Possibly I will set up HPI for myself some day.
- karlicoss 6y ago
  Looks cool! HPI is more about accessing data, so it should be possible to combine the projects. I was thinking of using Grafana too, and adding an automatic influxdb integration (via https://github.com/karlicoss/cachew https://github.com/karlicoss/cachew, which I'm already using for sqlite), so let me know if you set up something! :)
adrienchauvet 6y ago
I think you should check this project form Tim Berners Lee: https://solid.mit.edu/ https://solid.mit.edu/
- karlicoss 6y ago
  Hey, thanks. I'm listing it as once of the links, and I tried using it, but I kind of struggled to see if it can integrate with the existing infrastructure.
doelie_ 6y ago
Cool project. I have been on the lookout for something like this. The idea of exocortex seems to be something that many people want, and almost universially re-invent because the way in which structuring makes sense is different for everyone. I like the approach of making it modular so people can pick and choose parts and do their own integration. What I learned from thinking about this problem is that this kind of integration is really hard and almost impossible to tackle without an evolutionary approach, and it made me go meta and focus on incremental development of the integration layer first. My interpretation of the idea of exocortex might be a bit different in that the focus is not so much about information per se, but more about systems that interact with the physical world. Here's a writeup in case anyone is interested in where it took me: https://github.com/zwizwa/erl_tools/blob/master/doc/exo.org https://github.com/zwizwa/erl_tools/blob/master/doc/exo.org

478 points by karlicoss 6y ago

I just noticed about "own your data" movement ever since I know about IndieWeb. Your work is a great example for that and I like how you display them

srat17 6y ago

Cool! Offtopic: how did u create the blog? Did you use WordPress? If so, which theme? If not, what UI library did you use? I ask because I have a blog myself which looks like crap haha.

TkTech 6y ago
https://github.com/karlicoss/beepb00p https://github.com/karlicoss/beepb00p
diffeomorphism 6y ago
A right click, view source mentions hakyll https://jaspervdj.be/hakyll/ https://jaspervdj.be/hakyll/ Probably in combination with org-mode http://wikemacs.org/wiki/Blogging http://wikemacs.org/wiki/Blogging
- karlicoss 6y ago
  Yep, most sources are org-mode, there are few ipython notebooks too. Hakyll in HTML sources is actually just an artifact left in my template, from the times when I did use Hakyll. Should probably remove it. I found it a bit overkill for my purposes and often overly restricting me, so I've switched to a Python script to generate everything: https://github.com/karlicoss/beepb00p/blob/master/src/build.py https://github.com/karlicoss/beepb00p/blob/master/src/build.... It's a bit ad-hoc, but ended up the same length as my old Hakyll code, and allows me to experiment much faster.

random_kris 6y ago

that looks awesome. I think this area is ready for a great new application. I am not sure how it could be monetized or what it would even do.

vbsteven 6y ago

Cool project. I've been toying with and thinking about similar ideas lately but more focused on my productivity as a freelance software dev/ops person. My approach is basically to view myself as a distributed system that receives various inputs (jira tickets, PR comments, slack messages, emails) and performs work that produces outputs (PRs, emails, PR reviews, deploys). I'm trying to model workflows for all types of work I do, and write plugins that process various inputs and outputs so tasks can enter and progress through these workflows.

ajflores1604 6y ago

This is incredible. I've been slowly mapping out a similar idea, and gaining the programming chops to build it, so I thoroughly appreciate all the work this has to have taken to make real. Heres an older post I made with thoughts on this topic of centralizing data and what could be done with it: https://news.ycombinator.com/item?id=21673846 https://news.ycombinator.com/item?id=21673846

alexmingoia 6y ago

What’s the coolest thing or most useful thing you’ve been able to do as a result of having all this information?

karlicoss 6y ago
I list my most common usecases here [0], but if I had to name just one thing, it would probably be promnesia [1]. It's an extension that I'm working on, with the purpose to actually use all this information and integrate it in my browser, like a remembrance agent. [0] https://beepb00p.xyz/hpi.html#usecases https://beepb00p.xyz/hpi.html#usecases [1] https://github.com/karlicoss/promnesia#demos https://github.com/karlicoss/promnesia#demos

rl3 6y ago

Fantastic job with this! Feels like I've stumbled upon a [vastly more productive] clone of myself. The best way I can describe this type of thinking is a very strong, overriding compulsion to think and live in an augmented, external, hivemind-like capacity. It's beautiful, but it certainly isn't normal. It makes one wonder what the etiology of such thinking is. Were I to guess, based on personal experience: hypomania, ADHD, obsessive tendencies, indignation at tech platforms sucking so bad, technology fracturing our fucking minds. Any combination thereof. >I'm not willing to wait till some vaporwave project reinvents the whole computing model from scratch Sorry, turns out it's kind of hard! :)

metalliqaz 6y ago
>Feels like I've stumbled upon a [vastly more productive] clone of myself. yep, same

amelius 6y ago

If only we had well-maintained open source projects for scraping every data type.

slightwinder 6y ago
Maybe something in the spirit of Home Assistant? But for general service-interaction, not only home automation. That would be so useful, as also kinda dangerous. It would be also a damn amount of work, just for managing it. Maybe building some general framework and API and let people build it decentral?
the_duke 6y ago
Yes please. I've been working on something quite similar to this project (in Rust), and I had the same need. I started tinkering with a library + DSL that describes data fetching and transformation, where all data is transformed into schema.org types. The DSL can then be either interpreted or client code generated for various languages. This would potentially make such a API collection library much more feasible. Sadly such a DSL turns out to be more of a full programming language if all requirements are covered.

samanator 6y ago

This has been on my development todo list for a year! What are your thoughts about allowing others to contribute new modules? I'm interested in a my.spotify and a my.film.traktv

karlicoss 6y ago
I don't mind merging new modules at the moment, so would be happy if you try it out and contribute! But it's also possible to use your module without merging in the upstream HPI package [0] In the long run, I'm not sure how sustainable it is, since there are many different data sources and ways of representing them. This is also particularly difficult when you don't use the service, e.g. as a maintaner I wouldn't have any means of testing Traktv. So far I've kept all HPI modules close, because a monolith is easier for prototyping and refactoring. But my fear is the fate of oh-my-zsh, or spacemacs, which are overwhelmed by the pull requests. Ideally I think it should be a simple core, only containing the common utility functions, extraction helpers, error handling, caching, logging, that sort of thing; and make the rest third-party. It's possible to achieve this in Python, thanks to the namespace packages. The only problem I see is managing these small individual packages, and declaring dependencies between them. This is possible with PIP and setuptools, but there is certain overhead involved, I feel like this step is ought to be simpler, especially to people who don't want to fully dive into Python. [0] https://github.com/karlicoss/HPI/blob/master/doc/DEVELOPMENT.org#modifyingadding-modules https://github.com/karlicoss/HPI/blob/master/doc/DEVELOPMENT...
- the_duke 6y ago
  > since there are many different data sources and ways of representing them I've been working on something quite similar for a while (in Rust). I decided to normalize data to schema.org types where available and store JSON-LD documents in a key/value store. While the schema.org types are far from perfect, it at least morphs data into a standardized format.
- simonw 6y ago
  Have you looked at Pluggy much? It's the Python plugin system that was spun out of pytest. I'm really impressed by it - it's a very clean design, and integrates great with Python packaging. I'm using it extensively for Datasette, which means myself or others can add new features to the core software without needing to ask permission and in a way which supports trying out crackpot new ideas without sullying the design of the core software. I've been thinking about usi it for my Dogsheep personal analytics suite too, which is currently split up into a bunch of separate tools in separate repos (since the only unifying interface is that they all spit out SQLite databases).

jasode 6y ago

>HPI is a Python package (named my), a collection of modules for: - social networks: posts, comments, favorites - reading: e-books and pdfs - annotations: highlights and comments - todos and notes - health data: sleep, exercise, weight, heart rate, and other body metrics - location - photos & videos - browser history - instant messaging I've been studying the concept of "life management software" for decades so thanks for sharing your project and I enjoyed reading your thought process. The concepts I always think of that the idealized software would manage: + timeline: [past] --> [present] --> [future] + The Big 2: time & money + reflection vs growth : mining the past data for patterns and metrics -- vs -- managing and prioritizing a future wishlist to deliberately design a future life I've come to the conclusion that it's very hard to come up with a good schema that unifies all aspects of life that's important to me. For example, looking at your list above, it is heavily populated by digital artifacts (things that happened in the past). One of the exceptions in your list that is "future" oriented would be "todos" and possibly "notes". I'm also interested in life "situational awareness". E.g. the other aspect of life data is money which means financial budgeting/planning. Another aspect of life data is time so a digital calendar of future events -- and goals -- is essential. Yes, I keep a list of books I've read (the "quantified self") -- but also books I plan to read in the future (designing a future version of myself a.k.a personal "growth"). There's nothing wrong with your project. I'm just explaining my observations after seeing various attempts at this over the decades. This includes 1980s software like Borland Sidekick, 1990s PIMs (Personal Information Managers) like Lotus Agenda, late 1990s PDA (Personal Digital Assistants) like Palm Pilot, 2000s software like Evernote, and a hundreds of 2010s SaaS "todo/calendar/notes" websites, or uber-geek tools like Emacs Org-mode. None of the above really do what I want so for now, I just split my digital life across various tools and files. My daily notes in "journal.txt". My financial planning (and bank & credit card data downloads) in "budget.xls". Saved webpages in ".mhtml" files. Planned programming projects in "projects.xls". Etc.

karlicoss 6y ago
Thanks! Yep, totally agree it's hard to incorporate the future, especially when it's in free-form like org-mode notes. Past data is much easier becuase it's at least somewhat structured and easy for the computer. I guess my take on this is that subjective metrics, like the one people use to define 'success', 'growth', etc., are orders of magnitude harder to grasp than objective metrics (e.g. hard data, like sleep or excercise logs). Yet, we don't even have good means of incorporating the hard data in our lives, so I chose to start with an easier problem. Yeah you're right, my future planning is all in my org-mode notes, I reflect on it with my brain, but so far not sure how I can use software (apart from organizer software) to aid me with it. It would certainly be an interesting area to explore.

nicbou 6y ago

I have been toying with a similar idea for a very long time. I want to have an all-encompassing personal history for the same reasons I want good access and error logs. I want to see when something wrong happens, and I want the means to diagnose the problem and fix the damage. A few good examples: - Did I forget to write down any tax-deductible expenses? - What is my net worth? Is my current lifestyle sustainable? - Which parts of my city have I yet to explore? - Can I search all my conversation with John across 3 messaging apps, email, SMS and phone? I took a few jabs at these problems. Invariably, the services I consumed data from shut down their API or hid my data better. In some cases, I wanted to use a different service, but didn't feel like updating my scripts. I'm currently looking into this again, but this time it's also a matter of privacy. I want to own all that data I create. I'd rather integrate tools under my control than services who actively try to hold my data hostage.

chrisma0 6y ago
> Which parts of my city have I yet to explore? > Can I search all my conversation with John across 3 messaging apps, email, SMS and phone? These are totally pain points for me / questions I'd love to have some answers for... Anyone tried to get the historic iOS location data to answer the first question?
- eitland 6y ago
  If you live in Europe it should be possible..?
  - nicbou 6y ago
    The GDPR says you can get access to your data. It doesn't say it should be available in a convenient, computer-friendly format. Google Takeout gives me my location history, but getting it is a manual process. Banks must offer an API. I just haven't looked into it yet.
    - eitland 6y ago
      Bank APIs are kind of usable it seems, at least I can now see most of my accounts across all banks I ever used from my main bank.

pheme1 6y ago

I wish to see some kind of encryption and token access for each module. A quick glance of the setup.py, it seems to rely on appdirs to store user data unencrypted?

karlicoss 6y ago
appdirs is just a module to find the config directory in a portable way, the data can be stored anywhere you point the config to, but you're right that there is no encryption involved. Note that it's meant to be running on your own computer, and it's using the filesystem to access the data, no network interactions involved. Ultimately it's about how you're protecting the data on your disk and whether you trust youself with it. Of course, my code has to be trusted too, but it's at least possible to run it in a sandbox/use Docker, etc. The modules run untrusted Python code, so if you keep your token on the disk, they can potentially steal it token too, unless you use some elaborate authentication system.

GistNoesis 6y ago

Is there a myDigitalLifeInAGlimpse function that would generate a full synthetic report, ready to be shared on social networks ? If you can add some social scoring, it would be great too. Maybe if you have a way of certifying it too, it would be great indicator to put on resumes to show how much better you are than those other candidates. What about the next step ? Have you considered just putting screen recording (+ body cam if you want to extend it to all life monitoring and not just digital) and store all the footage. Then it's just some video indexing. You can always post-process the data as new deep learning algorithms become available. This way you don't have to mess up with specific api for each service. You handle the general case and you are done. (That's what smart people do https://writings.stephenwolfram.com/2019/02/seeking-the-productive-life-some-details-of-my-personal-infrastructure/ https://writings.stephenwolfram.com/2019/02/seeking-the-prod... ) Have you thought about plugging an optimizer into it so we could make the most of our lives ? I have got an Hedonic treadmill to climb. /s

karlicoss 6y ago
Hey, I've got a crude timeline of events (simply rendered as giant HTMLs), but it's private at the moment. I'll probably share some of it in the future, especially considering that some of the data is public anyway (e.g. online comments or tweets), it makes sense. Next step: yep, I throught about it a bit [0], agree that it would be great as a service-agnostic way of accessing the data, the same way we can see it with our eyes. But figured I should start with strictly easier problems for now, ones that I can solve at least to some extent. For now my goal is simply enjoying using it, making the setup simpler (to make it accessible to more people), making sure it's robust to the changes, flexible enough, etc. I want to inspire people to own their data. In many ways, I'm advocating an approach to owning it and working with it, rather than a specific set of tools, formats, etc. It would be great to have more people experimenting in the same area and trying to loosely integrate with existing efforts. Optimizer: to some extents, yes! Not sure about 'general happiness', but I'd happily let the computer dictate me what to eat, how to exercise and how to sleep. Maintenance is probably one of the most annoying things in my life. [0] https://beepb00p.xyz/pkm-search.html#future https://beepb00p.xyz/pkm-search.html#future
- 6y ago

Communitivity 6y ago

Kudos - this looks like a great project and is fully usable now. I have a similar project, DL, that's unfinished. Mine revolves around using a custom API in both Rust and REST to aggregate all my digital life events using ActivityStreams 2.0 and extensions to that, in a manner that is decentralized and ranked/categorized through machine learning. I am still working on it and releasing it Open Source is one of this year's goals. My motivation is that the amount of information I receive from Twitter,Mastodon,Facebook,Reddit,HN,various Stack exchanges,blog postings,etc. has gotten to the point where it's too easy to miss things. Jeremie Miller, one of the creators of XMPP, had something similar revolving around the Telehash protocol. As far as I can tell, that effort is discontinued, or at least no longer Open Source.

21stio 6y ago
Hey, we are building something similar. Please checkout https://metamate.io/blog/most_advanced_hackernews_api https://metamate.io/blog/most_advanced_hackernews_api. For now, we concentrate on reading public data from various social sources. We just released a HackerNews service and going to add more over the next couple of weeks. Here's a little application build on top of a MetaMate http://showcase.metamate.io/hackernews-activity http://showcase.metamate.io/hackernews-activity. Please let me know what you think :)

chrisma0 6y ago

The linked post and video (including a demo) of Andrew Louis's 'Memex' project is really neat as well! Really cool interface there: https://hyfen.net/memex/ https://hyfen.net/memex/

hyfen 6y ago
Thanks! Had a nice chat with karlicoss earlier this week to compare notes.
nikisweeting 6y ago
Memex is really awesome, I'm excited to see more of these solutions being developed with different approaches.

wakkaflokka 6y ago

Like a lot of other posters, I’ve been working on something like this off and on but never near to completion. Thank you for taking it all the way!

AJRF 6y ago

Polar opposite - Does anyone enjoy digital amnesia? I feel like I might be a little weird here, but every so often I'll just go in and delete all my notes, reminders, apps, photos, etc and start again. Wondering if anyone else does this? I like doing it because it helps clear my head, even though the files or data are digital they take up some space in my head. I do it every few months, just have a fresh slate. I wonder if that says something about me - I'm not really sure. It's probably just another form of procrastination. I once read Stephen Wolframs post about how he has a key logger that backs up every keystroke he types and its all indexed and he has a little front end over his enormous amount of data and it made me feel anxious just even reading it.

graphpapa 6y ago
I do something halfway in between - i routinely get frustrated with everything that’s not relevant to my present concerns, and just throw it into a folder called ‘archive’ and then move THAT into the big central archive folder. I never really look back through any of it but when I do it’s quite nostalgic.
- AJRF 6y ago
  I should probably start doing this for the (very) few times over the years I've regretted deleting something.
- wasureru 6y ago
  I do the same thing too. I’m not sure if I’ll ever look back through any of it but it’s nice to know that if for whatever reason I feel the need to review old data, my future self will be relieved that it’s not gone into the digital ether.
brian_herman 6y ago
I love deleting my accounts every year, It feels so liberating and cathartic to just click that button and have my information go to /dev/null.
pjc50 6y ago
I think I'm with the sibling commentators: deleting something when I could keep it feels like an obliteration, a destruction, a waste; on the other hand, realistically I don't need it and am almost certainly never going to look at it again. Almost. But if I did and wanted to and couldn't, I'd probably feel loss. So I chuck it in an archive. Gradually expanding storage at low prices means never having to think about deletion. I don't use any of the systems for sharing tabs or history across browsers, because each has a different set of almost abandoned tabs. Sometimes "tab bankruptcy" is forced on me by software failure, and it's frustrating.

Neph 6y ago

This is awesome, I'm working on an personal, self hosted, IoT System from scratch where I have several fetch agents in huggin to get data and store it in "The Archive" (DataStorage Server) Than I have different API's to that my IoT devices call to fetch the data from The Archive to trigger events and generate dashboards. This might be a great way to optimize it and save my some development time for things that I don't have yet implemented. (Twitter archive, messages, etc) Some examples of things already implemented: - Raspberry Pi Touchscreen on my desk with dashboards -> Calendar -> Financial Dashboard (Net Worth, Stocks, Expenses) -> Reminders (ongoing to do list, train delays etc) -> Dashboard with my indoor air quality/temp and plant sensor - Self made night light + alarm clock -> Calculates time I need to go to bed to get 8 hours of sleep and indicates it with a red light. -> In the morning triggers alarm and reads highlight, light shows weather forecast

21stio 6y ago
Sounds interested. Do you have a link to share?

ozim 6y ago

One day you wake up and realize that you spent your life maintaining your system to improve your life instead of actually improving your life.

keiferski 6y ago
https://en.wikipedia.org/wiki/Map–territory_relation https://en.wikipedia.org/wiki/Map–territory_relation
- notJim 6y ago
  Indeed. Today abstraction is no longer that of the map, the double, the mirror, or the concept. Simulation is no longer that of a territory, a referential being or substance. It is the generation by models of a real without origin or reality: A hyperreal[1]. The territory no longer precedes the map, nor does it survive it. *It is nevertheless the map that precedes the territory— precession of simulacra—that engenders the territory.* [1] https://en.wikipedia.org/wiki/Hyperreality https://en.wikipedia.org/wiki/Hyperreality
jl6 6y ago
Arnold Rimmer’s revision timetable.
themodelplumber 6y ago
Starting on a project to improve one's life by engaging latent gifts is one of the smartest things one can do. It is aligned with the best advice psychology can offer. If that means maintaining a measurement system works for you--great. It's not as if we lack data showing that measurement systems are beneficial to human development.
twicetwice 6y ago
Only if you think of "maintaining your system to improve your life" as work. I would think of it more as play. It's not time wasted if you enjoy it. Hm. I actually don't agree with that sentiment as written. It's not time wasted if you enjoy it and have something to show for it? That's not it either. Maybe this. I view the work we're discussing like a programmer's equivalent of woodworking in your garage. Sure, it's not necessary. But it's creative, fulfilling, and enjoyable. There's something immensely satisfying about using something you made with your own two hands, even if it's not perfect. As long as all these things remain true, I wouldn't call it time wasted. To me your critism reads something like, "if you wanted to canoe, why did you spend a year making your own? you could have bought one and spent that year doing what you really wanted to do!" Well, I wanted do that, but I also enjoyed making something to do that too. I feel my life is better for having made it.

op00to 6y ago

I just need a simple way to make sure I am doing the things that are important to me and make my life better.

A4ET8a8uTh0 6y ago

I am of two minds about it. First off, there is something deeply satisfying about seeing your stuff properly categorized and easily accessible when needed so I definitely understand the impulse. And the result looks like it does exactly that. Kudos. There is a part of me that dislikes this level of insight someone can have into my life and habits ( yes, it is intended for owner use, but the data is there, neatly organized for someone to access ). It is a weird comment for me, because I absolutely see the benefit of this project.

karlicoss 6y ago
I can see your concern! I guess I've got several thoughts on it: - One is along the lines of post-privacy: Lots of the stuff that I'm collecting one can find online anyway if they deliberately search for it. Tweets/reddit comments/instagram photos, etc. Some of it isn't public, let's say, Amazon purchases. But would I mind sharing it? Maybe not, who cares what I buy? Some ML algorithm that would show me ads? Whatever, I don't click on them anyway. The more stuff I don't mind to be public in the first place, the less is the chance someone can use it against me. But I understand it's ultimately something very personal. - Second is that I agree that the security professionals would handle the data much better than me. I don't even mind Google keeping my data, if only it was easier for me to access it when I need it! So generally I would happily pay professionals for a service to keep my data safe, being able to access it and integrate together, and for the infrastructure around it. Same way I'm using a bank to keep my money safe. But such infrastructure for data simply doesn't exist at the moment, because there is no demand for it from people. And silos only benefit the companies that keep this data, so the change isn't going to be come from them. E.g. if you can easily access your tweets through a nice and fast local app, why would you even visit bloated twitter.com, which is trying to 'entertain' and 'engage' you? I'm hoping that with projects like this, I can inspire people to think differently about their services and tools, and demand (from professionals, i.e. engineers/designers/product managers) for better means of using their own data.
YayamiOmate 6y ago
> There is a part of me that dislikes this level of insight someone can have into my life and habits ( yes, it is intended for owner use, but the data is there, neatly organized for someone to access ). Well this is already happening. Google and facebook for example earns basically all of their money from this. They replaced advertisment middle men because they can track and quantize human behaviours and identities. They do it because they are able to aggregate the data and provide API for marketing people. This is a tool that brings it closer to you as the creator of the data. it doesn't do anything new just exposes what's going on. If you are able to open pages through EU check out data processing partners from GDPR consent you have to accept. Lists on every professionally built services are long. There are literally thousands of companies accessing, aggregating, processing and reselling the data and their analysis. The tools you are afraid of already exist and are commercially available.

simonw 6y ago

My version of this is called Dogsheep, and involves building tools that suck my data from various sources into SQLite databases so I can query it all using Datasette. https://dogsheep.github.io/ https://dogsheep.github.io/ I enjoyed reading this author's essay on why he doesn't think loading everything into relational databases is the best approach: https://beepb00p.xyz/unnecessary-db.html https://beepb00p.xyz/unnecessary-db.html I think I have a workarounds for most of the issues he describes there. One example: I mostly create my base table schema automatically based on the shape of the incoming JSON, and automatically add new columns if that JSON has keys I haven't seen before. Very thought-provoking writing (and software) here.

karlicoss 6y ago
Ooooh, I totally forgot to add your link, I remember you commenting, sorry! Will amend now. Ah I see, so your approach is using databases as an 'intermediate' layer, rather than the main source of 'truth'. My main objection to databases is the fricton with the maintenance and the speed of prototyping, but if it's automatic and used as a cache, I agree it's reasonable. Although I'd still be worried about normalising something wrong, but this is my personal sort of anxiety :) I've got something similar [0]: if you have a sequence of NamedTuples/dataclasses, you can get a database 'for free'. [0] https://beepb00p.xyz/unnecessary-db.html#cachew_cachew https://beepb00p.xyz/unnecessary-db.html#cachew_cachew
mholt 6y ago
Cool, that's similar to my project Timeliner: https://github.com/mholt/timeliner https://github.com/mholt/timeliner It downloads your data from social media (or other online sites) into a single sqlite database. It can even draw relations between various accounts/people. I use it to back up my Google Photos.
- mathnmusic 6y ago
  There's also heedy: https://heedy.org/ https://heedy.org/
  - 6y ago

edraferi 6y ago

Very cool! Intake [0] is another package that might help here. It organizes a set of data sources into (1) plugins that actually connect to the data source and map the data to standard Python data structures like Data Frames (2) catalogs that reference the plugins you want to use alongside project specific metadata like usernames/passwords/source URIs (3) convenience functions that persistence, concatenation, etc (4) a GUI for browsing data sources Looks like OP has already written a lot of data access logic... I might fork them into Intake plugins for my personal use. [0] https://github.com/intake/intake/blob/master/README.md https://github.com/intake/intake/blob/master/README.md

karlicoss 6y ago
Wow, looks interesting, thanks for the link. Please share if you integrate it!

chrisweekly 6y ago

Fantastic! Thank you for this. Agreed on most of your points, and stoked to give your scripts a try soon. :)

jackklika 6y ago

> What it isn't? > It's not vaporwave I look forward to the day we replace vaporware with vaporwave, ala segue>segway

nico_h 6y ago

Reminds me of the Feltron report[1], in the self-quantization realm. The person did it for 10 years and stopped in 2014. The accent was put on the visualization. http://feltron.com/FAR14.html http://feltron.com/FAR14.html

andrey_utkin 6y ago

This is amazing. I am putting as much as i can into my TODO+archive database (see https://github.com/andrey-utkin/taskdb/wiki/Live-demo#workout-dashboard https://github.com/andrey-utkin/taskdb/wiki/Live-demo#workou... ), and it is pretty neat already for analysis with querying and visualization. But your stuff is orders of magnitude bigger. Possibly I will set up HPI for myself some day.

karlicoss 6y ago
Looks cool! HPI is more about accessing data, so it should be possible to combine the projects. I was thinking of using Grafana too, and adding an automatic influxdb integration (via https://github.com/karlicoss/cachew https://github.com/karlicoss/cachew, which I'm already using for sqlite), so let me know if you set up something! :)

adrienchauvet 6y ago

I think you should check this project form Tim Berners Lee: https://solid.mit.edu/ https://solid.mit.edu/

karlicoss 6y ago
Hey, thanks. I'm listing it as once of the links, and I tried using it, but I kind of struggled to see if it can integrate with the existing infrastructure.

doelie_ 6y ago

Cool project. I have been on the lookout for something like this. The idea of exocortex seems to be something that many people want, and almost universially re-invent because the way in which structuring makes sense is different for everyone. I like the approach of making it modular so people can pick and choose parts and do their own integration. What I learned from thinking about this problem is that this kind of integration is really hard and almost impossible to tackle without an evolutionary approach, and it made me go meta and focus on incremental development of the integration layer first. My interpretation of the idea of exocortex might be a bit different in that the focus is not so much about information per se, but more about systems that interact with the physical world. Here's a writeup in case anyone is interested in where it took me: https://github.com/zwizwa/erl_tools/blob/master/doc/exo.org https://github.com/zwizwa/erl_tools/blob/master/doc/exo.org