5 ms·
Surfer: Centralize all your personal data from online platforms
- doctorpangloss 2y agoThe most exciting thing to happen to programming is the chatbot enabling millions of enthusiastic people to write code.
- Terr_ 2y agoI dimly remember some kid--or maybe it was only apocryphal--in the early 2000s, where they tried typing "Like Halo but with X" into a text file before changing the extension to .exe... Still silly, but closer.
- cuu508 2y agoI remember a similar story from 2000s where somebody interested in graphics programming saved "make some cool effects" as exe.
- madamelic 2y agoNot sure if we are joking about ourselves but when I was a kid, I was so confused by how games were made. I started drawing individual frames of the game I wanted, I remember about 45 minutes into this venture I had an existential crisis about how how many frames you'd need for something like GTA, to show every possible combination. I had the right idea but wasn't thinking about how to leverage the computer correctly.
- mandmandam 2y agoHaha I had the exact same experience with Prince of Persia. The existential crisis took nearly a decade to wear off. The realization that every possible image that can fit on a screen can be stuck in a bitmap helped keep it going. Everything that could possible ever be photographed, just sitting there in the latent space waiting to be summoned. ... And now, I'm amazed that through some basically fancy noise we can type in words and get pictures in under a second. They even almost have the right number of fingers.
- colordrops 2y agoSeems like an easy way to get locked out of your accounts.
- noman-land 2y agoAs long as it happens after the scraping is over.
- Carrok 2y agoNo list of supported platforms. No example of what the extracted data looks like. No examples of what can be done with the extracted data.
- SJMosley 2y agoclosest thing to a supported scraper list https://github.com/CEREBRUS-MAXIMUS/Surfer-Data/tree/main/src/main/Scrapers https://github.com/CEREBRUS-MAXIMUS/Surfer-Data/tree/main/sr...
- swyx 2y agoposter here - yeah if i have one cricitism of their readme/marketing this is it. you can see it in the demo video but this needed to be up front
- slalani304 2y agothanks for the feedback, added the list of supported platforms and example of exported data as well to the readme. we're focusing mostly on the exporting part, so examples of what can be done with the data will come later.
- fudged71 2y agoOne example of how I used it: It exported 75MB json of ChatGPT "Conversations". I extracted 19MB or raw text from this as a CSV. I then took this into Nomic.ai and embedded all of the text to create a clustered visualization of topics in my ChatGPT conversations.
- slalani304 2y agoHey, would love to see a visualization of this / how you made it :)
- mcslurryhole 2y agoas someone who used to write scrapers for a living, this is going to break constantly. cool concept though.
- pogue 2y agoThat's what's going to make this software live or die. It needs 1) Constant updates to existing packages 2) Continued expansion of more sites/apps to export your data from
- slalani304 2y agoagreed. curious to hear what other sites/apps you would want to be able to export your data from.
- captn3m0 2y agoI’ve been working on a lot of similar ideas over the years, and my current ideal stack is to: 1. Use Mobile App APIs. 2. Generate OpenAPI Arrazo Workflows. 1 ensures breakage is minimal, since mobile apps are slow upgrades and older versions are expected to keep working. 2 lets you write repeatable recipes using YAML, and that makes it quite portable to other systems. The Arazzo spec is still quite early though, but I am hopeful of this approach.
- deleted 2y ago[deleted]
- hi-v-rocknroll 2y agoThe answers to online platforms trafficking in personal data and metadata is two parallel and concurrent efforts: 1. Much tougher data privacy regulations (needed per country) 2. A central trusted, international nonprofit clearinghouse and privacy grants/permissions repository that centralizes basic personal details and provides a central way to update name, address(es), email, etc. that are then used on-demand only by companies (no storage) By doing these, it simplifies things greatly for people and allows someone to audit and see what every company knows about them, can know about, and can remove allowances for companies they don't agree to. One of the worst cases is the US where personal information is not owned by the individual and there is almost zero control unless it's health related, and can be traded for profit.
- bboygravity 2y agoYes, I agree. I hate web-forms so much that I wrote a Firefox extension that fills them automatically (using LLM). It sounds to me like what you're describing under 2 is a real usecase for blockchain contracts? Store your latest data encrypted on-chain and give every 3rd party you trust a key that corresponds to the relevent part of the data? Curious about opinions on this.
- pogue 2y agoCan you share this Firefox extension? Sounds super handy!
- hi-v-rocknroll 2y agoI'm not talking about a distributed or self-hosted technical solution, but a centralized trusted nonprofit organization. Technology alone can't automate away privacy management issues.
- slalani304 2y ago3rd party would only have read access, I'm assuming? Also would love to try out the extension.
- ianopolous 2y ago
- methyl 2y agoNot to be confused with Surfer, SEO content optimization platform
- doodlebugging 2y agoOr the 40 year old geoscience software product Surfer [0] [0] https://www.goldensoftware.com/products/surfer/ https://www.goldensoftware.com/products/surfer/ Cool name but it was taken way back when I was writing geoscience software. That's been a while.
- MattJ100 2y agoDefinitely not the first such scraper. DogSheep has been around for a while: https://dogsheep.github.io/ https://dogsheep.github.io/ It is based around SQLite rather than Supabase (Postgres) which I think is a better choice for preservation/archival purposes.
- slalani304 2y agoOh, interesting will look more into this.
- AeZ1E 2y agothe idea of personal data centralization sounds intriguing, but let's be real - companies will always find a way to keep a grip on our info. maybe it's time for a digital revolution, or just another excuse for me to procrastinate on coding.
- BodyCulture 2y ago„Centralize“ is a privacy anti-pattern. Max centralisation should be your keepass file.
- markjgx 2y ago"Surfer: The World's First Digital Footprint Exporter" is dubious—it's clearly not the first. Kicking off with such a bold claim while only supporting seven major platforms? A scraper like this is only valuable if it has hundreds of integrations; the more niche, the better. The idea is great, but this needs a lot more time in the oven. I would prefer a cli tool with partial gather support. Something that I could easily setup to run on a cheap instance somewhere and have it scrape all my data continuously at set intervals, and then give me the data in the most readable format possible through an easy access path. I've been thinking of making something like that, but with https://github.com/microsoft/graphrag https://github.com/microsoft/graphrag at the center of it. A continuously rebuilt GraphRAG of all your data.
- slalani304 2y agoYeah it was honestly more of a marketing statement lol, but removing it for sure. Adding daily/interval exporting is one of our top priorities right now and after that and making the scraping more reliable, we'll add something similar to GraphRAG. Curious to hear what other integrations you would want built into this system.
- madamelic 2y agoTake a look at https://github.com/karlicoss/HPI https://github.com/karlicoss/HPI It builds an entire ecosystem around your data where it is programmatic rather than just dumping text files. The point of HPI is to build your own stuff onto it and it all integrates seamlessly together into one Python package. The next stop after Karlicoss is https://github.com/seanbreckenridge/HPI_API https://github.com/seanbreckenridge/HPI_API which creates a REST API on top of your HPI without any additional configuration. If you want to get more fancy / antithetical to HPI, you can use https://github.com/hpi/authenticated_hpi_api https://github.com/hpi/authenticated_hpi_api or https://github.com/hpi/hpi-graph https://github.com/hpi/hpi-graph so you can theoretically expose it to the web (I am squatting the HPI org, I am not the creator of HPI). I made the authentication method JWTs so you can create JWTs where it will give access to only certain services' data. (Beware, hpi-graph is very out of date and I haven't touched it lately but my HPI stuff has been chugging away downloading data). Some of the /hpi stuff I made is a bit mish-mash because it was rip-and-replace from a project I was making so you'll see references to "Archivist" or things that aren't local-first and depend on Vercel applications.
- slalani304 2y agohey, sahil here. i'm one of the contributors on surfer and have been working on this project for around three weeks now. we appreciate the feedback and are excited to keep pushing this project forward with your input!
- doodlebugging 2y agoSahil, I don't know anything about trademarks, service marks, etc but I do know that the product name "Surfer" has been in use for about 40 years in my industry, geoscience, by a company in Golden, Colorado. [0] Maybe you can make a new product in a different industry and recycle the name. I don't know how that works but right now, you're playing in an established product's namespace. [0]https://www.goldensoftware.com/products/surfer/ https://www.goldensoftware.com/products/surfer/
- bdcravens 2y agoAll that's needed is to extend the name with a modifier, which probably is a stronger branding IMO. https://surfer.nmr.mgh.harvard.edu/ https://surfer.nmr.mgh.harvard.edu/ https://towey-websurfer.apponic.com/ https://towey-websurfer.apponic.com/ Also many open source libraries have also used the name: https://rubygems.org/gems/surfer https://rubygems.org/gems/surfer https://www.npmjs.com/package/surfer https://www.npmjs.com/package/surfer etc
- bdcravens 2y agoMyself, I'd probably prefer to use something like Huginn to create a customized approach to all of my online platforms I'm interested in, rather than a curated list. https://github.com/huginn/huginn https://github.com/huginn/huginn
- zamubafoo 2y agoI made something like this since I was tired of the asymmetric nature of data collection that happens on the Internet. Still not where I would like to be, but it's been really nice being able to treat my browsing history as any old log that I can query over. Tools like dogsheep are nice, but they tend to rely on data being allowed to be removed from the platform. This bypasses those limits by just doing it on the client. This lets me create dashboards to see usage for certain topics. For example, I have a "Dev Browser" which tracks the latest sites I've visited that are related to development topics [1]. I similarly have a few for all the online reading I do. One for blogs, one for fanfiction, and one for webfiction in general. I've talked about my first iteration before on here [2]. My second iteration ended up with a userscript which sends the data on the sites I visit to a Vector instance (no affiliation; [3]). Vector is in there because for certain sites (ie. those behind draconian Cloudflare configuration), I want to save a local copy of the site. So Vector can pop that field save it to a local minio instance and at the same time push the rest of the record to something like Grafana Loki and Postgres while being very fast. I've started looking into a third iteration utilizing MITMproxy. It helps a lot with saving local copies since it's happening outside of the browser, so I don't feel the hitch when a page is inordinately heavy for whatever reason. It also is very nice that it'd work with all browsers just by setting a proxy which means I could set it up for my phone both as a normal proxy or as a wireguard "transparent" proxy. Only need to set up certificates for it work. --- [1] https://raw.githubusercontent.com/zamu-flowerpot/zamu-flowerpot/main/Screenshot_20240823_105819.webp https://raw.githubusercontent.com/zamu-flowerpot/zamu-flower... [2] https://news.ycombinator.com/item?id=31429221 https://news.ycombinator.com/item?id=31429221 [3] http://vector.dev http://vector.dev
- Xen9 2y agoA browser addon that takes one's password manager export & deletes every account, possibly after scraping the data, would be amazing. No one has done it and it can be done such that the system will eventually safely delete every account of every site (e.g using developers tools, accessability options, being intended to only be used in fresh browser install, sourcing information from volunteers). You'd have a sheet tracking the process, e.g. verification pending, manual intervention pending, deleted, waiting states. Many humans have hundreds of accounts they no longer use and this sort of tool could thus be a good Y-combinator or hobby project.