21 ms·
I reverse engineered Google Docs to play back any document's keystrokes (2014)
- edoceo 6y agoCan we add 2014 to title?
- deleted 6y ago[deleted]
- chews 6y agoFun read and clever hacking. It's fun to think about the insane resolution of the breadcrumbs we leave online.
- drtillberg 6y agoStoring keystrokes used to make a document reminds of a comparison between personal data and toxic asbestos. Storing and indexing keystrokes is unique, and it might even be useful, yes. However, this does not at all seem safe to store without user knowledge for every document created. What could possibly go wrong....
- judge2020 6y ago> without user knowledge File -> version history
- rement 6y agoTo be fair there is also: share, new, open, make a copy, email, download, make available offline, rename, move, make a shortcut in drive, move to trash, publish to the web, document details, language, page setup, and print. That is just for the file menu. There are 7 other menus each with their own long list of options.
- boogies 6y agoIIRC the version history only shows snapshots hours apart. Not the sensitive info some of the millions of users have accidentally pasted in and deleted within seconds — which now everyone they share edit access with years later can see, as well as Alphabet and the various alphabet-soup agencies of the government they share a revolving door with.
- asdfasgasdgasdg 6y agoThis has been a feature of Google Docs from jump. And it has never been a secret that you can browse doc version history.
- FartyMcFarter 6y agoYes, but that version history is not as detailed as every keystroke, so it's not obvious that every keystroke is recorded.
- wrnr 6y agoThis is just silly, I've worked on systems to write text and that is just wat you do to solve the problem of (collaborative) editing a document.
- drtillberg 6y agoIt's just super interesting that the granular record is persistent and can be replayed years later. Pretty non-obvious to me, and something I usually pay attention to!
- wrnr 6y agoThe event log of key strokes is the source of truth for the entire document editing system, I can't think of a reason why one can't rewrite history after a certain time, except to say that this requirement will probably impact a lot of sub-systems from storage to UI, that now have to keep working under historical revisionism. I certainly haven't bothered doing that. It's probably one of those situations where you'd write a "Diversion Report" for the security team where you talk about limited assets at rist and a huge investment in engineering resources.
- jariel 6y agoYou definitely don't need that level of granular detail from 10 years ago. There is no use-case for reproducing every keystroke. Major edits would be fine after literally only after a major save, or a day has passed. FYI - this is a kind of 'biometric' leakage eh. Your typing style is probably as unique as your finger print. It's not shockingly bad, but probably something, under thoughtful review, they should take care of. Edit: by 'no use case' I mean, no use case for long term history. Obviously there's a good use case for 'very short term' individual keystrokes. Also Edit: the answer to this issue is not 'Don't worry about it, it's a low risk problem'. This is the 'slow boil' SV way of thinking that is seemingly benign, but not acceptable in the long run. The answer is: don't store personal data that is unnecessary. It's that simple. It doesn't diminish the product a single bit.
- asdfasgasdgasdg 6y ago
- Jugurtha 6y agoI thought the post would address what they do with the changes from individual contributors and how they merge them and manage conflicts. It's been known for quite a while that Wave used OT, and if I recall correctly, some engineers there said that it would take them as long to implement it again [I think two years] in discussions about CRDT/OT, etc.
- manigandham 6y agoGoogle Docs also uses OT, with a single master server per document that handles all of the mutations.
- malka 6y ago>Everyone I’ve talked to about this has been surprised, and maybe a little unnerved, to discover that whenever they share a Google Doc with someone, they’re also sharing an extremely detailed record of them typing the thing.) No shit. I'm stopping to share anything on Google docs now. Who knows what secrets are here somewhere ?
- momokoko 6y agoI always make a copy and then share that when I’m ready. This is the only method I’m aware of to remove history.
- oauea 6y agoIndeed. It's definitely happened to me before that I accidentally pasted, then removed, some credentials from a Google doc. Usually the expectation is that as long as you don't hit "send" it's fine, but this is clearly not the case here.
- closeparen 6y agoWe've just made it a first class part of our workflow. Colleague tells me he's made a round of edits; I check the history to see what they are. Promotion committees also use edit history to validate whether "coauthors" really are.
- deleted 6y ago[deleted]
- Invictus0 6y agoI imagine the next chapter in this story will be someone scraping the web for public google doc links, reconstructing the keystroke timeline, and training a NN with the timeline in order to break captchas or other keystroke-pattern-based identification systems.
- dheera 6y agoIf it saves that much history I suppose in the future I'll be drafting my stuff outside Google docs and pasting it in, instead of drafting within Google docs.
- robertlagrant 6y agoIt sort of has to do this. How can you see others' keystrokes if they aren't being sent anywhere?
- dheera 6y agoI was more concerned about editing a doc privately, then sharing it, and then others being able to see all the keystrokes during the private edit time prior to sharing. I would have hoped that it cuts off diffs/undos to a certain point in history.
- parasubvert 6y agoYou can always make a copy which clears the history.
- aaronharnly 6y agoI’ve long wanted what seems like a simple feature in GDocs: right click and “show me who wrote this”. All the data is there, and can eventually be determined by inspecting the history, but it’s not well exposed.
- savrajsingh 6y agoI love how git calls this feature “blame” :)
- _fat_santa 6y agoFunny story, I was once in a company meeting and said something alone the lines of "blame is on X Engineer for that feature". Non technical product manager thought I was actually calling the guy out. Well sorta, but more technically.
- mrshu 6y agoStrangely enough, this does seem to exist for Google Sheets: https://webapps.stackexchange.com/questions/73080/is-there-a-quick-way-to-see-when-a-cell-in-a-google-sheets-was-last-edited/131938#131938 https://webapps.stackexchange.com/questions/73080/is-there-a...
- mfontani 6y ago$ git config --global alias.praise blame # Who wrote this most beautiful piece of.. code?
- kzrdude 6y agoI thought something like that shipped by default, but I can't find it. I find git annotate though, a confusing almost doppelgänger to git blame.
- gokhan 6y agoSVN called it blame, before git. There might be other predecessors.
- ve55 6y agoIt's crazy how vast and powerful databases of even the most simple things turn out to be. You just store characters and context information when someone types them, and this is the result. Using this data alongside ML, you could do things such as: - Identify people based upon their typing style easily, given so much data about them - Find a lot of secrets that were accidentally pasted into document including passwords, keys, etc - Try to model the emotions people were feeling as they typed things - Much, much more, given some data and time
- fouronnes3 6y agoAll valid points, but to be fair you can already do all those things with regular documents.
- bluntfang 6y agosure and i can find out your real time location by hiring a PI to follow you around all day, but it's a lot cheaper and easier if i just pay google for it. plus i cant hire a PI to follow everyone around, there's a cost to that.
- Xenograph 6y agoThe point is that all the data is centralized in a single database in this case.
- BorisMelnik 6y agothe difference being the database in this case is a culmination of god knows how many accounts on the www several TB-PB of data so it bit more scary.
- saadalem 6y agoCarmack2Vec: Identifying and quantifying productivity through keystroke patterns.
- o-__-o 6y agoI studied this vector for attack in college[0] almost 20 years ago based on work done in 1986[1]! [0] - https://dl.acm.org/doi/abs/10.1016/S0167-4048(03)00010-5 https://dl.acm.org/doi/abs/10.1016/S0167-4048(03)00010-5 [1] - https://patents.google.com/patent/US4805222A/en https://patents.google.com/patent/US4805222A/en
- Qahlel 6y agoThis sh*t is the reason that I write offline and then copy-paste the final text only at the end.
- cratermoon 6y agoThis sh*t is the reason that I carve my words into clay tablets using LinearB and have an army of assistants to translate and upload them. The only downside to this is that the change history is a bit of a mess until it's cleaned up and uploaded.
- Yetanfou 6y agoThis shit is the reason why we use LibreOffice online, running on my own server when an "office" thing is required. I can access it from all over the 'net, documents only leave my network when I want them to, nobody gets to data-mine them, it has supported nearly everything I've thrown at it.
- jakevoytko 6y agoFor what it's worth, you can make a copy with "File > Make a copy" and delete the edited doc. The copy will not preserve history. You can verify this yourself - after making the copy, the only thing in the copied doc's revision history is a single change that adds the entire document's contents.
- algo_trader 6y agoPET RANT: horrible use of monitor areas by Gdocs. Look at the image from the article. Ughh. Maybe 3rd of the monitor is actually used to display text https://cloud.githubusercontent.com/assets/21294/4930884/70fe3062-656c-11e4-9ce6-f83cfa84f095.png https://cloud.githubusercontent.com/assets/21294/4930884/70f...
- manigandham 6y agoThat's their history extension, not google docs. You can zoom in on documents in Google Docs (or any website in your browser) as much as you want. Also fixed width for text improves readability.
- 867-5309 6y ago>any website CSS permitting
- manigandham 6y agoBrowser zoom is independent of CSS.
- 867-5309 6y agoif that were true then all websites would scale uniformly
- manigandham 6y agoUniformity is an entirely different issue. All browsers can zoom in, regardless of the CSS, but how they implement it is different. Safari on mobile devices uses a visual zoom which is like scaling up an image, while desktop browsers usually scale up CSS units and reflow the layout with the bigger dimensions.
- 867-5309 6y ago
- krm01 6y agoI wonder if ‘journey data’ like this one is a better type of data to feed ML? Showing the corrections, improvements, etc. Any idea if someone tried this?
- dang 6y agoIf curious see also discussed at the time: https://news.ycombinator.com/item?id=8562483 https://news.ycombinator.com/item?id=8562483 also 2015 https://news.ycombinator.com/item?id=9686135 https://news.ycombinator.com/item?id=9686135
- parasubvert 6y agoGiven the history of Google Docs dating back to Google Wave with its architecture of Operational Transformations, this makes perfect sense and is a bit obvious if you know the history. https://en.wikipedia.org/wiki/Operational_transformation https://en.wikipedia.org/wiki/Operational_transformation The new alternative to this approach for collaborative editing (and large scale distributed database concurrency without locks) are CRDTs: https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...
- fortran77 6y agoI always wonder if Google Wave was just ahead of its time. Why can't any email be the start of a collaborative document, and if you and the recipient happen to be online and available at the same time, turn it into a chat?
- hayksaakian 6y agoI think Google Wave proved that UX confusion was a bigger hurdle than the tech for collaborative editing.
- saurik 6y agoI mean, Google Wave was also very slow and unusable in practice as it was invite only in a way that meant even in my pretty tight knit circles of "people who all went to Google I/O almost every year" there was always at less than one person who didn't have access to it... I don't see how it possibly could have succeeded; it is as if they fundamentally didn't understand that the reason that worked for Gmail was because Gmail was just one option in a federated ecosystem, and so everyone already had email, but would never work for a real communication tool building an audience from scratch.
- fortran77 6y agoBut groups are on Slack, etc, which everyone doesn't have.
- aboringusername 6y agoWhilst this is on the front of hackernews there is one element to this that intrigues me...And that's how this is compliant with the GDPR? For example, as far as I am aware this isn't information that is included in Google Takeout, nor is it included in any SAR that you may ask for. There is no ability to have a list of comments you've made and a choice to delete them. Even if you delete your account I am not sure these pointers are actually removed from the database. Google collects so much data across so many vast areas I am not sure how they can ever be truly GDPR compliant and in accordance with the law as it's written. For example, how do I delete my Google Fit data? It is seemingly impossible, of course this is where you have to ask them to. How about deleting every single edit history across all of my documents, or a specific edit in a specific document? Will they be able to remove that unique entry in the edit history? I do know they store edit history for an indefinite period of time, I've accessed documents 6 years old (never edited again, unopened for many years) and the edit history is still perfectly intact. I wonder how many PB's of data Google has of pure and utter data-rot, and why they don't have systems in place to clean that up (especially old, inactive data like document comments or revision history). I imagine every single record Google creates and logs must be liable to the GDPR, so of the trillions of data points they have on their systems it must be a pain to try and ensure any of them can be modified if required to do so.
- closeparen 6y agoHow do you delete a line from an email that you sent to someone else’s GMail account?
- Nextgrid 6y agoThis feature most likely predates the GDPR and it's just an oversight that a toggle hasn't been implemented for it. However, Google has demonstrated their bad faith and unwillingness to comply with the GDPR with their non-compliant consent popup so I would be very surprised if they suddenly decide to spend resources on modifying this feature to make it compliant.
- mandeepj 6y agocan someone please throw a light on the framework used behind the UI of Google docs or sheets? Is it opensource?
- manigandham 6y agoIt's Google's proprietary code. Google Docs is too unique, complex and valuable to use any typical opensource UI framework.
- applecrazy 6y agoActually, Google Docs was (and possibly still is) built on Google Web Toolkit (http://www.gwtproject.org/ http://www.gwtproject.org/)
- mandeepj 6y agoI believe other apps like Microsoft office 365 word, Airtable, Quip, Smartsheet etc. - are very similar in frontend architecture to Google Sheets. So, I was looking if there's a common engine or approach to handle UI
- erickj 6y agoI was working on the Google Slides team at the time this blog post was originally published back in 2014. Slides and Doc are extremely closely related projects, sharing lots of code (and even sitting next to eachother, at the time at least). Our entire team was impressed with the details of the operational transform protocols that the author was able to reverse engineer, and for the most part get correct. IIRC our engineering director ended up calling the guy up for a lunch.
- londons_explore 6y agoSending around protobufs encoded as JSON makes reverse engineering much easier than you might imagine. I can just imagine what the internal .proto files must look like - especially the arrangement/ordering of fields which tells you how the software design evolved. Two fields with tags next to eachother? They're probably part of the same feature and they were added by the same person as part of the same commit.
- harikb 6y agoEdit: u meant encoded as Json - never mind. Just curious, how does Protobuf make it easier? Easier than Json? Protobuf doesn’t even include field names or readable descriptors - unless of course you are talking about reading client side JS
- londons_explore 6y agoNot easier than JSON, but it's much easier than reversing some C++ structs that some webassembly based application might send over the wire... Messages and submessages and arrays make figuring out what data is what very easy for the most part.
- the_arun 6y agoJust curious - Did you guys hire that person to work for Google? or let them go just with a lunch?
- 6y ago
- z3t4 6y agoI use operational transformation (OT) for undo/redo and collaborate editing in an editor that I wrote (for) myself. Inspired by Scrimba I made it so I can "record" tutorials by adding time stamps to the OT undo/redo history. Then when replaying the tutorial/session (which is also voice annotated) you can write code as well while the record is playing thanks to the OT.
- kccqzy 6y agoFun fact: when you do a "virtual on-site interview" (an oxymoron I know) at Google, the interviewer does get to see a playback of whatever the interviewee typed. There's a bar that allows you to select the time just like a movie playback UI.
- yjftsjthsd-h 6y ago> a "virtual on-site interview" (an oxymoron I know) ... But seriously, what? What is a virtual on-site meant to be?
- LeegleechN 6y agoIt's an interview that would have been an onsite in non-pandemic times. In other words it means you're past the screens and you're now doing a full panel of interviews, possibly including members of your prospective team.
- philsnow 6y agoIn hiring committees I've sat on, people for some reason had a lower bar for code that was written during a "phone screen" than during an "on-site", even when both were done with the same tech "stack" (phone call / Skype + coderpad or similar). Maybe the name "virtual on-site" is supposed to convey to both interviewer and interviewee that "this is for serious".
- bsoft16385 6y agoTypically you only get one or two phone screen interviews at Google, so the bar should be much lower. The purpose of the phone screen is to see whether it's worth the time for onsite (now virtual onsite) interviews. This may vary by hiring committee, but on mine we don't look at feedback from the phone screen interviews at all.
- dannyw 6y agoAs a candidate, I will put in as much effort into phone screens (with real code pads) as the real interview. As an interviewer, I will give a slight benefit to the doubt for “screens” because I understand some candidates don’t take it as seriously as the main interview.
- vkammerer 6y agoIt would be wise to support https when publishing content that may be security sensitive. Both OP's site and the site used for the extension run on http.
- Johnyma22 6y agoheh I maintain the timeslider code in Etherpad, it's actually really quite simple and there is a nice set of visual representation and auditing/diff plugins now available made by the community :)
- cantrevealname 6y agoThis is going to open a huge can of worms when lawyers demand the document history for discovery. You think that the final contract, patent application, user manual, or product specification you wrote is what matters? When the lawyers, regulators, law enforcement, and litigious people want to get you, you'll have to defend every edit back to beginning of time. "Yes, we know what the final contract says, but based on your edit on January 7, 2003, at 5:21 pm and 0.210403 microseconds, your real intent was ...". "And then on June 12, 2006, you took out some words that have a bearing on this lawsuit -- obviously you were trying to cover things up." I was quite surprised to learn that during patent litigation for example, the history and changes made to your application do matter. I assumed that the final patent as issued is what matters and everything else is irrelevant. Well, lawyers have made it matter, and now things will be 10,000 times worse when they want to find something to argue. So great, now we have to worry about how someone might interpret every single edit rather than the final document.
- deleted 6y ago[deleted]
- simonh 6y agoThen use something else. That doesn’t mean those of us who find such a feature valuable shouldn’t have tools with these capabilities.
- cantrevealname 6y ago> Then use something else. Sure, until that feature is built into almost everything and it becomes mandatory to use it. Filing an income tax return? You must use this online system that captures every edit and every keystroke. There is no other way to file. When people complain about lack of privacy when using credit cards and bank cards, would you dismiss it by saying, use something else -- just use cash? I could list a hundred things that you can no longer pay with cash.
- mcherm 6y agoLook, if you can assume that the legal system will start caring about specific edits along the way to the final draft, and you can assume that essentially all tools will start tracking the history of these changes, and you can assume that the trivial fix (copy paste the final version into a new document) will be underlined either technically or legally, then I can assume that Congress will change the law to make these inconsequential details no longer matter. (It's highly unlikely that Congress would actually do a useful thing, but then the sum total of all your assumptions was also highly unlikely, wasn't it?)
- porpoise 6y ago“We cannot judge either of the feelings or of the characters of men with perfect accuracy from their actions or their appearance in public; it is from their careless conversations, their half finished sentences, that we may hope with the greatest probability of success to discover their real characters.” - Maria Edgeworth It's interesting to think about what is it about keystrokes that tend to make people worry about privacy. I don't think it's as simple as "they are storing a lot more data than I assumed". It can't just be a matter of degree. The reason must be broadly along the lines given in the Edgeworth quote.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]