6 ms·
Ask HN: Do you have more or less than 1Mb text of your own writing?
Of all that you have written in ypur lifetime, do you have in your possession more or less than 1Mb (180,000 words) of text?
Printed, blogged, work-related, personal diary, fiction practice, emails you personally wrote (not so much copies of boilerplate, or coauthoring when added for other reasons than actual written words).
Asking out of curiosity.
- apothegm 3mo agoEasily. I probably generate 180k words per quarter at work and another 180k for personal and social media purposes. Maybe 180k/month, even.
- cwmoore 3mo agoThanks for the info. Wow, 9000 words per workday seems extraordinary.
- apothegm 3mo agoI might be off by a factor of 2 in that estimate, but not much more. Most of our communication is via chat. Then there are ticket specs and comments. Project overviews and status updates. PR descriptions and comments. Documentation. ADRs. Retros and incident reports. Meeting agendas, notes, and writeups. The occasional email. LLM prompts. Last week I spent 5k words in a few hours just breaking down a large planned project to ticket-level bullet points so we could give an appropriate estimate for roadmapping. (Which turned out to be crucial, because it revealed that the requested timeline for a pilot was too short by at least a factor of 2 even for the most descoped version the stakeholders would accept, and we had the receipts to convince the CTO.) I’m definitely not spending most of my days writing novels, but all the sentences add up.
- l4v3rna 3mo ago[dead]
- KashifNY 3mo agoYes i do, i like to put down my thoughts in writing as well as planning out strategies and such as i found out that i could not find a pen when i needed it much which was a dilemma of mine and so i started just using my laptop and phone to jot down what came to mind and was worth recording
- Bender 3mo agoSome here probably have more than 1mb in comments. There's probably a way to calculate that through the API if someone were so inclined.
- eth0up 3mo agoI just pinged the api to dump my 10 year comment history. Yes, that definitely can be done, in a few seconds and a script. Edit: the csv = 698kb and the json = 866kb Edit 2: api was the wrong term. I used algolia.
- cwmoore 3mo agoGreat datapoint, thanks. My “Ask HN” was not intended to focus on HN comment writing only, but since it headed that way on its own, can you share a scriptlet so that others may do more than speculate?
- eth0up 3mo agoYou ok with python?
- cwmoore 3mo agoI am
- eth0up 3mo ago#!/usr/bin/env python3 """ WARNING: For use on mortals only -- WIll break on tptacek and other mutant laureates """ import argparse import csv import json import sys import time from pathlib import Path import requests API = "https://hn.algolia.com/api/v1/search_by_date https://hn.algolia.com/api/v1/search_by_date" HITS_PER_PAGE = 1000 ALGOLIA_DEEP_PAGE_LIMIT = 10000 # page * hitsPerPage must remain <= this def fetch_page(username, page, created_after=None, created_before=None): params = { "tags": f"comment,author_{username}", "hitsPerPage": HITS_PER_PAGE, "page": page, } filters = [] if created_after is not None: filters.append(f"created_at_i>{created_after}") if created_before is not None: filters.append(f"created_at_i<{created_before}") if filters: params["numericFilters"] = ",".join(filters) resp = requests.get(API, params=params, timeout=30) resp.raise_for_status() return resp.json() def get_total_in_range(username, created_after=None, created_before=None): data = fetch_page(username, 0, created_after, created_before) return data["nbHits"], data def dump_range(username, created_after, created_before, seen_ids, all_records, depth=0): """Recursively bisect time ranges to stay under Algolia's 10k page wall.""" total, first_page = get_total_in_range(username, created_after, created_before) indent = " " * depth lo = created_after if created_after is not None else "start" hi = created_before if created_before is not None else "now" print(f"{indent}Range [{lo} .. {hi}]: {total} comments") if total == 0: return max_reachable = ALGOLIA_DEEP_PAGE_LIMIT # page*hitsPerPage cap if total <= max_reachable: # safe to page through directly page = 0 while True: data = first_page if page == 0 else fetch_page(username, page, created_after, created_before) hits = data["hits"] if not hits: break new = 0 for h in hits: if h["objectID"] not in seen_ids: seen_ids.add(h["objectID"]) all_records.append({ "id": h["objectID"], "author": h.get("author"), "created_at": h.get("created_at"), "created_at_i": h.get("created_at_i"), "comment_text": h.get("comment_text"), "story_id": h.get("story_id"), "story_title": h.get("story_title"), "story_url": h.get("story_url"), "parent_id": h.get("parent_id"), "points": h.get("points"), }) new += 1 print(f"{indent} page {page}: +{new} (running total {len(all_records)})") if (page + 1) * HITS_PER_PAGE >= data["nbHits"]: break page += 1 time.sleep(0.5) # polite pacing, Be a nice human else: # bisect on time: find midpoint of range and recurse both halves lo_i = created_after if created_after is not None else 0 # use "now" as a safe upper bound if open-ended hi_i = created_before if created_before is not None else int(time.time()) mid = (lo_i + hi_i) // 2 print(f"{indent} -> exceeds {max_reachable}, bisecting at {mid}") dump_range(username, lo_i, mid, seen_ids, all_records, depth + 1) dump_range(username, mid, hi_i, seen_ids, all_records, depth + 1) def main(): ap = argparse.ArgumentParser() ap.add_argument("username") args = ap.parse_args() username = args.username seen_ids = set() all_records = [] print(f"Fetching full comment history for '{username}'...") dump_range(username, None, None, seen_ids, all_records) all_records.sort(key=lambda r: r.get("created_at_i") or 0) json_out = f"hn_comments_{username}.json" csv_out = f"hn_comments_{username}.csv" with open(json_out, "w", encoding="utf-8") as f: json.dump(all_records, f, indent=2, ensure_ascii=False) if all_records: fieldnames = list(all_records[0].keys()) with open(csv_out, "w", newline="", encoding="utf-8") as f: writer = csv.DictWriter(f, fieldnames=fieldnames) writer.writeheader() writer.writerows(all_records) print(f"\nDone. {len(all_records)} total comments.") print(f" -> {json_out}") print(f" -> {csv_out}") if __name__ == "__main__": main()
- deleted 3mo ago[deleted]
- billybuckwheat 3mo agoFar, far more than 1 MB.
- cwmoore 3mo agoThanks, that’s a lot. Can I ask for a ballpark figure? And how you have written so much?
- eth0up 3mo agoIn case you don't receive an answer, an alternative to whatever they might have said or not have said, is stupidly simple: Just force yourself to write. Sit down. Lay down, stand on your head. Just do what ever you need to do to force yourself to write. Small personal essays are my suggestion. And a tip: never pause to edit -- just keep writing until you tire, and only then, if you must, go back and scrutinize yourself.
- cwmoore 3mo agoHah! Good advice! What I was asking was for anyone, prolific at writing or or not, to understand their own output in relation to…L1 cache or a context window. But thanks! I guess my issue with writing, as a complete amateur, is the later stages, and followup, compiling or reviewing. Reading is not the only reason to write, but it is an important one. I think it is a relevant human-scale question, relating to and understanding a paltry personal megabyte (or less) on a forum where gigabyte models are half the conversation. Its the wrong metric in a way like LoC, to compare to your comment.
- eth0up 3mo agoIf I am understanding correctly, you are also putting into perspective the dizzying contrast of human vs LLM, where the mere context window far exceeds an average lifetime of writing, which if so, yeah, ... interesting times. Before they ruined Claude with 4.7 and 5, I had more than a few extensive, hundred-page discussions which still kind of blow my mind in review. I miss it as I do some old departed friends. It would get quite amnesiac toward the ends, but still beat the hell out of many conversations I've had with PhDs. On that note, my generated content, or product of my LLM interactions, exceeds 6GB, probably a lot more. Of course, on average, the LLM is producing far more output. But that's ~ two years of heavy, mostly research-based dialog, much of it adversarial, probing the frontier systems themselves.
- dlcarrier 3mo agoIt doesn't take long to build up that many words on online forums, like Hacker News.
- cwmoore 3mo agoThanks for your input. Apparently it does! There’s another commenter on this thread who found less than that in their 10 year history.
- cwmoore 3mo agoThe text content of my iOS Notes, when zipped, is about 500Kb. Some small fraction of that was copy-pasted from someone else’s writing. I probably have an order of magnitude more than that. Bigger than a context window, smaller than a useful model, and fits in L2 cache.
- glitch-hunter 3mo ago[dead]
- senordevnyc 3mo agoWay way more. I write about that in journal entries alone per year
- cwmoore 3mo agoThanks for the feedback! Sounds like you are prolific and consistent! Do you write outside of journaling? Same amount?
- mikewarot 3mo agoI've got 459,203 words posted here on HN alone, according to clickhouse I'm currently ranked #495 https://serjaimelannister.github.io/hn-words/ https://serjaimelannister.github.io/hn-words/ I'm out of the labor pool with nothing but time until I age out. I also have strong minority opinions.
- cwmoore 3mo agoSure, and thanks for the ranking context also. I think 1Mb is an interesting ballpark figure.
- CM30 3mo agoI haven't exactly counted it, but I'd say I've almost certainly have access to more than 1MB of my own writing. I've written articles about a variety of the topics for the last decade or two, posted far more on forums than is probably healthy, and have an untold number of my own emails in my inbox.
- cwmoore 3mo agoThat's the kind of info I am interested in, thanks.