3 ms·

Great datapoint, thanks. My “Ask HN” was not intended to focus on HN comment writing only, but since it headed that way on its own, can you share a scriptlet s

• eth0up 3mo ago
  You ok with python?
  • cwmoore 3mo ago
    I am
    • eth0up 3mo ago
      #!/usr/bin/env python3 """ WARNING: For use on mortals only -- WIll break on tptacek and other mutant laureates """ import argparse import csv import json import sys import time from pathlib import Path import requests API = "https://hn.algolia.com/api/v1/search_by_date https://hn.algolia.com/api/v1/search_by_date" HITS_PER_PAGE = 1000 ALGOLIA_DEEP_PAGE_LIMIT = 10000 # page * hitsPerPage must remain <= this def fetch_page(username, page, created_after=None, created_before=None): params = { "tags": f"comment,author_{username}", "hitsPerPage": HITS_PER_PAGE, "page": page, } filters = [] if created_after is not None: filters.append(f"created_at_i>{created_after}") if created_before is not None: filters.append(f"created_at_i<{created_before}") if filters: params["numericFilters"] = ",".join(filters) resp = requests.get(API, params=params, timeout=30) resp.raise_for_status() return resp.json() def get_total_in_range(username, created_after=None, created_before=None): data = fetch_page(username, 0, created_after, created_before) return data["nbHits"], data def dump_range(username, created_after, created_before, seen_ids, all_records, depth=0): """Recursively bisect time ranges to stay under Algolia's 10k page wall.""" total, first_page = get_total_in_range(username, created_after, created_before) indent = " " * depth lo = created_after if created_after is not None else "start" hi = created_before if created_before is not None else "now" print(f"{indent}Range [{lo} .. {hi}]: {total} comments") if total == 0: return max_reachable = ALGOLIA_DEEP_PAGE_LIMIT # page*hitsPerPage cap if total <= max_reachable: # safe to page through directly page = 0 while True: data = first_page if page == 0 else fetch_page(username, page, created_after, created_before) hits = data["hits"] if not hits: break new = 0 for h in hits: if h["objectID"] not in seen_ids: seen_ids.add(h["objectID"]) all_records.append({ "id": h["objectID"], "author": h.get("author"), "created_at": h.get("created_at"), "created_at_i": h.get("created_at_i"), "comment_text": h.get("comment_text"), "story_id": h.get("story_id"), "story_title": h.get("story_title"), "story_url": h.get("story_url"), "parent_id": h.get("parent_id"), "points": h.get("points"), }) new += 1 print(f"{indent} page {page}: +{new} (running total {len(all_records)})") if (page + 1) * HITS_PER_PAGE >= data["nbHits"]: break page += 1 time.sleep(0.5) # polite pacing, Be a nice human else: # bisect on time: find midpoint of range and recurse both halves lo_i = created_after if created_after is not None else 0 # use "now" as a safe upper bound if open-ended hi_i = created_before if created_before is not None else int(time.time()) mid = (lo_i + hi_i) // 2 print(f"{indent} -> exceeds {max_reachable}, bisecting at {mid}") dump_range(username, lo_i, mid, seen_ids, all_records, depth + 1) dump_range(username, mid, hi_i, seen_ids, all_records, depth + 1) def main(): ap = argparse.ArgumentParser() ap.add_argument("username") args = ap.parse_args() username = args.username seen_ids = set() all_records = [] print(f"Fetching full comment history for '{username}'...") dump_range(username, None, None, seen_ids, all_records) all_records.sort(key=lambda r: r.get("created_at_i") or 0) json_out = f"hn_comments_{username}.json" csv_out = f"hn_comments_{username}.csv" with open(json_out, "w", encoding="utf-8") as f: json.dump(all_records, f, indent=2, ensure_ascii=False) if all_records: fieldnames = list(all_records[0].keys()) with open(csv_out, "w", newline="", encoding="utf-8") as f: writer = csv.DictWriter(f, fieldnames=fieldnames) writer.writeheader() writer.writerows(all_records) print(f"\nDone. {len(all_records)} total comments.") print(f" -> {json_out}") print(f" -> {csv_out}") if __name__ == "__main__": main()
  • Bender 3mo ago
    eth0up your last comment went auto-[dead] probably because of the python. I vouched for it, think that much worked. I would just grab it and add to a text file URL but python cares about spaces and the script is not consistently indented 4 spaces to make it into HN code. Do you by chance have a git repo you can save it to? Another option is to sftp it to hn@nochan.net (no password), put in /pub/ and I can turn it into a URL. If you can reach it.
    • cwmoore 3mo ago
      [flagged]
    • eth0up 3mo ago
      If you end up truly at a dead end, let me know and I'll go the sftp route. But if you would, please try dropping the rough paste into GPT, Claude, Gemini, etc, and just ask it to clean it up. I don't think there are any missing lines. If that fails let me know, but I generally avoid direct IP contact with strangers, and I am kind of not crazy about git and most things.
      • Bender 3mo ago
        It's ok, I was just trying to help cwmoore get the script but it looks like they already play with LLM's so I will let them figure it out from what you pasted. Thanks for contributing your script. [Edit] Here [1] you go. [1] - https://www.cloudflare.com/drop/ https://www.cloudflare.com/drop/