13 ms·

Archivists work to save disappearing data.gov datasets

801 points by johnneville 2y ago

notavalleyman 2y ago
I read, in past days, that the man who ordered the construction of the nearly infinite Wall of China was that First Emperor, Shih Huang Ti, who likewise ordered the burning of all the books before him. That the two gigantic operations - the five or six hundred leagues of stone to oppose the barbarians, the rigorous abolition of history, that is of the past - issued from one person and were in a certain sense his attributes, inexplicably satisfied me and, at the same time, disturbed me. - Borges
- itronitron 2y ago
  can't be "First Emperor" if it's known that someone else was in charge before you
  - zamadatix 2y ago
    Sure, they just used a different title.
  - o11c 2y ago
    Eh, the history doesn't support that meme; only the details and not the existence was allegedly burned (if it even happened at all). It's pretty well established that there were 7 major states (Qin, Qi, Chu, Yan, and the 3 Jin (Han, Wei, Zhao); notably, not including the official Zhou king) at the end of the Warring States period, and the Qin conquered the other 6 within a decade. (Fun fact: the lesser-known state of Wey actually lasted longer than Wei due to not being worth conquering. In modern Chinese they are pronounced the same, but at the time they weren't.)
- o11c 2y ago
  For reference (since it both uses a non-Pinyin transcription and adds/drops characters), this refers to https://en.wikipedia.org/wiki/Qin_Shi_Huang https://en.wikipedia.org/wiki/Qin_Shi_Huang
  - quuxplusone 2y ago
    Yes; and for further reference, the non-Pinyin transliteration is https://en.wikipedia.org/wiki/Wade%E2%80%93Giles https://en.wikipedia.org/wiki/Wade%E2%80%93Giles and the "add/drop" is that instead of the title 秦始皇 Qín Shǐ Huáng (lit. "first Qin emperor") Borges is using the title 始皇帝 Shǐ Huángdì (lit. "first emperor"). https://en.wikipedia.org/wiki/Shi%20Huangdi https://en.wikipedia.org/wiki/Shi%20Huangdi (redirects to Qin Shi Huang)
    - numpad0 2y ago
      It's astonishing how little information that phonetic representations of Chinese allow to be retained. Qin Shi Huang/Shih Huang Ti both means [Qin]Founding Emperor[-King]. In layperson's approximation, He's so ancient that his canonical name and title is debatable, and he's almost always referred to as "the first emperor" or variants thereof.
- belter 2y ago
  “Now I will tell you the answer to my question. It is this. The Party seeks power entirely for its own sake. We are not interested in the good of others; we are interested solely in power, pure power. What pure power means you will understand presently. We are different from the oligarchies of the past in that we know what we are doing. All the others, even those who resembled ourselves, were cowards and hypocrites. The German Nazis and the Russian Communists came very close to us in their methods, but they never had the courage to recognize their own motives. They pretended, perhaps they even believed, that they had seized power unwillingly and for a limited time, and that just around the corner there lay a paradise where human beings would be free and equal. We are not like that. We know that no one ever seizes power with the intention of relinquishing it. Power is not a means; it is an end. One does not establish a dictatorship in order to safeguard a revolution; one makes the revolution in order to establish the dictatorship. The object of persecution is persecution. The object of torture is torture. The object of power is power. Now you begin to understand me.” ― George Orwell, 1984
  - rightbyte 2y ago
    I think quotes like this should be attributed to the character (the secret police commissaire?) not the author?
    - matwood 2y ago
      Including 1984 in the cite makes it clear IMO. I referenced 1984 in comment this morning related to websites disappearing. We’ve (always|never) been at was with Eurasia…
      - rightbyte 2y ago
        Ye I get that but my main point is that it is not Orwell the real person that is quoted, but a fictional character that is a high level official in a dystopic police agency. If you haven't read the book it might be as Orwell the author is complicit. But sry might be a nitpick and obvious from context.
deleted 2y ago
[deleted]
debeloo 2y ago
Is this normal when there's change in presidency?
- BowBun 2y ago
  From the article: > Changes in presidential administrations have led to datasets being deleted in the past, either on purpose or by accident. When Biden took office, 1,000 datasets were deleted according to the Wayback Machine, via 404 Media's reporting.
  - derbOac 2y ago
    I think the question is the nature of the losses in the two cases, the transparency circumstances about them, and who exactly is making the decisions about specific datasets. Time will tell but loss of public datasets is probably not usually good in general.
    - doener 2y ago
      Yes, the context in which this happens could provide clues as to the nature of these losses: https://news.ycombinator.com/item?id=42898165 https://news.ycombinator.com/item?id=42898165
  - animal_spirits 2y ago
    This is not a direct quote, the actual quote from the article is > But archivists who have been working on analyzing the deletions and archiving the data it held say that while some of the deletions are surely malicious information scrubbing, some are likely routine artifacts of an administration change, and they are working to determine which is which. For example, in the days after Joe Biden was inaugurated, data.gov showed about 1,000 datasets being deleted as compared to a day before his inauguration, according to the Wayback Machine.
  - akudha 2y ago
    Why are the datasets deleted though? Biden or Trump, Democrats or Republicans - What do they gain?
smrtinsert 2y ago
Are datasets mirrored anywhere where the govt doesn't automatically have a take down authority? If not there should be a mirroring effort.
- lhl 2y ago
  There's been a lot of discussion in https://www.reddit.com/r/DataHoarder/ https://www.reddit.com/r/DataHoarder/ Here's documentation on independent backup efforts of various government websites: https://www.reddit.com/r/DataHoarder/comments/1ifalwe/us_gov_ftp_and_http_file_servers/ https://www.reddit.com/r/DataHoarder/comments/1ifalwe/us_gov... Also here: https://www.reddit.com/r/DataHoarder/comments/1idj6dm/all_us_federal_government_websites_are_already/ https://www.reddit.com/r/DataHoarder/comments/1idj6dm/all_us... Apparently, much of the data has been back up here: https://eotarchive.org/ https://eotarchive.org/ Here's also a discussion on whether the Internet Archive is sufficiently backed up/decentralized (it is not): https://www.reddit.com/r/DataHoarder/comments/1if32iq/does_internet_archive_have_any_plans_to_move/ https://www.reddit.com/r/DataHoarder/comments/1if32iq/does_i...
  - deleted 2y ago
    [deleted]
sunk1st 2y ago
I don’t see a list of the datasets that have gone missing. Is there a list?
- mistrial9 2y ago
  [flagged]
  - NortySpock 2y ago
    And you could also run your own archive bot (x86 only). I've got one running in a docker container, it downloads a webpage and auto-uploads it to archive.org https://tracker.archiveteam.org/ https://tracker.archiveteam.org/ Edit to add: docker_compose.yml example: services: archiveteam: image: atdr.meo.ws/archiveteam/warrior-dockerfile ports: - '8101:8001' mem_limit: 4G cpus: 3 dns: - 9.9.9.10 - 8.8.8.8 labels: - com.centurylinklabs.watchtower.enable=true container_name: archiveteam-warrior environment: - DOWNLOADER=asdf # Change this to your nickname - SELECTED_PROJECT=auto # Change this to your project of preference or let the archiveteam decide with 'auto' - CONCURRENT_ITEMS=6 # Change this to the amount of concurrent download threads you can handle watchtower: command: '--label-enable --include-restarting --cleanup --interval 3600' cpu_shares: 128 mem_limit: 1G cpus: 1 image: containrrr/watchtower volumes: - '/var/run/docker.sock:/var/run/docker.sock' container_name: watchtower
jl6 2y ago
> The outlet reports that deleted datasets "disproportionately" come from environmental science agencies like the Department of Energy, National Oceanic and Atmospheric Administration (NOAA), and the Environmental Protection Agency (EPA). Was there an EO targeting these areas?
- bilbo0s 2y ago
  Don’t worry, it is a matter of great doctrinal import that all scientific datasets be replaced with datasets that have been properly refined in accordance with scripture. /s Maybe this administration will get better over time?
  - uni_rule 2y ago
    Nah, the whole executive branch is getting Jack Welch'ed. Hopefully your tap water cleanliness regulations are strong on a state level.
    - BeefWellington 2y ago
      After the utter bullshit pulled in California, better hope your state is willing to defend its water reservoirs or for some places clean tap water may be the least of their problems.
- deleted 2y ago
  [deleted]
- johnneville 2y ago
  Politico reports that USDA landing pages regarding climate change were ordered to be deleted by a directive from the USDA's office of communications. I think it is likely that orders to these other agencies follows this model. Many other datasets are being targeted via EO 14168 which has quite wide impacts but doesn't appear at first glance to apply to what i would expect to be a part of NOAA and EPA reports. https://www.politico.com/news/2025/01/31/usda-climate-change-websites-00201826 https://www.politico.com/news/2025/01/31/usda-climate-change...
exe34 2y ago
First week we had mass deportation, second week we've heard of the building of concentration camps for undesirables, and now the modern version of book burning. There's something different about this republican government.
- southernplaces7 2y ago
  >second week we've heard of the building of concentration camps for undesirables Mind posting a reference for such a declaration?
  - johnneville 2y ago
    I assume they are referring to this: https://www.whitehouse.gov/presidential-actions/2025/01/expanding-migrant-operations-center-at-naval-station-guantanamo-bay-to-full-capacity/ https://www.whitehouse.gov/presidential-actions/2025/01/expa...
    - southernplaces7 2y ago
      I assumed they were referring to something like this but wanted to be sure. It's also way the fuck off base to call this a concentration camp for undesirables with all the insinuations of Nazism that come with that. This kind of Trump = Hilter hyperbole at the drop of a hat hazes all the very real and heavy criticisms that can be made of this administration. Gitmo being used as a transit camp for deported illegal immigrants is not at all as say, Sachsenhausen or Dachau being used to illegally dumb german citizens and torture them there indefinitely because they criticized the ruling regime. When Trump starts trying to dump opposing intellectuals, Hollywood producers and "wayward" reporters or writers into Gitmo for indefinite detention, then we can make some firmer claims of literal dictatorship. If anything, the use of Gitmo under Bush II, when actual U.S citizens were held there indefinitely under terrorism charges was closer to the mark than this now.
      - misnome 2y ago
        Did the Nazi program start at those levels? Or did it start with exactly this rhetoric and excuses?
cle 2y ago
I’ve been archiving data.gov for over a year now and it’s not unusual to see large fluctuations on the order of hundreds or thousands of datasets. I’ve never bothered trying to figure out what exactly is changing, maybe I should build a tool for that…
- nemofoo 2y ago
  Do you mirror these data sets anywhere?
  - cle 2y ago
    It's not in any sort of format to do this kind of analysis unfortunately. I'm also missing some data b/c I throw away certain kinds of datasets that are not useful for me. I can probably write some scripts to diff my archives with the current data.gov and see what's missing, but it won't be "complete". But it might still be useful... I did however just write a Python script to pull data.gov from archive.org and check the dataset count on the front page for all of 2024, here are the results: https://postimg.cc/1V0WYtRt https://postimg.cc/1V0WYtRt As you can see, there were multiple drops on the order of ~10,000 during 2024. So it's not that unusual. There could be something bad going on right now, but just from the numbers I can't conclude that yet. (Specifically it takes the first snapshot of every Wednesday of 2024). If I get around to re-formatting my archives this week, I'll follow up on HN :).
deleted 2y ago
[deleted]
pluto_modadic 2y ago
don't they have to have to have done this /before/ it gets deleted?
dang 2y ago
Related ongoing thread: CDC data are disappearing - https://news.ycombinator.com/item?id=42897696 https://news.ycombinator.com/item?id=42897696 - Feb 2025 (216 comments)
chrishoyle 2y ago
Related ongoing discussion The government information crisis is bigger than you think it is - https://news.ycombinator.com/item?id=42895331 https://news.ycombinator.com/item?id=42895331
JackC 2y ago
I'm quoted in this article. Happy to discuss what we're working on at the Library Innovation Lab if anyone has questions. There's lots of people making copies of things right now, which is great -- Lots Of Copies Keeps Stuff Safe. It's your data, why not have a copy? One thing I think we can contribute here as an institution is timestamping and provenance. Our copy of data.gov is made with https://github.com/harvard-lil/bag-nabit https://github.com/harvard-lil/bag-nabit , which extends BagIt format to sign archives with email/domain/document certificates. That way (once we have a public endpoint) you can make your own copy with rclone, pass it around, but still verify it hasn't been modified since we made it. Some open questions we'd love help on -- * One is that it's hard to tell what's disappearing and what's just moving. If you do a raw comparison of snapshots, there's things like 2011-glass-buttes-exploration-and-drilling-535cf being replaced by 2011-glass-buttes-exploration-and-drilling-236cf, but it's still exactly the same data; it's a rename rather than a delete and add. We need some data munging to work out what's actually changing. * Another is how to find the most valuable things to preserve that aren't directly linked from the catalog. If a data.gov entry links to a csv, we have it. If it links to an html landing page, we have the landing page. It would be great to do some analysis to figure out the most valuable stuff behind the landing pages.
- jszymborski 2y ago
  Hi! Is there any one place that would be easiest for folks to grab these snapshots from? Would love to try my hand at finding documents that moved/documents that were removed.
  - JackC 2y ago
    Hmm, I can put them here for now: https://source.coop/harvard-lil/data-gov-metadata https://source.coop/harvard-lil/data-gov-metadata Unfortunately it's a bit messy because we weren't initially thinking about tracking deletions. data_20241119.jsonl.zip (301k rows) and data_20250130.jsonl.zip (305k rows) are simple captures of the API on those dates. data_db_dump_20250130.jsonl.zip (311k rows) is a sqlite dump of all the entries we saw at some point between those dates. My hunch is there's something like 4,000 false positives and 2,000 deletions between the 311k and 305k set, but that could be way off.
crowcroft 2y ago
Still, even with best efforts this is such a shame. There is always going to be a question around governance over the data, integrity, and potentially chain of custody as well. If the goal is to muddy the waters and create a narrative that whatever might be in this data isn't reliable or accurate then mission accomplished. I don't see how anything can stop that. Not to say the data isn't incredibly valuable and should be preserved for many other reasons of course. All the best to anyone archiving this, this is important work.
0n0n0m0uz 2y ago
One of the USA greatest strengths is the almost unprecedented degree of transparency of governments records going back decades. We can actually see the true facts including when our government has lied to us or covered things up. Many other nations do not have this luxury and it has provided the evidentiary basis for both legal cases and "progress" in general. Not surprising that authoritarians would target and destroy data as it makes their objective of a post-truth society that much easier
- bamboozled 2y ago
  It’s also the driver for a great economy imo. Why can’t Russia and China build and innovate like the USA can? Because they spend a lot of time censoring and covering things up. RIP USA.
  - stackedinserter 2y ago
    > Why can’t Russia and China build and innovate like the USA can? > Because they spend a lot of time censoring and covering things up. What a statement! How did you come to this conclusion?
    - bamboozled 2y ago
      Who built all Russia’s oil infrastructure ? Who has the strongest economy? Who builds the airliners ? Has the most advanced space programs ? Landed men on the moon ? Etc ? It’s not just America either. Europe too. Australia. Authoritarians copy and steal. Democracies innovate and adapt. Democracies are more transparent and honest, which leads to better outcomes in all areas. Time spent hiding information to appease the dictator and protecting “the party” is time wasted.
      - achierius 2y ago
        > Who built all Russia's oil infrastructure ? The Soviet Union? Yeah, the new offshore stuff (mostly? developed after the fall of the USSR) relied on western firms, but I believe the bulk of oil production still comes from / goes through domestically sourced systems. Gas production on the other hand was initially almost entirely built out by western firms, though that's obviously begun to change since the war in Ukraine.
eh_why_not 2y ago
What's a good way to be an "Archivist" on a low budget these days? Say you have a few TBs of disk space, and you're willing to capture some public datasets (or parts of them) that interest you, and publish them in a friendly jurisdiction - keyed by their MD5/SHA1 - or make them available upon request. I.e. be part of a large open-source storage network, but only for objects/datasets you're willing to store (so there are no illegal shenanigans). Is this a use case for Torrents? What's the most suitable architecture available today for this?
- josh-sematic 2y ago
  I’m not an expert in such things, but this seems like a good use case for IPFS. Kinda similar to a torrent except that it is natively content-addressed (essentially the key to access is a hash of the data).
- honestSysAdmin 2y ago
  In my experience, to archive effectively you need a physical datacenter footprint, or to rent capacity of someone who does. Over a longer timespan (even just 6 months), having your own footprint is a lower total cost of ownership, provided you have the skills or access to someone with the skills to run Kubernetes + Ceph (or something similar). . > Is this a use case for Torrents? Yes, provided you have a good way to dynamically append a distributed index of torrents and users willing to run that software in addition to the torrent software. Should be easy enough to define in container-compose.
- smallerize 2y ago
  https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem Set up a scrape using ArchiveTeam's fork of wget. It can save all the requests and responses into a single WARC file. Then you can use https://replayweb.page/ https://replayweb.page/ or some other tool to browse the contents.
choobacker 2y ago
It's impressive that volunteers are stepping up to archive this. I understand the desire to keep this open data available. How much of this sort of effort results in that data being used? Are there success stories for these datasets being discoverable enough and useful to others?
chrishoyle 2y ago
Beyond federal websites (.gov, .mil) there are lot of gov contractor websites that are being taken down (presumably at the demand of agencies) that contain a wealth of information and years of project research. Some below of contractors that work with US AID: - https://www.edu-links.org/ https://www.edu-links.org/ (taken down) - https://www.genderlinks.org/ https://www.genderlinks.org/ (taken down) - https://usaidlearninglab.org/ https://usaidlearninglab.org/ (taken down) - https://agrilinks.org/ https://agrilinks.org/ (presumably at risk) - https://www.climatelinks.org/ https://www.climatelinks.org/ (presumably at risk) - https://biodiversitylinks.org/ https://biodiversitylinks.org/ (presumably at risk)
- honestSysAdmin 2y ago
  The pedestrian "right", which I encounter on a day-to-day basis the months I visit client sites a couple hundred miles inland of the Gulf of America, will look at climatelinks.org and say something like: "all I see are foreign countries, why are we spending money on this instead of citizens of the United States?".
  - cscurmudgeon 2y ago
    [flagged]
    - harimau777 2y ago
      IMHO, we should do it because the person who pays tends to have more power over what happens. Just like how in high school the kid who drives everyone tends to have a higher than normal say in what the friend group does.
      - honestSysAdmin 2y ago
        Smart.
    - kergonath 2y ago
      > Why should US fund WHO ~5-6 times more than China [0] (and more than EU) The base contributions are a function of GDP. The extra contributions are voluntary, and the US did it because it was in the US’ interests. It’s a founding error in the US foreign policy budget and was a good investment in terms of goodwill and data for American health research institutions. WHO must focus where it is needed most. Public health is much better in the EU (and even in Europe, accounting for places like Belarus and Ukraine) than in China, and there are much fewer epidemics that emerge in Europe in general. The whole idea is that if we limit the emergence of epidemics where they are likely to happen, we end up with fewer pandemics after these epidemics spread worldwide (which includes Europe and North America). The whole world is better without another COVID, Ebola, or Polio. > only to have the WHO be controlled by China This is bullshit. The WHO is not controlled by China any more than other UN institutions. What is certain, though, is that the US won’t have any say whatsoever once they are out.
derektank 2y ago
Does anyone know if the St Louis Federal Reserve (and I guess the federal reserve banks generally) is subject to presidential executive orders or is it entirely responsible to the Federal Reserve Board and the St. Louis Bank president? FRED is the only dataset I access regularly
bawolff 2y ago
Tbh, im kind of surprised these things weren't being archived as they were being published. Trump is an extreme case, but its not the first time a change in administration resulted in removing old websites.
strictnein 2y ago
99.9% of commenters here seem to have missed this: > For example, in the days after Joe Biden was inaugurated, data.gov showed about 1,000 datasets being deleted as compared to a day before his inauguration It's almost like this stuff happens regularly. If <insert Dem savior> wins in 2028, tons of government websites will also change in the first couple weeks of their presidency. Is it because they're a fascist dictator? Or is it because those websites reflect the administration's viewpoints on issues? Wish people would take a deep breath and step back and think a little more. I despise Trump, but there's crying wolf and then there's the current state of media and online discourse. Trump thrives in this type of environment. He purposefully fosters it. Playing gotcha with him doesn't work because he doesn't care.
- BeetleB 2y ago
  Plenty of comments older than yours are pointing this out. A bit hyperbolic to claim 99.9%.
- blindriver 2y ago
  The depth to which the media can whip people up into a hysteria is truly striking. I think most people that voted for Trump like myself aren’t buying into the propaganda and will rather wait for the dust to settle. All I’m seeing is a president that is doing exactly what he promised, so I’m very much in the “let the chips fall where they may” category.
generalizations 2y ago
Do we know what datasets these are? Do we actually have a diff here so we know what's been removed? There's a lot of assumptions being thrown around here, but we don't even know if this is some kind of malicious compliance. An actual list of what's been removed would probably clear the air a lot. As one of the reddit comments (in the thread linked by the article) pointed out, > During the start of Biden’s term, On 6th feb data.gov had “218,384 DATASETS” but on 7th feb it only had “192,180 DATASETS”
honestSysAdmin 2y ago
Let's make torrents and seed them.
- bamboozled 2y ago
  They might come after you for hosting it ?
  - bbwbsb 2y ago
    i2p could be an option
tmshapland 2y ago
[flagged]
- p3rls 2y ago
  really looking forward to four more years of astute political commentary like this
  - gopher_space 2y ago
    For all of the downsides to a Trump administration, one of the unexpected upsides is that the people who don't have a problem with Nazis will self-identify. I feel really bad for the kids who've developed an alt-right persona during the pandemic, though. There used to be time to work that out for yourself, but now I think social media will provide the kiss of death for many career ambitions.
  - pessimizer 2y ago
    They can't keep it up for another decade, the financing is running out. Anti-Trump hysteria was largely financed by actors who dominated establishment Republican and Democratic corruption networks, but had no connection to Trump (who's a real estate guy, which is like the ghetto of finance.) During that decade, and especially during the four years he was out of office, a most of them have solidly established relationships with Trump and his network, especially tech guys. One day, upper-middle class people will stop seeing as brave and insightful calling half the country Nazis for not conforming to their suburban aesthetic. They've even eventually stopped progressively describing him as the submissive homosexual partner to Putin, I suppose because we've run out of fighting aged adult males in Ukraine so there's no reason to care about that any more. If Trump starts a war with Iran, they'll start praising him as the new Lincoln. It's not real. The only weird part is that the people attacking Trump for censorship that hasn't happened are delighted by censorship. They think the main problem with the world is that too many people are allowed to speak about whatever they want.
    - mpalmer 2y ago
      > calling half the country Nazis for not conforming to their suburban aesthetic > attacking Trump for censorship that hasn't happened Federal funds are being withheld - illegally, in most cases - from any entity that supports or even acknowledges what the administration calls "radical gender ideology". It's totalitarian and regressive. Nazi's a fine word for what the leaders of the party are showing themselves to be. I wouldn't call half the country Nazis - unless they chose to stand with the human trash in the White House. Then I wouldn't really have a problem with it.
andyjohnson0 2y ago
If the intention is to restore these data sets at some future date, when sanity has possibly been restored, then there needs to be a way to demonstrate that the archived data hasn't itself been modified. Without that, malign actors (e.g. oil/gas lobby) could very easily poison the future.
- DJBunnies 2y ago
  If only we had checksums or file hash integrity.
  - andyjohnson0 2y ago
    If the originals are gone, and if the original authoritative, trustworthy data custodian no longer exists, then how will we know what the checksum/hash should be?
downrightmike 2y ago
Already seeing: 404 Not Found: Requested route ('ed-public-download.app.cloud.gov') does not exist.
liontwist 2y ago
I think people are interested in archiving and the political image associated with that but I don’t think anybody cares about the content. Who is going to go back and read Biden era agency publications?
- acureau 2y ago
  Historians, political scientists, and social scientists come to mind
  - deleted 2y ago
    [deleted]
Slava_Propanei 2y ago
[dead]
ThinkBeat 2y ago
I hope volunteers and others are able to save as much as possible of the data. Removing and altering of the information and data is one of the fundamental threats in our digital world. It probably makes the most sense to do this on a daily basis. If something new, if published, grab it as soon as possible. Data can also be redacted or altered for a variety of reasons, being able to see the before and after states can be illuminating. Something I feel is missing here are statistics for each administration. Does this only happen under a Trump administration, or does it happen to smaller or larger extent under other administrations? I don't know how far back this federal goes so it might not be easy,

801 points by johnneville 2y ago

I read, in past days, that the man who ordered the construction of the nearly infinite Wall of China was that First Emperor, Shih Huang Ti, who likewise ordered the burning of all the books before him. That the two gigantic operations - the five or six hundred leagues of stone to oppose the barbarians, the rigorous abolition of history, that is of the past - issued from one person and were in a certain sense his attributes, inexplicably satisfied me and, at the same time, disturbed me. - Borges

itronitron 2y ago
can't be "First Emperor" if it's known that someone else was in charge before you
- zamadatix 2y ago
  Sure, they just used a different title.
- o11c 2y ago
  Eh, the history doesn't support that meme; only the details and not the existence was allegedly burned (if it even happened at all). It's pretty well established that there were 7 major states (Qin, Qi, Chu, Yan, and the 3 Jin (Han, Wei, Zhao); notably, not including the official Zhou king) at the end of the Warring States period, and the Qin conquered the other 6 within a decade. (Fun fact: the lesser-known state of Wey actually lasted longer than Wei due to not being worth conquering. In modern Chinese they are pronounced the same, but at the time they weren't.)
o11c 2y ago
For reference (since it both uses a non-Pinyin transcription and adds/drops characters), this refers to https://en.wikipedia.org/wiki/Qin_Shi_Huang https://en.wikipedia.org/wiki/Qin_Shi_Huang
- quuxplusone 2y ago
  Yes; and for further reference, the non-Pinyin transliteration is https://en.wikipedia.org/wiki/Wade%E2%80%93Giles https://en.wikipedia.org/wiki/Wade%E2%80%93Giles and the "add/drop" is that instead of the title 秦始皇 Qín Shǐ Huáng (lit. "first Qin emperor") Borges is using the title 始皇帝 Shǐ Huángdì (lit. "first emperor"). https://en.wikipedia.org/wiki/Shi%20Huangdi https://en.wikipedia.org/wiki/Shi%20Huangdi (redirects to Qin Shi Huang)
  - numpad0 2y ago
    It's astonishing how little information that phonetic representations of Chinese allow to be retained. Qin Shi Huang/Shih Huang Ti both means [Qin]Founding Emperor[-King]. In layperson's approximation, He's so ancient that his canonical name and title is debatable, and he's almost always referred to as "the first emperor" or variants thereof.
belter 2y ago
“Now I will tell you the answer to my question. It is this. The Party seeks power entirely for its own sake. We are not interested in the good of others; we are interested solely in power, pure power. What pure power means you will understand presently. We are different from the oligarchies of the past in that we know what we are doing. All the others, even those who resembled ourselves, were cowards and hypocrites. The German Nazis and the Russian Communists came very close to us in their methods, but they never had the courage to recognize their own motives. They pretended, perhaps they even believed, that they had seized power unwillingly and for a limited time, and that just around the corner there lay a paradise where human beings would be free and equal. We are not like that. We know that no one ever seizes power with the intention of relinquishing it. Power is not a means; it is an end. One does not establish a dictatorship in order to safeguard a revolution; one makes the revolution in order to establish the dictatorship. The object of persecution is persecution. The object of torture is torture. The object of power is power. Now you begin to understand me.” ― George Orwell, 1984
- rightbyte 2y ago
  I think quotes like this should be attributed to the character (the secret police commissaire?) not the author?
  - matwood 2y ago
    Including 1984 in the cite makes it clear IMO. I referenced 1984 in comment this morning related to websites disappearing. We’ve (always|never) been at was with Eurasia…
    - rightbyte 2y ago
      Ye I get that but my main point is that it is not Orwell the real person that is quoted, but a fictional character that is a high level official in a dystopic police agency. If you haven't read the book it might be as Orwell the author is complicit. But sry might be a nitpick and obvious from context.

deleted 2y ago

[deleted]

debeloo 2y ago

Is this normal when there's change in presidency?

BowBun 2y ago
From the article: > Changes in presidential administrations have led to datasets being deleted in the past, either on purpose or by accident. When Biden took office, 1,000 datasets were deleted according to the Wayback Machine, via 404 Media's reporting.
- derbOac 2y ago
  I think the question is the nature of the losses in the two cases, the transparency circumstances about them, and who exactly is making the decisions about specific datasets. Time will tell but loss of public datasets is probably not usually good in general.
  - doener 2y ago
    Yes, the context in which this happens could provide clues as to the nature of these losses: https://news.ycombinator.com/item?id=42898165 https://news.ycombinator.com/item?id=42898165
- animal_spirits 2y ago
  This is not a direct quote, the actual quote from the article is > But archivists who have been working on analyzing the deletions and archiving the data it held say that while some of the deletions are surely malicious information scrubbing, some are likely routine artifacts of an administration change, and they are working to determine which is which. For example, in the days after Joe Biden was inaugurated, data.gov showed about 1,000 datasets being deleted as compared to a day before his inauguration, according to the Wayback Machine.
- akudha 2y ago
  Why are the datasets deleted though? Biden or Trump, Democrats or Republicans - What do they gain?

smrtinsert 2y ago

Are datasets mirrored anywhere where the govt doesn't automatically have a take down authority? If not there should be a mirroring effort.

lhl 2y ago
There's been a lot of discussion in https://www.reddit.com/r/DataHoarder/ https://www.reddit.com/r/DataHoarder/ Here's documentation on independent backup efforts of various government websites: https://www.reddit.com/r/DataHoarder/comments/1ifalwe/us_gov_ftp_and_http_file_servers/ https://www.reddit.com/r/DataHoarder/comments/1ifalwe/us_gov... Also here: https://www.reddit.com/r/DataHoarder/comments/1idj6dm/all_us_federal_government_websites_are_already/ https://www.reddit.com/r/DataHoarder/comments/1idj6dm/all_us... Apparently, much of the data has been back up here: https://eotarchive.org/ https://eotarchive.org/ Here's also a discussion on whether the Internet Archive is sufficiently backed up/decentralized (it is not): https://www.reddit.com/r/DataHoarder/comments/1if32iq/does_internet_archive_have_any_plans_to_move/ https://www.reddit.com/r/DataHoarder/comments/1if32iq/does_i...
- deleted 2y ago
  [deleted]

sunk1st 2y ago

I don’t see a list of the datasets that have gone missing. Is there a list?

mistrial9 2y ago
[flagged]
- NortySpock 2y ago
  And you could also run your own archive bot (x86 only). I've got one running in a docker container, it downloads a webpage and auto-uploads it to archive.org https://tracker.archiveteam.org/ https://tracker.archiveteam.org/ Edit to add: docker_compose.yml example: services: archiveteam: image: atdr.meo.ws/archiveteam/warrior-dockerfile ports: - '8101:8001' mem_limit: 4G cpus: 3 dns: - 9.9.9.10 - 8.8.8.8 labels: - com.centurylinklabs.watchtower.enable=true container_name: archiveteam-warrior environment: - DOWNLOADER=asdf # Change this to your nickname - SELECTED_PROJECT=auto # Change this to your project of preference or let the archiveteam decide with 'auto' - CONCURRENT_ITEMS=6 # Change this to the amount of concurrent download threads you can handle watchtower: command: '--label-enable --include-restarting --cleanup --interval 3600' cpu_shares: 128 mem_limit: 1G cpus: 1 image: containrrr/watchtower volumes: - '/var/run/docker.sock:/var/run/docker.sock' container_name: watchtower

jl6 2y ago

> The outlet reports that deleted datasets "disproportionately" come from environmental science agencies like the Department of Energy, National Oceanic and Atmospheric Administration (NOAA), and the Environmental Protection Agency (EPA). Was there an EO targeting these areas?

bilbo0s 2y ago
Don’t worry, it is a matter of great doctrinal import that all scientific datasets be replaced with datasets that have been properly refined in accordance with scripture. /s Maybe this administration will get better over time?
- uni_rule 2y ago
  Nah, the whole executive branch is getting Jack Welch'ed. Hopefully your tap water cleanliness regulations are strong on a state level.
  - BeefWellington 2y ago
    After the utter bullshit pulled in California, better hope your state is willing to defend its water reservoirs or for some places clean tap water may be the least of their problems.
deleted 2y ago
[deleted]
johnneville 2y ago
Politico reports that USDA landing pages regarding climate change were ordered to be deleted by a directive from the USDA's office of communications. I think it is likely that orders to these other agencies follows this model. Many other datasets are being targeted via EO 14168 which has quite wide impacts but doesn't appear at first glance to apply to what i would expect to be a part of NOAA and EPA reports. https://www.politico.com/news/2025/01/31/usda-climate-change-websites-00201826 https://www.politico.com/news/2025/01/31/usda-climate-change...

exe34 2y ago

First week we had mass deportation, second week we've heard of the building of concentration camps for undesirables, and now the modern version of book burning. There's something different about this republican government.

southernplaces7 2y ago
>second week we've heard of the building of concentration camps for undesirables Mind posting a reference for such a declaration?
- johnneville 2y ago
  I assume they are referring to this: https://www.whitehouse.gov/presidential-actions/2025/01/expanding-migrant-operations-center-at-naval-station-guantanamo-bay-to-full-capacity/ https://www.whitehouse.gov/presidential-actions/2025/01/expa...
  - southernplaces7 2y ago
    I assumed they were referring to something like this but wanted to be sure. It's also way the fuck off base to call this a concentration camp for undesirables with all the insinuations of Nazism that come with that. This kind of Trump = Hilter hyperbole at the drop of a hat hazes all the very real and heavy criticisms that can be made of this administration. Gitmo being used as a transit camp for deported illegal immigrants is not at all as say, Sachsenhausen or Dachau being used to illegally dumb german citizens and torture them there indefinitely because they criticized the ruling regime. When Trump starts trying to dump opposing intellectuals, Hollywood producers and "wayward" reporters or writers into Gitmo for indefinite detention, then we can make some firmer claims of literal dictatorship. If anything, the use of Gitmo under Bush II, when actual U.S citizens were held there indefinitely under terrorism charges was closer to the mark than this now.
    - misnome 2y ago
      Did the Nazi program start at those levels? Or did it start with exactly this rhetoric and excuses?

cle 2y ago

I’ve been archiving data.gov for over a year now and it’s not unusual to see large fluctuations on the order of hundreds or thousands of datasets. I’ve never bothered trying to figure out what exactly is changing, maybe I should build a tool for that…

nemofoo 2y ago
Do you mirror these data sets anywhere?
- cle 2y ago
  It's not in any sort of format to do this kind of analysis unfortunately. I'm also missing some data b/c I throw away certain kinds of datasets that are not useful for me. I can probably write some scripts to diff my archives with the current data.gov and see what's missing, but it won't be "complete". But it might still be useful... I did however just write a Python script to pull data.gov from archive.org and check the dataset count on the front page for all of 2024, here are the results: https://postimg.cc/1V0WYtRt https://postimg.cc/1V0WYtRt As you can see, there were multiple drops on the order of ~10,000 during 2024. So it's not that unusual. There could be something bad going on right now, but just from the numbers I can't conclude that yet. (Specifically it takes the first snapshot of every Wednesday of 2024). If I get around to re-formatting my archives this week, I'll follow up on HN :).

deleted 2y ago

[deleted]

pluto_modadic 2y ago

don't they have to have to have done this /before/ it gets deleted?

dang 2y ago

Related ongoing thread: CDC data are disappearing - https://news.ycombinator.com/item?id=42897696 https://news.ycombinator.com/item?id=42897696 - Feb 2025 (216 comments)

chrishoyle 2y ago

Related ongoing discussion The government information crisis is bigger than you think it is - https://news.ycombinator.com/item?id=42895331 https://news.ycombinator.com/item?id=42895331

JackC 2y ago

I'm quoted in this article. Happy to discuss what we're working on at the Library Innovation Lab if anyone has questions. There's lots of people making copies of things right now, which is great -- Lots Of Copies Keeps Stuff Safe. It's your data, why not have a copy? One thing I think we can contribute here as an institution is timestamping and provenance. Our copy of data.gov is made with https://github.com/harvard-lil/bag-nabit https://github.com/harvard-lil/bag-nabit , which extends BagIt format to sign archives with email/domain/document certificates. That way (once we have a public endpoint) you can make your own copy with rclone, pass it around, but still verify it hasn't been modified since we made it. Some open questions we'd love help on -- * One is that it's hard to tell what's disappearing and what's just moving. If you do a raw comparison of snapshots, there's things like 2011-glass-buttes-exploration-and-drilling-535cf being replaced by 2011-glass-buttes-exploration-and-drilling-236cf, but it's still exactly the same data; it's a rename rather than a delete and add. We need some data munging to work out what's actually changing. * Another is how to find the most valuable things to preserve that aren't directly linked from the catalog. If a data.gov entry links to a csv, we have it. If it links to an html landing page, we have the landing page. It would be great to do some analysis to figure out the most valuable stuff behind the landing pages.

jszymborski 2y ago
Hi! Is there any one place that would be easiest for folks to grab these snapshots from? Would love to try my hand at finding documents that moved/documents that were removed.
- JackC 2y ago
  Hmm, I can put them here for now: https://source.coop/harvard-lil/data-gov-metadata https://source.coop/harvard-lil/data-gov-metadata Unfortunately it's a bit messy because we weren't initially thinking about tracking deletions. data_20241119.jsonl.zip (301k rows) and data_20250130.jsonl.zip (305k rows) are simple captures of the API on those dates. data_db_dump_20250130.jsonl.zip (311k rows) is a sqlite dump of all the entries we saw at some point between those dates. My hunch is there's something like 4,000 false positives and 2,000 deletions between the 311k and 305k set, but that could be way off.

crowcroft 2y ago

Still, even with best efforts this is such a shame. There is always going to be a question around governance over the data, integrity, and potentially chain of custody as well. If the goal is to muddy the waters and create a narrative that whatever might be in this data isn't reliable or accurate then mission accomplished. I don't see how anything can stop that. Not to say the data isn't incredibly valuable and should be preserved for many other reasons of course. All the best to anyone archiving this, this is important work.

0n0n0m0uz 2y ago

One of the USA greatest strengths is the almost unprecedented degree of transparency of governments records going back decades. We can actually see the true facts including when our government has lied to us or covered things up. Many other nations do not have this luxury and it has provided the evidentiary basis for both legal cases and "progress" in general. Not surprising that authoritarians would target and destroy data as it makes their objective of a post-truth society that much easier

bamboozled 2y ago
It’s also the driver for a great economy imo. Why can’t Russia and China build and innovate like the USA can? Because they spend a lot of time censoring and covering things up. RIP USA.
- stackedinserter 2y ago
  > Why can’t Russia and China build and innovate like the USA can? > Because they spend a lot of time censoring and covering things up. What a statement! How did you come to this conclusion?
  - bamboozled 2y ago
    Who built all Russia’s oil infrastructure ? Who has the strongest economy? Who builds the airliners ? Has the most advanced space programs ? Landed men on the moon ? Etc ? It’s not just America either. Europe too. Australia. Authoritarians copy and steal. Democracies innovate and adapt. Democracies are more transparent and honest, which leads to better outcomes in all areas. Time spent hiding information to appease the dictator and protecting “the party” is time wasted.
    - achierius 2y ago
      > Who built all Russia's oil infrastructure ? The Soviet Union? Yeah, the new offshore stuff (mostly? developed after the fall of the USSR) relied on western firms, but I believe the bulk of oil production still comes from / goes through domestically sourced systems. Gas production on the other hand was initially almost entirely built out by western firms, though that's obviously begun to change since the war in Ukraine.

eh_why_not 2y ago

What's a good way to be an "Archivist" on a low budget these days? Say you have a few TBs of disk space, and you're willing to capture some public datasets (or parts of them) that interest you, and publish them in a friendly jurisdiction - keyed by their MD5/SHA1 - or make them available upon request. I.e. be part of a large open-source storage network, but only for objects/datasets you're willing to store (so there are no illegal shenanigans). Is this a use case for Torrents? What's the most suitable architecture available today for this?

josh-sematic 2y ago
I’m not an expert in such things, but this seems like a good use case for IPFS. Kinda similar to a torrent except that it is natively content-addressed (essentially the key to access is a hash of the data).
honestSysAdmin 2y ago
In my experience, to archive effectively you need a physical datacenter footprint, or to rent capacity of someone who does. Over a longer timespan (even just 6 months), having your own footprint is a lower total cost of ownership, provided you have the skills or access to someone with the skills to run Kubernetes + Ceph (or something similar). . > Is this a use case for Torrents? Yes, provided you have a good way to dynamically append a distributed index of torrents and users willing to run that software in addition to the torrent software. Should be easy enough to define in container-compose.
smallerize 2y ago
https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem Set up a scrape using ArchiveTeam's fork of wget. It can save all the requests and responses into a single WARC file. Then you can use https://replayweb.page/ https://replayweb.page/ or some other tool to browse the contents.

choobacker 2y ago

It's impressive that volunteers are stepping up to archive this. I understand the desire to keep this open data available. How much of this sort of effort results in that data being used? Are there success stories for these datasets being discoverable enough and useful to others?

chrishoyle 2y ago

Beyond federal websites (.gov, .mil) there are lot of gov contractor websites that are being taken down (presumably at the demand of agencies) that contain a wealth of information and years of project research. Some below of contractors that work with US AID: - https://www.edu-links.org/ https://www.edu-links.org/ (taken down) - https://www.genderlinks.org/ https://www.genderlinks.org/ (taken down) - https://usaidlearninglab.org/ https://usaidlearninglab.org/ (taken down) - https://agrilinks.org/ https://agrilinks.org/ (presumably at risk) - https://www.climatelinks.org/ https://www.climatelinks.org/ (presumably at risk) - https://biodiversitylinks.org/ https://biodiversitylinks.org/ (presumably at risk)

honestSysAdmin 2y ago
The pedestrian "right", which I encounter on a day-to-day basis the months I visit client sites a couple hundred miles inland of the Gulf of America, will look at climatelinks.org and say something like: "all I see are foreign countries, why are we spending money on this instead of citizens of the United States?".
- cscurmudgeon 2y ago
  [flagged]
  - harimau777 2y ago
    IMHO, we should do it because the person who pays tends to have more power over what happens. Just like how in high school the kid who drives everyone tends to have a higher than normal say in what the friend group does.
    - honestSysAdmin 2y ago
      Smart.
  - kergonath 2y ago
    > Why should US fund WHO ~5-6 times more than China [0] (and more than EU) The base contributions are a function of GDP. The extra contributions are voluntary, and the US did it because it was in the US’ interests. It’s a founding error in the US foreign policy budget and was a good investment in terms of goodwill and data for American health research institutions. WHO must focus where it is needed most. Public health is much better in the EU (and even in Europe, accounting for places like Belarus and Ukraine) than in China, and there are much fewer epidemics that emerge in Europe in general. The whole idea is that if we limit the emergence of epidemics where they are likely to happen, we end up with fewer pandemics after these epidemics spread worldwide (which includes Europe and North America). The whole world is better without another COVID, Ebola, or Polio. > only to have the WHO be controlled by China This is bullshit. The WHO is not controlled by China any more than other UN institutions. What is certain, though, is that the US won’t have any say whatsoever once they are out.

derektank 2y ago

Does anyone know if the St Louis Federal Reserve (and I guess the federal reserve banks generally) is subject to presidential executive orders or is it entirely responsible to the Federal Reserve Board and the St. Louis Bank president? FRED is the only dataset I access regularly

bawolff 2y ago

Tbh, im kind of surprised these things weren't being archived as they were being published. Trump is an extreme case, but its not the first time a change in administration resulted in removing old websites.

strictnein 2y ago

99.9% of commenters here seem to have missed this: > For example, in the days after Joe Biden was inaugurated, data.gov showed about 1,000 datasets being deleted as compared to a day before his inauguration It's almost like this stuff happens regularly. If <insert Dem savior> wins in 2028, tons of government websites will also change in the first couple weeks of their presidency. Is it because they're a fascist dictator? Or is it because those websites reflect the administration's viewpoints on issues? Wish people would take a deep breath and step back and think a little more. I despise Trump, but there's crying wolf and then there's the current state of media and online discourse. Trump thrives in this type of environment. He purposefully fosters it. Playing gotcha with him doesn't work because he doesn't care.

BeetleB 2y ago
Plenty of comments older than yours are pointing this out. A bit hyperbolic to claim 99.9%.
blindriver 2y ago
The depth to which the media can whip people up into a hysteria is truly striking. I think most people that voted for Trump like myself aren’t buying into the propaganda and will rather wait for the dust to settle. All I’m seeing is a president that is doing exactly what he promised, so I’m very much in the “let the chips fall where they may” category.

generalizations 2y ago

Do we know what datasets these are? Do we actually have a diff here so we know what's been removed? There's a lot of assumptions being thrown around here, but we don't even know if this is some kind of malicious compliance. An actual list of what's been removed would probably clear the air a lot. As one of the reddit comments (in the thread linked by the article) pointed out, > During the start of Biden’s term, On 6th feb data.gov had “218,384 DATASETS” but on 7th feb it only had “192,180 DATASETS”

honestSysAdmin 2y ago

Let's make torrents and seed them.

bamboozled 2y ago
They might come after you for hosting it ?
- bbwbsb 2y ago
  i2p could be an option

tmshapland 2y ago

[flagged]

p3rls 2y ago
really looking forward to four more years of astute political commentary like this
- gopher_space 2y ago
  For all of the downsides to a Trump administration, one of the unexpected upsides is that the people who don't have a problem with Nazis will self-identify. I feel really bad for the kids who've developed an alt-right persona during the pandemic, though. There used to be time to work that out for yourself, but now I think social media will provide the kiss of death for many career ambitions.
- pessimizer 2y ago
  They can't keep it up for another decade, the financing is running out. Anti-Trump hysteria was largely financed by actors who dominated establishment Republican and Democratic corruption networks, but had no connection to Trump (who's a real estate guy, which is like the ghetto of finance.) During that decade, and especially during the four years he was out of office, a most of them have solidly established relationships with Trump and his network, especially tech guys. One day, upper-middle class people will stop seeing as brave and insightful calling half the country Nazis for not conforming to their suburban aesthetic. They've even eventually stopped progressively describing him as the submissive homosexual partner to Putin, I suppose because we've run out of fighting aged adult males in Ukraine so there's no reason to care about that any more. If Trump starts a war with Iran, they'll start praising him as the new Lincoln. It's not real. The only weird part is that the people attacking Trump for censorship that hasn't happened are delighted by censorship. They think the main problem with the world is that too many people are allowed to speak about whatever they want.
  - mpalmer 2y ago
    > calling half the country Nazis for not conforming to their suburban aesthetic > attacking Trump for censorship that hasn't happened Federal funds are being withheld - illegally, in most cases - from any entity that supports or even acknowledges what the administration calls "radical gender ideology". It's totalitarian and regressive. Nazi's a fine word for what the leaders of the party are showing themselves to be. I wouldn't call half the country Nazis - unless they chose to stand with the human trash in the White House. Then I wouldn't really have a problem with it.

andyjohnson0 2y ago

If the intention is to restore these data sets at some future date, when sanity has possibly been restored, then there needs to be a way to demonstrate that the archived data hasn't itself been modified. Without that, malign actors (e.g. oil/gas lobby) could very easily poison the future.

DJBunnies 2y ago
If only we had checksums or file hash integrity.
- andyjohnson0 2y ago
  If the originals are gone, and if the original authoritative, trustworthy data custodian no longer exists, then how will we know what the checksum/hash should be?

downrightmike 2y ago

Already seeing: 404 Not Found: Requested route ('ed-public-download.app.cloud.gov') does not exist.

liontwist 2y ago

I think people are interested in archiving and the political image associated with that but I don’t think anybody cares about the content. Who is going to go back and read Biden era agency publications?

acureau 2y ago
Historians, political scientists, and social scientists come to mind
- deleted 2y ago
  [deleted]

Slava_Propanei 2y ago

[dead]

ThinkBeat 2y ago

I hope volunteers and others are able to save as much as possible of the data. Removing and altering of the information and data is one of the fundamental threats in our digital world. It probably makes the most sense to do this on a daily basis. If something new, if published, grab it as soon as possible. Data can also be redacted or altered for a variety of reasons, being able to see the before and after states can be illuminating. Something I feel is missing here are statistics for each administration. Does this only happen under a Trump administration, or does it happen to smaller or larger extent under other administrations? I don't know how far back this federal goes so it might not be easy,