33 ms·

Feynman’s Maze-Running Story

210 points by telotortium 3y ago

diziet 3y ago
If this was discovered in the 1930s, why haven't others attempted to remove side channel information as they perfected their experiments, even if the original research was not published broadly?
- bottom999mottob 3y ago
  Feynman implies that psychology is a "cargo-cult" science where variations of experiments and assumed hypothesis are valued over scientific rigor or reproducibility. That's my guess too. See: misattribution of arousal studies, Millikan oil drop studies, Covid-19 and Didier Raoult Livre with hydroxychloroquine... etc. etc. Edit: The article states B.F. skinner did create a box to remove these side-channel effects for rat experiments
- bigbillheck 3y ago
  > why haven't others attempted to remove side channel information as they perfected their experiments How do we know they haven't?
  - aeternum 3y ago
    The replication crisis represents decent evidence that this kind of thing is still a major problem today.
    - gopher_space 3y ago
      I’d consider that a broad label covering all sorts of situations rather than a smoking gun. To replicate the paper I’m currently reading you’d need to source or rebuild some really shitty 1970s tech and then ignore decades of thinking around the domain. It’d be a waste of time.
      - JeremyBanks 3y ago
        [dead]
- AtlasBarfed 3y ago
  Math and hard science have political and cultural biases and inertia as well. At a minimum, hard science is still manipulated by justifying funding. And then there's this: https://esoterx.com/2014/12/03/murder-by-math-the-irrational-demise-of-hippasus/ https://esoterx.com/2014/12/03/murder-by-math-the-irrational... There was institutional pushback on Godel's incompleteness from the axiomatic mathematicians that thought they could prove everything as they reconstructed math from axioms (if I understand that). From what I recall reading the axiomatic reconstruction was basically a huge industry of mathematical work, and suddenly some pissant young guy shows up and says "yeah, that not going to end the way you hope". The mechanism of tenure alone will lead to political grandstanding. If your research is borderline on tenure but some other research completely undermines it, that is millions of lifetime dollars on the line. See: MOND vs Dark Matter, string theory, etc.
telotortium 3y ago
This may have been submitted before, but the update today is that Gwern appears to have found the research that Feynman was referring to.
MrLeap 3y ago
The idea of an academic organization "fixing its hermeti⁣cism" is thought provoking. Were these nerds toiling to the benefit or tune of some charlatan, some _Trismegistus_ -- or were they too much the hermit, one who toils without the clutch or gearing for meshing with the gears of society? With the central limit theorem their sovereign then probably both I guess, instinct says more the latter. The way you file the teeth of your gears is a real struggle. I wish there were more resources for it, AI has an opportunity to fill a gap here. I know the path of salvation is doing my normal do _just more aggressively and in public_. I managed to carve out a real small niche making an indie game publicly over a year and a half, posted a few hundred videos. Some knocks and I lost the momentum for constant publishing. The work hasn't stopped, but the ephemeral threads of a narrative feel harder to grasp. Do I just start posting things out of nowhere, acting like I never stopped? Do I montage the unpublished backlog? Do I wait until I've achieved some scientific breakthrough? I've got videos of flood fill lightning and neural networks and sheep shapeshifting and explosions and probably 100 other things I've forgotten. Maybe the problem is that I never knew who my audience was. Shouting into the canyon is harder than it looks.
- senectus1 3y ago
  got a link to it?
  - MrLeap 3y ago
    https://twitter.com/leapjosh https://twitter.com/leapjosh is where I posted most of my things. I've got a small demo here -- https://store.steampowered.com/app/1537490/Tentacle_Typer/ https://store.steampowered.com/app/1537490/Tentacle_Typer/ Basically all of my textual content ends up here on HN :)
    - bottom999mottob 3y ago
      Wow this is one of those gems on Steam I was looking for last night... I hope you release it one day because I've never seen anything like it: a Lovecraft-ian white-collar spell-casting acid trip?
hilbert42 3y ago
I have long complained about the orphan works problem and the need to free them up through copyright reform. This case is a quintessential example and copyright reformers should take note.
- gerikson 3y ago
  Are university results covered by copyright in the US?
  - drdec 3y ago
    Typically the copyright belongs to the journal which published the results.
    - frumiousirc 3y ago
      And that (c) applies to only the published copy. It is very common, even standard in many fields, for researchers to make the same material, just not exactly the published copy, publicly available. See eg arxiv.org.
- deleted 3y ago
  [deleted]
- rnhmjoj 3y ago
  Copyright needs some kind of usucaption mechanism: if the holder doesn't make any effort in maintaining his works available for an extended period of time they should automatically enter public domain.
tlb 3y ago
The real morality play here is to publish your work. I suggest that Shepard had a net-negative effect on the field. Anyone who might eventually have done the same research would have heard about Shepard's work and been discouraged from re-doing it. Yet, it remained obscure and little-used. It may be better to not do the research at all than to do it, mark the territory, and not publish.
- macleginn 3y ago
  This. I had a grant proposal rejected because the reviewers knew that something similar was being done by someone else -- who then never bothered to complete the work and publish properly.
  - JohnKemeny 3y ago
    What recently happened with me is that we proved an open problem, and while writing it up, we found that the result was claimed in an arxiv paper that never got published anywhere. Their proof was incomplete. This meant that we had to acknowledge that the result already was claimed by someone else, but that we couldn't understand the proof attempt. This of course leads to problems because now our result is somehow viewed as less valuable, and more difficult to publish. Which again means that research in that area will suffer from an unpublished claim.
    - itronitron 3y ago
      What are you measuring to determine that now your result is viewed as less valuable? Is it the peer reviewers or personal conviction? In the end all that matters for you, to the extent that peer-reviewed publications are important to your career, is that your paper gets cited. And if people are citing the unfinished arxiv paper instead of yours then that speaks poorly of their own research and not yours.
- deleted 3y ago
  [deleted]
- derbOac 3y ago
zabzonk 3y ago
i did my a-level (british pre-university exam) project in 1971 on maze-running rats. i soon worked out that the horrible creatures were pissing everywhere and following the scent (and difficult to deal with for a plethora of other reasons) and switched to gerbils - a slight improvement (desert animals - don't pee so much) , and less likely to have your finger taken off compared with an enraged rat.
- bottom999mottob 3y ago
  I'm curious, if it was a possibility that your finger get bitten off, did people not wear leather handling gloves or could rats bite through the gloves or something else?
  - zabzonk 3y ago
    most breeds of lab rats are pretty docile, but sometimes even they get pissed off when they have been starved (to motivate them) and then stuffed in a maze. no, i never wore gloves. them was hard times!
- biofox 3y ago
  How times change. To run the same experiment in the UK today you'd need a Home Office licence, a project licence, and on-site veterinary facilities.
  - zabzonk 3y ago
    possibly. i wasn't doing anything nasty to the rats/gerbils except keeping them a bit hungry, and i despise animal cruelty in all its forms. i actually gave up attending my HND biology course (needed to progress in career as a microbiology tech) because they needlessly made us pith a frog (i like frogs). still won the prize for best student - god knows how! stick with your convictions is probably the answer.
  - bigbillheck 3y ago
    Sounds like an improvement to me.
gnfargbl 3y ago
> The biggest problem the Young dead-end poses is that having failed to find any trace in it, one must begin to assume that Feynman got the year, surname, or both wrong. Alternative hypothesis: Feynman knew who the individual was but, having not asked them if they objected to being used to illustrate his anecdote, chose to hide their identity with a pseudonym. In that case the reference wasn't to "Mr Q. Young", it was to "Mr Young Researcher".
- stdbrouw 3y ago
  I thought so too at first, but that would only make sense if "Cargo Cult Science" were an unedited third-party transcript of the commencement address, and I don't think that it is, given its inclusion in "Surely You're Joking, Mr. Feynman!." The address refers to "a man named Young", not "a man, let's call him Young".
  - gnfargbl 3y ago
    I see your point, and it's a good one. I wonder if it's possible that there was a transcription error, or perhaps something introduced by the editor of the journal where the speech was first published [1]. If there was, then it's perhaps also plausible that the error could have been carried forward by copying and pasting the journal article into Surely you're joking. All wild speculation on my part, of course. I wonder if any further evidence will ever turn up! [1] https://calteches.library.caltech.edu/51/2/CargoCult.pdf https://calteches.library.caltech.edu/51/2/CargoCult.pdf
  - ShamelessC 3y ago
    > that would only make sense You're suggesting that _no one_ ever used a pseudonym for another without explicitly mentioning it in advance/shortly thereafter? Besides, "Young" as a false-name doesn't have to be a direct clever "reference" to "a young person". It could just be a name he thought of at the time. In any case, I think Gwern is far too conspiratorial in the reasoning for this article. There are some interesting connections found but the evidence, while reasonable, is not nearly as conclusive as he suggests.
bottom999mottob 3y ago
Absolutely ridiculous that Caltech has had Feynman's papers for over 30 years without digitizing them or allowing people to read them without visiting LA! One of the greatest minds to ever exist, and Caltech is doing a disservice to his name with their white tower holding his documents. My blood's boiling just reading about Gwern's search through the literature. Shoutout to Gwern for digging this deep into one of my hero's citations
- nullc 3y ago
  to quote myself, "All too often journals, galleries, and museums are becoming not disseminators of knowledge—as their lofty mission statements suggest—but censors of knowledge, __because censoring is the one thing they do better than the Internet does__."
  - chinchilla2020 3y ago
    It's a great quote and I will use it.
    - nullc 3y ago
      Context: https://nt4tn.net/papers/201107.PTRSOL_statement.txt https://nt4tn.net/papers/201107.PTRSOL_statement.txt
- mustacheemperor 3y ago
  The OP says Feynman "made a practice of donating" his papers to Caltech. Did he know or expect they'd be managed in such a manner? Edit: According to Caltech, they have "93 boxes" of Feynman's work that has never been backed up digitally[0]. Beyond the issue of access, what happens if there's ever a sprinkler burst in the wrong room of the archives? [0]https://collections.archives.caltech.edu/repositories/2/resources/168 https://collections.archives.caltech.edu/repositories/2/reso...
derbOac 3y ago
The piece kind of focuses on psychology, but this amnesia and collective forgetting tends to happen in all fields to some extent. It might be more significant in psychology, but there is research showing that some of the most cited papers tend to cite both new and very very old work (that is the references cited tend to vary in chronological age significantly), that significant research is often forgotten and sometimes rediscovered. I think it goes along with the Kuhnian (?) idea of paradigms becoming overly dominant and overly entrenched, and coming and going wholesale in waves. It's also consistent with Stigler's law of eponymy. As fields lurch from one idea or paradigm to another they tend to pretend that anything similar didn't exist beforehand. There's a need to represent ones's self and ideas as revolutionary and genius and implicitly, by comparison, to denigrate everything that came before as irrelevant. This article describes a related but different phenomenon: https://www.nature.com/nature-index/news-blog/review-articles-cause-dramatic-loss-in-citations-for-original-research https://www.nature.com/nature-index/news-blog/review-article... One of my curiosities is whether or not this has gotten worse in science and academics. It often feels like in academics there's such a pressure for self-promotion and receipt of attention, to capture attention and be at the center of what is the focus of attention, that issues of veridicality or what has and should be the focus of attention gets swept away.
- joe_the_user 3y ago
  ...this amnesia and collective forgetting tends to happen in all fields to some extent... Well, I think there can an opposite problem that appears looking at these things. Psychology is such garbage, so likely to produce worthless experiments, that those that don't want to call a spade a spade instead try to take all the garbage of the field and average it over all science so that they can have something different to say. This relates to the "replication crisis" [1]. You can say "The replication crisis is frequently discussed in relation to psychology and medicine..." but also said to extend throughout science. It may indeed extend throughout sciences but including a data group (experimental psychology) that you is garbage isn't necessarily a good way to see what's happening in the rest of science. And, of course, a field "having amnesia" by itself doesn't necessarily imply any kind of problem. A given practitioner A) can only remember so much and B) benefits from a compact statement of a result, which may be quite different from how the result was originally stated. The math I learned in my MA came from text books that gave self-contained summaries of the various topics, rather than from the original papers the ideas came from and that made it much easier. [1] https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
kjellsbells 3y ago
The anecdote is telling, and of course for the HN crowd we are as susceptible to cargo cults as anyone. Personally I'm very interested in the survivability of scientific information. I used to think that this was a worry of the Internet age, but papers locked up in working group reports and journals are increasingly hard to find beyond about a 25 year window. There was a thread on HN in 2020 where someone was looking for a NATO working group report from 1960 and you'd think NATO of all people would preserve records...but there's a black hole.

210 points by telotortium 3y ago

If this was discovered in the 1930s, why haven't others attempted to remove side channel information as they perfected their experiments, even if the original research was not published broadly?

bottom999mottob 3y ago
Feynman implies that psychology is a "cargo-cult" science where variations of experiments and assumed hypothesis are valued over scientific rigor or reproducibility. That's my guess too. See: misattribution of arousal studies, Millikan oil drop studies, Covid-19 and Didier Raoult Livre with hydroxychloroquine... etc. etc. Edit: The article states B.F. skinner did create a box to remove these side-channel effects for rat experiments
bigbillheck 3y ago
> why haven't others attempted to remove side channel information as they perfected their experiments How do we know they haven't?
- aeternum 3y ago
  The replication crisis represents decent evidence that this kind of thing is still a major problem today.
  - gopher_space 3y ago
    I’d consider that a broad label covering all sorts of situations rather than a smoking gun. To replicate the paper I’m currently reading you’d need to source or rebuild some really shitty 1970s tech and then ignore decades of thinking around the domain. It’d be a waste of time.
    - JeremyBanks 3y ago
      [dead]
AtlasBarfed 3y ago
Math and hard science have political and cultural biases and inertia as well. At a minimum, hard science is still manipulated by justifying funding. And then there's this: https://esoterx.com/2014/12/03/murder-by-math-the-irrational-demise-of-hippasus/ https://esoterx.com/2014/12/03/murder-by-math-the-irrational... There was institutional pushback on Godel's incompleteness from the axiomatic mathematicians that thought they could prove everything as they reconstructed math from axioms (if I understand that). From what I recall reading the axiomatic reconstruction was basically a huge industry of mathematical work, and suddenly some pissant young guy shows up and says "yeah, that not going to end the way you hope". The mechanism of tenure alone will lead to political grandstanding. If your research is borderline on tenure but some other research completely undermines it, that is millions of lifetime dollars on the line. See: MOND vs Dark Matter, string theory, etc.

telotortium 3y ago

This may have been submitted before, but the update today is that Gwern appears to have found the research that Feynman was referring to.

MrLeap 3y ago

The idea of an academic organization "fixing its hermeti⁣cism" is thought provoking. Were these nerds toiling to the benefit or tune of some charlatan, some _Trismegistus_ -- or were they too much the hermit, one who toils without the clutch or gearing for meshing with the gears of society? With the central limit theorem their sovereign then probably both I guess, instinct says more the latter. The way you file the teeth of your gears is a real struggle. I wish there were more resources for it, AI has an opportunity to fill a gap here. I know the path of salvation is doing my normal do _just more aggressively and in public_. I managed to carve out a real small niche making an indie game publicly over a year and a half, posted a few hundred videos. Some knocks and I lost the momentum for constant publishing. The work hasn't stopped, but the ephemeral threads of a narrative feel harder to grasp. Do I just start posting things out of nowhere, acting like I never stopped? Do I montage the unpublished backlog? Do I wait until I've achieved some scientific breakthrough? I've got videos of flood fill lightning and neural networks and sheep shapeshifting and explosions and probably 100 other things I've forgotten. Maybe the problem is that I never knew who my audience was. Shouting into the canyon is harder than it looks.

senectus1 3y ago
got a link to it?
- MrLeap 3y ago
  https://twitter.com/leapjosh https://twitter.com/leapjosh is where I posted most of my things. I've got a small demo here -- https://store.steampowered.com/app/1537490/Tentacle_Typer/ https://store.steampowered.com/app/1537490/Tentacle_Typer/ Basically all of my textual content ends up here on HN :)
  - bottom999mottob 3y ago
    Wow this is one of those gems on Steam I was looking for last night... I hope you release it one day because I've never seen anything like it: a Lovecraft-ian white-collar spell-casting acid trip?

hilbert42 3y ago

I have long complained about the orphan works problem and the need to free them up through copyright reform. This case is a quintessential example and copyright reformers should take note.

gerikson 3y ago
Are university results covered by copyright in the US?
- drdec 3y ago
  Typically the copyright belongs to the journal which published the results.
  - frumiousirc 3y ago
    And that (c) applies to only the published copy. It is very common, even standard in many fields, for researchers to make the same material, just not exactly the published copy, publicly available. See eg arxiv.org.
deleted 3y ago
[deleted]
rnhmjoj 3y ago
Copyright needs some kind of usucaption mechanism: if the holder doesn't make any effort in maintaining his works available for an extended period of time they should automatically enter public domain.

tlb 3y ago

The real morality play here is to publish your work. I suggest that Shepard had a net-negative effect on the field. Anyone who might eventually have done the same research would have heard about Shepard's work and been discouraged from re-doing it. Yet, it remained obscure and little-used. It may be better to not do the research at all than to do it, mark the territory, and not publish.

macleginn 3y ago
This. I had a grant proposal rejected because the reviewers knew that something similar was being done by someone else -- who then never bothered to complete the work and publish properly.
- JohnKemeny 3y ago
  What recently happened with me is that we proved an open problem, and while writing it up, we found that the result was claimed in an arxiv paper that never got published anywhere. Their proof was incomplete. This meant that we had to acknowledge that the result already was claimed by someone else, but that we couldn't understand the proof attempt. This of course leads to problems because now our result is somehow viewed as less valuable, and more difficult to publish. Which again means that research in that area will suffer from an unpublished claim.
  - itronitron 3y ago
    What are you measuring to determine that now your result is viewed as less valuable? Is it the peer reviewers or personal conviction? In the end all that matters for you, to the extent that peer-reviewed publications are important to your career, is that your paper gets cited. And if people are citing the unfinished arxiv paper instead of yours then that speaks poorly of their own research and not yours.
deleted 3y ago
[deleted]
derbOac 3y ago

zabzonk 3y ago

i did my a-level (british pre-university exam) project in 1971 on maze-running rats. i soon worked out that the horrible creatures were pissing everywhere and following the scent (and difficult to deal with for a plethora of other reasons) and switched to gerbils - a slight improvement (desert animals - don't pee so much) , and less likely to have your finger taken off compared with an enraged rat.

bottom999mottob 3y ago
I'm curious, if it was a possibility that your finger get bitten off, did people not wear leather handling gloves or could rats bite through the gloves or something else?
- zabzonk 3y ago
  most breeds of lab rats are pretty docile, but sometimes even they get pissed off when they have been starved (to motivate them) and then stuffed in a maze. no, i never wore gloves. them was hard times!
biofox 3y ago
How times change. To run the same experiment in the UK today you'd need a Home Office licence, a project licence, and on-site veterinary facilities.
- zabzonk 3y ago
  possibly. i wasn't doing anything nasty to the rats/gerbils except keeping them a bit hungry, and i despise animal cruelty in all its forms. i actually gave up attending my HND biology course (needed to progress in career as a microbiology tech) because they needlessly made us pith a frog (i like frogs). still won the prize for best student - god knows how! stick with your convictions is probably the answer.
- bigbillheck 3y ago
  Sounds like an improvement to me.

gnfargbl 3y ago

> The biggest problem the Young dead-end poses is that having failed to find any trace in it, one must begin to assume that Feynman got the year, surname, or both wrong. Alternative hypothesis: Feynman knew who the individual was but, having not asked them if they objected to being used to illustrate his anecdote, chose to hide their identity with a pseudonym. In that case the reference wasn't to "Mr Q. Young", it was to "Mr Young Researcher".

stdbrouw 3y ago
I thought so too at first, but that would only make sense if "Cargo Cult Science" were an unedited third-party transcript of the commencement address, and I don't think that it is, given its inclusion in "Surely You're Joking, Mr. Feynman!." The address refers to "a man named Young", not "a man, let's call him Young".
- gnfargbl 3y ago
  I see your point, and it's a good one. I wonder if it's possible that there was a transcription error, or perhaps something introduced by the editor of the journal where the speech was first published [1]. If there was, then it's perhaps also plausible that the error could have been carried forward by copying and pasting the journal article into Surely you're joking. All wild speculation on my part, of course. I wonder if any further evidence will ever turn up! [1] https://calteches.library.caltech.edu/51/2/CargoCult.pdf https://calteches.library.caltech.edu/51/2/CargoCult.pdf
- ShamelessC 3y ago
  > that would only make sense You're suggesting that _no one_ ever used a pseudonym for another without explicitly mentioning it in advance/shortly thereafter? Besides, "Young" as a false-name doesn't have to be a direct clever "reference" to "a young person". It could just be a name he thought of at the time. In any case, I think Gwern is far too conspiratorial in the reasoning for this article. There are some interesting connections found but the evidence, while reasonable, is not nearly as conclusive as he suggests.

bottom999mottob 3y ago

Absolutely ridiculous that Caltech has had Feynman's papers for over 30 years without digitizing them or allowing people to read them without visiting LA! One of the greatest minds to ever exist, and Caltech is doing a disservice to his name with their white tower holding his documents. My blood's boiling just reading about Gwern's search through the literature. Shoutout to Gwern for digging this deep into one of my hero's citations

nullc 3y ago
to quote myself, "All too often journals, galleries, and museums are becoming not disseminators of knowledge—as their lofty mission statements suggest—but censors of knowledge, __because censoring is the one thing they do better than the Internet does__."
- chinchilla2020 3y ago
  It's a great quote and I will use it.
  - nullc 3y ago
    Context: https://nt4tn.net/papers/201107.PTRSOL_statement.txt https://nt4tn.net/papers/201107.PTRSOL_statement.txt
mustacheemperor 3y ago
The OP says Feynman "made a practice of donating" his papers to Caltech. Did he know or expect they'd be managed in such a manner? Edit: According to Caltech, they have "93 boxes" of Feynman's work that has never been backed up digitally[0]. Beyond the issue of access, what happens if there's ever a sprinkler burst in the wrong room of the archives? [0]https://collections.archives.caltech.edu/repositories/2/resources/168 https://collections.archives.caltech.edu/repositories/2/reso...

derbOac 3y ago

The piece kind of focuses on psychology, but this amnesia and collective forgetting tends to happen in all fields to some extent. It might be more significant in psychology, but there is research showing that some of the most cited papers tend to cite both new and very very old work (that is the references cited tend to vary in chronological age significantly), that significant research is often forgotten and sometimes rediscovered. I think it goes along with the Kuhnian (?) idea of paradigms becoming overly dominant and overly entrenched, and coming and going wholesale in waves. It's also consistent with Stigler's law of eponymy. As fields lurch from one idea or paradigm to another they tend to pretend that anything similar didn't exist beforehand. There's a need to represent ones's self and ideas as revolutionary and genius and implicitly, by comparison, to denigrate everything that came before as irrelevant. This article describes a related but different phenomenon: https://www.nature.com/nature-index/news-blog/review-articles-cause-dramatic-loss-in-citations-for-original-research https://www.nature.com/nature-index/news-blog/review-article... One of my curiosities is whether or not this has gotten worse in science and academics. It often feels like in academics there's such a pressure for self-promotion and receipt of attention, to capture attention and be at the center of what is the focus of attention, that issues of veridicality or what has and should be the focus of attention gets swept away.

joe_the_user 3y ago
...this amnesia and collective forgetting tends to happen in all fields to some extent... Well, I think there can an opposite problem that appears looking at these things. Psychology is such garbage, so likely to produce worthless experiments, that those that don't want to call a spade a spade instead try to take all the garbage of the field and average it over all science so that they can have something different to say. This relates to the "replication crisis" [1]. You can say "The replication crisis is frequently discussed in relation to psychology and medicine..." but also said to extend throughout science. It may indeed extend throughout sciences but including a data group (experimental psychology) that you is garbage isn't necessarily a good way to see what's happening in the rest of science. And, of course, a field "having amnesia" by itself doesn't necessarily imply any kind of problem. A given practitioner A) can only remember so much and B) benefits from a compact statement of a result, which may be quite different from how the result was originally stated. The math I learned in my MA came from text books that gave self-contained summaries of the various topics, rather than from the original papers the ideas came from and that made it much easier. [1] https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis

kjellsbells 3y ago

The anecdote is telling, and of course for the HN crowd we are as susceptible to cargo cults as anyone. Personally I'm very interested in the survivability of scientific information. I used to think that this was a worry of the Internet age, but papers locked up in working group reports and journals are increasingly hard to find beyond about a 25 year window. There was a thread on HN in 2020 where someone was looking for a NATO working group report from 1960 and you'd think NATO of all people would preserve records...but there's a black hole.