9 ms·

This group of researchers consistently demonstrates a degree of empirical rigor that is unmatched across any other ML lab in industry or academia - remarkable e

by ansk 5y ago

This group of researchers consistently demonstrates a degree of empirical rigor that is unmatched across any other ML lab in industry or academia - remarkable empirical results as always, reproducible experiments, open-source and well-engineered codebase, and valuable insights about low-level learning dynamics and high-level emergent artifacts. Applied ML wouldn't have such a bad rap if more researchers held themselves to similar standards.

sillysaurusx 5y ago
This isn't true. I do ML every day. You are mistaken. I click the website. I search "model". I see two results. Oh no, that means no download link to model. I go to the github. Maybe model download link is there. I see zero code: https://github.com/NVlabs/alias-free-gan https://github.com/NVlabs/alias-free-gan Zero code. Zero model. You, and everyone like you, who are gushing with praise and hypnotized by pretty images and a nice-looking pdf, are doing damage by saying that this is correct and normal. The thing that's useful to me, first and foremost, is a model. Code alone isn't useful. Code, however, is the recipe to create the model. It might take 400 hours on a V100, and it might not actually result in the model being created, but it slightly helps me. There is no code here. Do you think that the pdf is helpful? Yeah, maybe. But I'm starting to suspect that the pdf is in fact a tech demo for nVidia, not a scientific contribution whose purpose is to be helpful to people like me. Okay? Model first. Code second. Paper third. Every time a tech demo like this comes out, I'd like you to check that those things exist, in that order. If it doesn't, it's not reproducible science. It's a tech demo. I need to write something about this somewhere, because a large number of people seem to be caught in this spell. You're definitely not alone, and I'm sorry for sounding like I was singling you out. I just loaded up the comment section, saw your comment, thought "Oh, awesome!" clicked through, and went "Oh no..."
- aaron-santos 5y ago
  Thank you for calling this out. It's critically important that people understand the difference between model, code, and paper and what they mean. It's also important that people understand that even if code is provided, it's commercially useless. From the NVAE license as an example[1] > The Work and any derivative works thereof only may be used or intended for use non-commercially. It's a great example of the difference between open source (which it is) and free software which it is not. So we're back to square one where it is probably best to clean-room the implementation from the paper, which is nearly useless to reproduce the model. [1] https://github.com/NVlabs/NVAE/blob/master/LICENSE https://github.com/NVlabs/NVAE/blob/master/LICENSE
  - sillysaurusx 5y ago
    Unfortunately, I must call you out too, my friend. With love. Because it’s crucially important that we protect the scientific method here. The sole goal is to help people like me reproduce the model. If I can’t reproduce the model, I can’t verify the paper. When I saw “commercial” and then “open source” in your comment, I said “oh no…” My duty is to the scientific method, so I don’t care if it’s the most restrictive code on the planet as long as I can use it to reproduce the model in the paper. Because at that point, I have a baseline for evaluating the paper’s claims. The reason I assume the paper is false until proven otherwise, is because the paper often doesn’t have enough detail to reproduce the model shown in the videos on this tech demos. Meaning, if they’re the it to help me, the ML researcher, then they’re failing to tell me how to evaluate their claims rigorously. (That said, it’s breaking my heart that I can’t agree with you here, because I want to so badly. I’ve felt similarly for years that scientific contributions need to be “free as in beer” commercially. But I recognize signs of zealotry when I see them, and I can’t let my personal views creep in, because people like me would stop listening if I was here e.g. arguing vehemently that nVidia needed to be delivering us something commercially viable along with a high quality codebase. The price for entry to the scientific method isn’t so high.)
    - aaron-santos 5y ago
      That's fine, at least you're open about the knowledge for knowledge's sake position. There's more than one way to judge something.
      - xwolfi 5y ago
        It's more than that. Here there's a fake demo we can only consider an advertisement. If they wanted it to be a scientific paper it should be reproducible and counter checked. What is the point of making a paper full of screenshot ? It's not just a knowledge for knowledge's sake issue here, it's that it's not even knowledge they're publishing. They're publishing nothing. They would make a license that says the code can only be provided for peer review and counter validation, then that'd be knowledge. Then, the sake of it is another secondary problem.
- ansk 5y ago
  You're clearly disillusioned with the general accessibility of ML research, but I don't think your cynicism is warranted here. Take a look at their prior works[1], and I think you'll agree they go above and beyond in making their work accessible and reproducible. There is no reason to doubt the open-source release of this work will be any different. As to why the release is delayed, I'd speculate it's because they put a significant additional amount of work into releases and because releasing code in a large corporation is a bureaucratic hassle. [1] https://nvlabs.github.io/stylegan2/versions.html https://nvlabs.github.io/stylegan2/versions.html
  - sillysaurusx 5y ago
    There is no reason to doubt the open-source release of this work will be any different. Then this is not a scientific contribution yet. We must wait and see. The most important tenet of science, is to doubt. I didn’t even read the name on the paper before I wrote my comment. Yes, I know this group. They’re why I got into ML, along with the group from OpenAI who published GPT-2. Because A+ science. Their claims here are likely wrong unless and until proven otherwise. This isn’t a hardline position. It’s been my experience across many codebases, during my two years of trying to reproduce many ideas. I agree that that is an example of A+ science. But why do you think they’re punishing this now, today? Either because conference deadline or because nVidia pressure. Neither of those are related to helping me achieve the scientific method: reproducing the idea in the paper, to verify their claims. All I can do is kind of try to reverse engineer some vague claims in a pdf, without those things. -- Let me tell you a little bit about my job, because my time with my job may soon come to an end. I think that might clear up some confusion. My job, as an ML researcher, is to learn techniques that may or may not be true, combine them in novel ways, and present results to others. Knowledge, Contribution, Presentation, in that order. The first step is to obtain knowledge. Let's set aside the question of why, because why is a question for me personally, which is unrelated. Scientific knowledge comes when Knowledge, Contribution, and Presentation are all achieved in a rigorous way. The rigor allows people like me to verify that I have knowledge. Without this, I have mistaken knowledge, which is worse than useless. It's an illusion – I'm fooling myself. When I got into ML two years ago, I thought that knowledge would come from reading scientific papers. I was wrong. Most papers, are wrong. That's been my experience for the past two years. My experience may be wrong. Maybe others obtain rigorous scientific knowledge through the paper alone. But researchers happen to obtain a dangerous thing: prestige. Unfortunately, prestige doesn't come from helping others obtain knowledge. It comes from that last step -- presentation. The presentation on this thread is excellent. It's another Karras release. I agree; there's no reason to doubt they'll be just as rigorous with this release as they are with stylegan2. But knowledge doesn't come from presentation. Only prestige. Prestige makes a lot of new researchers try very hard to obtain the wrong things. If all of these were small concerns, or curious quirks, they'd be a footnote in my field guide. But I submit that these things are front and center to the current state of affairs in 2021. Every time a release like this happens, it generates a lot of fanfare and we come together in celebration because ML Is Happening, Yay! And then I try to obtain the Knowledge in the fanfare, and discover that either it's absent or mistaken. Because there are no tools for me to verify their claims -- and when I do, I often see that they don't work! That's right. I kept finding out that these things being claimed, just aren't true. No matter how enticing the claim is, or whether it sounds like "Foobars are Aligned in the Convolution Digit," the claim, from where I was sitting, seemed to be wrong. It contained mistaken knowledge -- worse than useless. Unfortunately, two years with no salary takes a toll. I could spend another few years doing this if I wanted to. But I wound up so disgusted with discovering that we're all just chasing prestige, not knowledge, that I'd rather ship production-grade software for the world's most boring commercial work, as long as the work seems useful and the team seems interesting. Because at least I'd be doing something useful.
- minimaxir 5y ago
  The repo says the code will be available in September: that's a reasonable timeframe for the necessary polish/legal clearance.
  - sillysaurusx 5y ago
    (Agreed, fwiw. What’s going on here isn’t a criticism of this work specifically, but the trend of everyone thinking that this is science generally. For example, it’s true the code is coming in September. And, you and I both know it’s probably gonna have a model release, just because it’s more impressive, big-name Karras nVidia work. But it might not have a model release. I give that at least 40% odds. If it doesn’t, then everything I said above will be true about that too, Karras or not. People keep doing that, and we have to call out that this is approximately useless for you and me. Actually, I was going to say maybe it’s useful for you, but you’re the language model hacker and I’m the GAN hacker, and I assure you, code alone is useless for me. If it’s useful to you, I would love to know how it helps you verify the scientific method.)
- varispeed 5y ago
  I am not into ML, but from time to time I like to look how this is made and remember only once seeing the code and a model, which I thought was exception from the "norm". Good that more people are calling this out!
- godelski 5y ago
  > I do ML every day. > I go to the github. Maybe model download link is there. I see zero code Paper was released today. Chill. They said they will release the code in September (I'm guessing late September). The paper is also a pre-print. They're probably aiming for CVPR and don't want to get scooped. > Model first. Code second. Paper third. That's how you produce ML code and documentation but that is not how you release it. I guarantee you that they are still tuning and making the model better. They're were still updating ADA till pretty recently (last commit on the pytorch version is 4 months ago, to code). I originally wasn't in CS, and when I first came over I wasn't in ML. We never had code. The fact that ML publishes models AND checkpoints is a godsend. I love it. Makes work so much easier and helps the community advance faster. I love this, but just chill. The paper isn't peer-reviewed. It is a pre-print. They're showing people what they've done in the last 6 months. It's part publicity stunt, part flex, part staking claim, but it is also part sharing with the community. Even without the code we learn a lot because they attached a paper to it. So chill.
  - sillysaurusx 5y ago
    The fact that you had to say "chill" three times indicates that you're trying to convince yourself, not me. None of what you said is responsive to what I wrote. I think it's an opinion piece, but I'm not sure. The issue here is the scientific method. I've listed the things that are required, as I see it. And I've also listed the reasons why I haven't been able to verify it exists here, despite trying for two years. I'm glad that you like ML hacking, and I like it too. But models aren't a godsend; they're "the most basic, bare-minimum requirements of reproducibility." Your reaction shouldn't be "I'm incredibly grateful you'd be willing to do this." It should be "You're required to do this, because if I can't verify your claims, your claims might be mistaken." To leave it off on a softer note, normally I'd bond with you, ML hacker to ML hacker. Because I love ML, and I love hearing what you've been up to in ML. It's the best job in the world, as far as I'm concerned. (Could any other career give you the opportunity to be a developer advocate for high-performance computing in such an interesting way? https://github.com/google/jax/issues/2108#issuecomment-866238579 https://github.com/google/jax/issues/2108#issuecomment-86623... Definitely looking for more examples of "Github Larping," if you know of any.) If you agree that the scientific method is the reason ML moves forward, all I'm doing here is protecting it.
- pabs3 5y ago
  Would you not want the data and code used to train the model, rather than the trained model itself? Edit: the Debian Deep Learning Team's Machine Learning Policy explains why. https://salsa.debian.org/deeplearning-team/ml-policy https://salsa.debian.org/deeplearning-team/ml-policy
  - sillysaurusx 5y ago
    For various reasons, I left data out of the requirements because most interesting research uses data unavailable to the community. CLIP is such an example, and it's A+ science: Model, Code, Paper. Having the model is enough to verify the paper's claims, and also to experiment with new approaches (since you can fine-tune the model). That said, I make this concession as a "meet you halfway" compromise between hard-line positions: "We can't release models, because we trained them on private data" and "You must release both models and data." In other words, you're technically correct, but in my estimation it would do more harm to the end goal: the whole reason the scientific method is useful, is because it makes the world more useful. The world would be less useful if fewer commercial companies participated in the scientific method. It's an inclusive group, not an exclusive clique. All you have to do, is give me the tools to verify your claims.
    - pabs3 5y ago
      Yeah, for the goal of verifying claims that is good enough indeed. I'm not sure it is good enough for a variety of other possibly useful use-cases though; eliminating bias in an existing model, correcting a flaw in the training code, creating a different model and proving it is better by training on the same data etc. It would be nice if there were more public/libre data sets for ML stuff.
- clircle 5y ago
  I hardly understand this comment. Statisticians have been publishing and arguing about models for 100 years now. No one required code to verify authenticity of research. I suppose it is the sorry state of machine learning research that the methodology is so poor that a person cannot verify the research from the paper.
- fxtentacle 5y ago
  You won't need that much source code to verify their claims. Their central improvement is that they limit the generation of high frequencies by ReLU through a upsample-ReLU-filter-downsample sequence. Their theoretical section explains quite well why high frequencies can be proven mathematically to cause issues. And their practical implementation using filters to cut those off is very straightforward. If someone tells you "The microphone recording had 50Hz noise so I used a filter to remove it", that's pretty much good enough for someone with experience in the field to replicate their results. This is the equivalent in AI. They uncovered a simple basic issue that everyone else overlooked, but once you know it, it seems obvious in retrospect.

by ansk 5y ago

sillysaurusx 5y ago

This isn't true. I do ML every day. You are mistaken. I click the website. I search "model". I see two results. Oh no, that means no download link to model. I go to the github. Maybe model download link is there. I see zero code: https://github.com/NVlabs/alias-free-gan https://github.com/NVlabs/alias-free-gan Zero code. Zero model. You, and everyone like you, who are gushing with praise and hypnotized by pretty images and a nice-looking pdf, are doing damage by saying that this is correct and normal. The thing that's useful to me, first and foremost, is a model. Code alone isn't useful. Code, however, is the recipe to create the model. It might take 400 hours on a V100, and it might not actually result in the model being created, but it slightly helps me. There is no code here. Do you think that the pdf is helpful? Yeah, maybe. But I'm starting to suspect that the pdf is in fact a tech demo for nVidia, not a scientific contribution whose purpose is to be helpful to people like me. Okay? Model first. Code second. Paper third. Every time a tech demo like this comes out, I'd like you to check that those things exist, in that order. If it doesn't, it's not reproducible science. It's a tech demo. I need to write something about this somewhere, because a large number of people seem to be caught in this spell. You're definitely not alone, and I'm sorry for sounding like I was singling you out. I just loaded up the comment section, saw your comment, thought "Oh, awesome!" clicked through, and went "Oh no..."

aaron-santos 5y ago
Thank you for calling this out. It's critically important that people understand the difference between model, code, and paper and what they mean. It's also important that people understand that even if code is provided, it's commercially useless. From the NVAE license as an example[1] > The Work and any derivative works thereof only may be used or intended for use non-commercially. It's a great example of the difference between open source (which it is) and free software which it is not. So we're back to square one where it is probably best to clean-room the implementation from the paper, which is nearly useless to reproduce the model. [1] https://github.com/NVlabs/NVAE/blob/master/LICENSE https://github.com/NVlabs/NVAE/blob/master/LICENSE
- sillysaurusx 5y ago
  Unfortunately, I must call you out too, my friend. With love. Because it’s crucially important that we protect the scientific method here. The sole goal is to help people like me reproduce the model. If I can’t reproduce the model, I can’t verify the paper. When I saw “commercial” and then “open source” in your comment, I said “oh no…” My duty is to the scientific method, so I don’t care if it’s the most restrictive code on the planet as long as I can use it to reproduce the model in the paper. Because at that point, I have a baseline for evaluating the paper’s claims. The reason I assume the paper is false until proven otherwise, is because the paper often doesn’t have enough detail to reproduce the model shown in the videos on this tech demos. Meaning, if they’re the it to help me, the ML researcher, then they’re failing to tell me how to evaluate their claims rigorously. (That said, it’s breaking my heart that I can’t agree with you here, because I want to so badly. I’ve felt similarly for years that scientific contributions need to be “free as in beer” commercially. But I recognize signs of zealotry when I see them, and I can’t let my personal views creep in, because people like me would stop listening if I was here e.g. arguing vehemently that nVidia needed to be delivering us something commercially viable along with a high quality codebase. The price for entry to the scientific method isn’t so high.)
  - aaron-santos 5y ago
    That's fine, at least you're open about the knowledge for knowledge's sake position. There's more than one way to judge something.
    - xwolfi 5y ago
      It's more than that. Here there's a fake demo we can only consider an advertisement. If they wanted it to be a scientific paper it should be reproducible and counter checked. What is the point of making a paper full of screenshot ? It's not just a knowledge for knowledge's sake issue here, it's that it's not even knowledge they're publishing. They're publishing nothing. They would make a license that says the code can only be provided for peer review and counter validation, then that'd be knowledge. Then, the sake of it is another secondary problem.
ansk 5y ago
You're clearly disillusioned with the general accessibility of ML research, but I don't think your cynicism is warranted here. Take a look at their prior works[1], and I think you'll agree they go above and beyond in making their work accessible and reproducible. There is no reason to doubt the open-source release of this work will be any different. As to why the release is delayed, I'd speculate it's because they put a significant additional amount of work into releases and because releasing code in a large corporation is a bureaucratic hassle. [1] https://nvlabs.github.io/stylegan2/versions.html https://nvlabs.github.io/stylegan2/versions.html
- sillysaurusx 5y ago
  There is no reason to doubt the open-source release of this work will be any different. Then this is not a scientific contribution yet. We must wait and see. The most important tenet of science, is to doubt. I didn’t even read the name on the paper before I wrote my comment. Yes, I know this group. They’re why I got into ML, along with the group from OpenAI who published GPT-2. Because A+ science. Their claims here are likely wrong unless and until proven otherwise. This isn’t a hardline position. It’s been my experience across many codebases, during my two years of trying to reproduce many ideas. I agree that that is an example of A+ science. But why do you think they’re punishing this now, today? Either because conference deadline or because nVidia pressure. Neither of those are related to helping me achieve the scientific method: reproducing the idea in the paper, to verify their claims. All I can do is kind of try to reverse engineer some vague claims in a pdf, without those things. -- Let me tell you a little bit about my job, because my time with my job may soon come to an end. I think that might clear up some confusion. My job, as an ML researcher, is to learn techniques that may or may not be true, combine them in novel ways, and present results to others. Knowledge, Contribution, Presentation, in that order. The first step is to obtain knowledge. Let's set aside the question of why, because why is a question for me personally, which is unrelated. Scientific knowledge comes when Knowledge, Contribution, and Presentation are all achieved in a rigorous way. The rigor allows people like me to verify that I have knowledge. Without this, I have mistaken knowledge, which is worse than useless. It's an illusion – I'm fooling myself. When I got into ML two years ago, I thought that knowledge would come from reading scientific papers. I was wrong. Most papers, are wrong. That's been my experience for the past two years. My experience may be wrong. Maybe others obtain rigorous scientific knowledge through the paper alone. But researchers happen to obtain a dangerous thing: prestige. Unfortunately, prestige doesn't come from helping others obtain knowledge. It comes from that last step -- presentation. The presentation on this thread is excellent. It's another Karras release. I agree; there's no reason to doubt they'll be just as rigorous with this release as they are with stylegan2. But knowledge doesn't come from presentation. Only prestige. Prestige makes a lot of new researchers try very hard to obtain the wrong things. If all of these were small concerns, or curious quirks, they'd be a footnote in my field guide. But I submit that these things are front and center to the current state of affairs in 2021. Every time a release like this happens, it generates a lot of fanfare and we come together in celebration because ML Is Happening, Yay! And then I try to obtain the Knowledge in the fanfare, and discover that either it's absent or mistaken. Because there are no tools for me to verify their claims -- and when I do, I often see that they don't work! That's right. I kept finding out that these things being claimed, just aren't true. No matter how enticing the claim is, or whether it sounds like "Foobars are Aligned in the Convolution Digit," the claim, from where I was sitting, seemed to be wrong. It contained mistaken knowledge -- worse than useless. Unfortunately, two years with no salary takes a toll. I could spend another few years doing this if I wanted to. But I wound up so disgusted with discovering that we're all just chasing prestige, not knowledge, that I'd rather ship production-grade software for the world's most boring commercial work, as long as the work seems useful and the team seems interesting. Because at least I'd be doing something useful.
minimaxir 5y ago
The repo says the code will be available in September: that's a reasonable timeframe for the necessary polish/legal clearance.
- sillysaurusx 5y ago
  (Agreed, fwiw. What’s going on here isn’t a criticism of this work specifically, but the trend of everyone thinking that this is science generally. For example, it’s true the code is coming in September. And, you and I both know it’s probably gonna have a model release, just because it’s more impressive, big-name Karras nVidia work. But it might not have a model release. I give that at least 40% odds. If it doesn’t, then everything I said above will be true about that too, Karras or not. People keep doing that, and we have to call out that this is approximately useless for you and me. Actually, I was going to say maybe it’s useful for you, but you’re the language model hacker and I’m the GAN hacker, and I assure you, code alone is useless for me. If it’s useful to you, I would love to know how it helps you verify the scientific method.)
varispeed 5y ago
I am not into ML, but from time to time I like to look how this is made and remember only once seeing the code and a model, which I thought was exception from the "norm". Good that more people are calling this out!
godelski 5y ago
> I do ML every day. > I go to the github. Maybe model download link is there. I see zero code Paper was released today. Chill. They said they will release the code in September (I'm guessing late September). The paper is also a pre-print. They're probably aiming for CVPR and don't want to get scooped. > Model first. Code second. Paper third. That's how you produce ML code and documentation but that is not how you release it. I guarantee you that they are still tuning and making the model better. They're were still updating ADA till pretty recently (last commit on the pytorch version is 4 months ago, to code). I originally wasn't in CS, and when I first came over I wasn't in ML. We never had code. The fact that ML publishes models AND checkpoints is a godsend. I love it. Makes work so much easier and helps the community advance faster. I love this, but just chill. The paper isn't peer-reviewed. It is a pre-print. They're showing people what they've done in the last 6 months. It's part publicity stunt, part flex, part staking claim, but it is also part sharing with the community. Even without the code we learn a lot because they attached a paper to it. So chill.
- sillysaurusx 5y ago
  The fact that you had to say "chill" three times indicates that you're trying to convince yourself, not me. None of what you said is responsive to what I wrote. I think it's an opinion piece, but I'm not sure. The issue here is the scientific method. I've listed the things that are required, as I see it. And I've also listed the reasons why I haven't been able to verify it exists here, despite trying for two years. I'm glad that you like ML hacking, and I like it too. But models aren't a godsend; they're "the most basic, bare-minimum requirements of reproducibility." Your reaction shouldn't be "I'm incredibly grateful you'd be willing to do this." It should be "You're required to do this, because if I can't verify your claims, your claims might be mistaken." To leave it off on a softer note, normally I'd bond with you, ML hacker to ML hacker. Because I love ML, and I love hearing what you've been up to in ML. It's the best job in the world, as far as I'm concerned. (Could any other career give you the opportunity to be a developer advocate for high-performance computing in such an interesting way? https://github.com/google/jax/issues/2108#issuecomment-866238579 https://github.com/google/jax/issues/2108#issuecomment-86623... Definitely looking for more examples of "Github Larping," if you know of any.) If you agree that the scientific method is the reason ML moves forward, all I'm doing here is protecting it.
pabs3 5y ago
Would you not want the data and code used to train the model, rather than the trained model itself? Edit: the Debian Deep Learning Team's Machine Learning Policy explains why. https://salsa.debian.org/deeplearning-team/ml-policy https://salsa.debian.org/deeplearning-team/ml-policy
- sillysaurusx 5y ago
  For various reasons, I left data out of the requirements because most interesting research uses data unavailable to the community. CLIP is such an example, and it's A+ science: Model, Code, Paper. Having the model is enough to verify the paper's claims, and also to experiment with new approaches (since you can fine-tune the model). That said, I make this concession as a "meet you halfway" compromise between hard-line positions: "We can't release models, because we trained them on private data" and "You must release both models and data." In other words, you're technically correct, but in my estimation it would do more harm to the end goal: the whole reason the scientific method is useful, is because it makes the world more useful. The world would be less useful if fewer commercial companies participated in the scientific method. It's an inclusive group, not an exclusive clique. All you have to do, is give me the tools to verify your claims.
  - pabs3 5y ago
    Yeah, for the goal of verifying claims that is good enough indeed. I'm not sure it is good enough for a variety of other possibly useful use-cases though; eliminating bias in an existing model, correcting a flaw in the training code, creating a different model and proving it is better by training on the same data etc. It would be nice if there were more public/libre data sets for ML stuff.
clircle 5y ago
I hardly understand this comment. Statisticians have been publishing and arguing about models for 100 years now. No one required code to verify authenticity of research. I suppose it is the sorry state of machine learning research that the methodology is so poor that a person cannot verify the research from the paper.
fxtentacle 5y ago
You won't need that much source code to verify their claims. Their central improvement is that they limit the generation of high frequencies by ReLU through a upsample-ReLU-filter-downsample sequence. Their theoretical section explains quite well why high frequencies can be proven mathematically to cause issues. And their practical implementation using filters to cut those off is very straightforward. If someone tells you "The microphone recording had 50Hz noise so I used a filter to remove it", that's pretty much good enough for someone with experience in the field to replicate their results. This is the equivalent in AI. They uncovered a simple basic issue that everyone else overlooked, but once you know it, it seems obvious in retrospect.