35 ms·

That's why I wish there were exams just like high-school ending exams. This way it aint as relevent where you went, but what your score was, so people from the

That's why I wish there were exams just like high-school ending exams. This way it aint as relevent where you went, but what your score was, so people from the "bottom" can bootstrap themselves easier. In India there's such a thing >Graduate Aptitude Test in Engineering (GATE) is an entrance examination conducted in India that primarily tests the comprehensive understanding of undergraduate subjects in engineering and sciences for admission into postgraduate programs.

godelski 3y ago
> That's why I wish there were exams just like high-school ending exams. I'm not sure exams are good at measuring actual abilities. Measuring things is an insanely difficult problem that isn't given enough credit. All too often people think it is just to perform an experiment and measure the outcome but this is really only a tractable when we have a well defined mathematical theory to test our results against. Most often in the real world this is done empirically and in these cases there are loads of assumptions that are being made and often not realized being made. Assumptions that are highly impactful. They come from the data that is being measured as well as the metrics being used to measure them. Remember, metrics are models too, and like all models are "wrong." I'll give two points of evidence to why having a testing system is likely ineffective: 1) These elite schools already have exams that correlate strongly with who is admitted. This gives evidence that exams like SAT and ACT or even the GRE do not do a good job at filtering the issue given the downstream observation. 2) You mention India and the testing structure. Several other countries have similar structures. Personal beliefs aside[0], the fact that India and other countries do this already results in a baked in effect into the US job market. Presuming the effect that I and others are observing is real, then these too look to be ineffective metrics. Since these metrics are likely going to correlate strongly with university rankings as well. [0] Side note, I do not think these systems are good for countries. They appear to be highly ineffective and just promote gate-keeping. They additionally create a lot of strain and pressure on the society because they result in career bottlenecks. Without these systems people have larger opportunities to change later in life. People make highly effective career changes even into their late 30's but these tests are usually performed on people under 18. [1] Other note: meritocracies are flawed at the core due to many effects and this can be quite counterintuitive. First off is the well known Goodhart's Law, where systems optimize the cost function rather than the intended goal of the cost function. This is seen in India's famous Cobra effect, reinforcement (and other ML) learning models, bureaucracies, and so on. This is because the metric is not perfectly aligned with the intention behind the metric, which is often ill-defined and static in nature compared to the dynamic solution space. Aligning metrics is nearly impossible and has only been achieved in the most simple of cases. To learn more take any optimization class, learn about optimal transport and non-convex solutions. Metric hacking is quite common in the wild and we should be really aware of it. This even happens in the precise example we're talking about. I'm sure you've known people who are really good at taking tests but are not so good at generalizing their knowledge (using it in the real world or adapting to new situations). It is because they overfit the cost function. Remember, costs functions are models too. So the question here is about how to properly evaluate things (distinguishing from measuring). Evaluation requires nuance and processing the data and metrics. Most importantly interpreting them (this is where bureaucracies fail and why everyone hates them. Lack of nuance). The truth is that this is always fuzzy and inexact. The only real way to perform this properly is to incorporate noise and uncertainty into your evaluation method. Not to remove it completely, but to acknowledge it and have it play a role in your model. You should of course remove as much noise as possible (reduce), but you can never reduce it to zero and if you remove it from your model you won't be able to capture your uncertainties. These problems only get harder as we move into multi-dimensional data (things we try to measure). High dimensional spaces are highly non-intuitive and the geodesics are not going to be what you expect. This is actually a relatively new area of mathematics and it would be extremely naive to think we have a good understanding of this. (not to knock on the great work that people have done in these areas, but these are fucking hard problems and we haven't had as much time)
- tester756 3y ago
  All I want, is fair and transparent system that allows me to compete with people from $top_school. Why somebody from $top_school should have an advantage just because he/she went there, not that they were better? If I went to $no_name school, but put effort and scored well, then I advertise myself with "scored well", meanwhile now I cant. No system is perfect, but at least there's minimal amount of bias in standardized tests.
  - godelski 3y ago
    Of course want a fair system, but I'm trying to explain why a fair system is quite difficult to create. It isn't about perfection getting in the way of good enough, it is that we've already been trying your suggestion and that's how we've ended up with the system we're at. To be clear, there is a lot of bias in standardized tests. There's also, as we've discussed, not a strong correlation with test performance and work performance. But I don't just preach this, I practice this. When I teach I make my students do projects. The projects are far more important than the single test I have (to fit department suggestions). I give them lots of freedom and let them explore. Basically I treat them like junior engineers but let them pick their projects. Honestly, I see all aspects of students improve through this rather than when I tried testing or homeworks because they are more likely to find motivation. Because memory is based on how useful you think something is. Granted I teach a popular class but still. I can definitely judge a student's performance and how much they've learned FAR better from a project or two than from any amount of testing or homeworks. They just don't even compare in ability to evaluate. Problem is, there's absolutely no way to standardize such a thing. In fact, I'm literally arguing that the problem here is that these aspects are diametrically opposed. The thing is that what sounds nice and "should work" often isn't. It's also often the thing people have already tried. You have to actually embrace the noise and complexity of the environment otherwise you just get hacking. I know it sucks, but you sometimes gotta do things the hard way.
    - tester756 3y ago
      >To be clear, there is a lot of bias in standardized tests Is there? I'd say that there is very, very minimal amount of bias in them. >When I teach I make my students do projects. The projects are far more important than the single test I have Projects are better, but you cannot do them at scale and compare them.

godelski 3y ago

> That's why I wish there were exams just like high-school ending exams. I'm not sure exams are good at measuring actual abilities. Measuring things is an insanely difficult problem that isn't given enough credit. All too often people think it is just to perform an experiment and measure the outcome but this is really only a tractable when we have a well defined mathematical theory to test our results against. Most often in the real world this is done empirically and in these cases there are loads of assumptions that are being made and often not realized being made. Assumptions that are highly impactful. They come from the data that is being measured as well as the metrics being used to measure them. Remember, metrics are models too, and like all models are "wrong." I'll give two points of evidence to why having a testing system is likely ineffective: 1) These elite schools already have exams that correlate strongly with who is admitted. This gives evidence that exams like SAT and ACT or even the GRE do not do a good job at filtering the issue given the downstream observation. 2) You mention India and the testing structure. Several other countries have similar structures. Personal beliefs aside[0], the fact that India and other countries do this already results in a baked in effect into the US job market. Presuming the effect that I and others are observing is real, then these too look to be ineffective metrics. Since these metrics are likely going to correlate strongly with university rankings as well. [0] Side note, I do not think these systems are good for countries. They appear to be highly ineffective and just promote gate-keeping. They additionally create a lot of strain and pressure on the society because they result in career bottlenecks. Without these systems people have larger opportunities to change later in life. People make highly effective career changes even into their late 30's but these tests are usually performed on people under 18. [1] Other note: meritocracies are flawed at the core due to many effects and this can be quite counterintuitive. First off is the well known Goodhart's Law, where systems optimize the cost function rather than the intended goal of the cost function. This is seen in India's famous Cobra effect, reinforcement (and other ML) learning models, bureaucracies, and so on. This is because the metric is not perfectly aligned with the intention behind the metric, which is often ill-defined and static in nature compared to the dynamic solution space. Aligning metrics is nearly impossible and has only been achieved in the most simple of cases. To learn more take any optimization class, learn about optimal transport and non-convex solutions. Metric hacking is quite common in the wild and we should be really aware of it. This even happens in the precise example we're talking about. I'm sure you've known people who are really good at taking tests but are not so good at generalizing their knowledge (using it in the real world or adapting to new situations). It is because they overfit the cost function. Remember, costs functions are models too. So the question here is about how to properly evaluate things (distinguishing from measuring). Evaluation requires nuance and processing the data and metrics. Most importantly interpreting them (this is where bureaucracies fail and why everyone hates them. Lack of nuance). The truth is that this is always fuzzy and inexact. The only real way to perform this properly is to incorporate noise and uncertainty into your evaluation method. Not to remove it completely, but to acknowledge it and have it play a role in your model. You should of course remove as much noise as possible (reduce), but you can never reduce it to zero and if you remove it from your model you won't be able to capture your uncertainties. These problems only get harder as we move into multi-dimensional data (things we try to measure). High dimensional spaces are highly non-intuitive and the geodesics are not going to be what you expect. This is actually a relatively new area of mathematics and it would be extremely naive to think we have a good understanding of this. (not to knock on the great work that people have done in these areas, but these are fucking hard problems and we haven't had as much time)

tester756 3y ago
All I want, is fair and transparent system that allows me to compete with people from $top_school. Why somebody from $top_school should have an advantage just because he/she went there, not that they were better? If I went to $no_name school, but put effort and scored well, then I advertise myself with "scored well", meanwhile now I cant. No system is perfect, but at least there's minimal amount of bias in standardized tests.
- godelski 3y ago
  Of course want a fair system, but I'm trying to explain why a fair system is quite difficult to create. It isn't about perfection getting in the way of good enough, it is that we've already been trying your suggestion and that's how we've ended up with the system we're at. To be clear, there is a lot of bias in standardized tests. There's also, as we've discussed, not a strong correlation with test performance and work performance. But I don't just preach this, I practice this. When I teach I make my students do projects. The projects are far more important than the single test I have (to fit department suggestions). I give them lots of freedom and let them explore. Basically I treat them like junior engineers but let them pick their projects. Honestly, I see all aspects of students improve through this rather than when I tried testing or homeworks because they are more likely to find motivation. Because memory is based on how useful you think something is. Granted I teach a popular class but still. I can definitely judge a student's performance and how much they've learned FAR better from a project or two than from any amount of testing or homeworks. They just don't even compare in ability to evaluate. Problem is, there's absolutely no way to standardize such a thing. In fact, I'm literally arguing that the problem here is that these aspects are diametrically opposed. The thing is that what sounds nice and "should work" often isn't. It's also often the thing people have already tried. You have to actually embrace the noise and complexity of the environment otherwise you just get hacking. I know it sucks, but you sometimes gotta do things the hard way.
  - tester756 3y ago
    >To be clear, there is a lot of bias in standardized tests Is there? I'd say that there is very, very minimal amount of bias in them. >When I teach I make my students do projects. The projects are far more important than the single test I have Projects are better, but you cannot do them at scale and compare them.