10 ms·

The "overlearning the game" problem

362 points by sendos 15y ago

deleted 15y ago
[deleted]
- sien 15y ago
  The original purpose of Gerrymandering was not to enhance the voting power of minorities, it was to game elections. http://en.wikipedia.org/wiki/Gerrymandering http://en.wikipedia.org/wiki/Gerrymandering Black majority electorates came decades after Gerrymandering. Not a bad article though.
  - sendos 15y ago
    Thanks for the heads-up. I didn't know that Gerrymandering had not-so-pure origins. Maybe I'll remove that example from the article. EDIT: Removed Gerrymandering example.
  - yahelc 15y ago
    +1. Also, while gerrymandering is indeed an abhorrent practice, its not clear it actually leads, in aggregate, to more extreme politicians. Gerrymandering is a delicate balance, and is often designed to create "just safe enough" seats (rule of thumb is 55% projected support), which, it turns out, are rarely safe at the margins in tumultuous times. Indeed, its unclear that gerrymandering is hampering the ability of voters to change their minds. The last 3 elections have all led to historic upheaval, +30D, +23D, and -44R.
kragen 15y ago
Less Wrong talks about this a lot; their term for one variant of it is "superstimuli". What's the difference between your "overlearning" and "hacking"? They sound like the same thing. This phenomenon is the reason for the Wikipedia rule, "Ignore All Rules".
- sendos 15y ago
  > What's the difference between your "overlearning" and "hacking"? They sound like the same thing. They are similar, but I think one difference between "hacking" and "overlearning" is that a hacker can learn to subvert/circumvent a system quite quickly, and usually does so in a way that breaks the rules. Overlearning usually takes longer to achieve, and is a more insidious problem because it achieves its goals without actually breaking any rules.
  - kragen 15y ago
    It depends on what you think "the rules" are. Generally speaking, hacking happens in a framework that has unbreakable rules, whether those are rules of a game, of a CPU, or of an OS. A solution becomes a hack when it exploits those rules in a way that was previously thought impossible. Of course, there may be other, breakable, rules that are broken — "You can't render 3-D in real time on a PC!" — but those aren't the relevant rules. Many hacks are not quick at all to achieve.
    - sendos 15y ago
      I guess an example of hacking would be someone hacking the phone system to make free long distance calls. An example of overlearning would be someone learning the rules of society to such a degree that he gets free long distance calls (e.g. using lawsuits against phone companies). The former would get fined or imprisoned if caught, but the latter is fine under our legal system.
      - kragen 15y ago
        It's pretty bizarre that you're using the derogatory definition of "hack" given that you're posting on Hacker News. And it doesn't incline me to want to talk to you further.
        sendos 15y ago
        Sorry, I was just using the colloquial meaning of hack. Did not at all mean to appear derogatory for the type of hacking exemplified by HN, of which I take part in. In any case, I do see your point about similarities between hacking and overlearning, but I think there are some differences. If I can articulate those difference better, I'll reply a little later.
drhouse_md 15y ago
This article seems to be addressing 'the spirit' of a document or system. Perhaps another way to look at the problem is to imagine creating an A.I. that you want to succeed at whatever system you present. In most cases, an A.I. will take the literal interpretation of the system and become a test-taker, an electable 'gotcha-game' politician or even an entity that finds it can maximize game theory to its own ends by complicating the rules of an existing system to the point of absurdity once it becomes powerful enough to modify and create rules. So then how do we create systems resistant to beings that take everything literally? I suppose the only way is to reward certain outcomes as opposed to rewarding the direct product of the system itself. Examples: After an election, have we elected someone who has met with a high degree of favorability in the electorate by the end of his term? After having students become proficient test-takers, do they then become excellent doctors, lawyers, engineers, etc.. ? In a game invented to be fun or fair, once overlearned, do they produce fun or fairness? If not, then a new game needs to be created or the existing one might need to be extensively modified to produced the desired product. This is where the internet shines, where everyone is welcome to take an existing system and modify it to something better. The problem with politics, law, stock market, etc... is that they have become the only method adopted in real society (there is only one game in town). If reality were allowed to adopt, incorporate or evolve from systems/games from virtual reality (i.e. internet) there might be some productive change. But first we would need to see the first step taking place, that being even flawed virtual systems are allowed to manifest in significant proportion within real life society. Don't hate the game, hate the player. Every game that profits a winner will have its cheaters.
shubber 15y ago
I'm reminded of Mechanism Design: a Nobel prize winning theory of economics that starts with the supposition that agents in any system will exploit its rules to maximize their personal gain. The corollary, which I've been quite taken by, is that where we can influence the rules of the game, we should design them such that exploitation serves a social good.
wisty 15y ago
It's related to Goodhart's Law: Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. This is often the result of attempting to overoptimize a system. You can optimize a race car to a huge degree, because you know exactly what you want it to do. You can't optimize a schooling system, because you don't know exactly what you want it to do. A little noise is a good thing, because the you want a little wiggle room for teachers to sidestep the dictums of education czars, and students to sidestep the dictums of teachers. The Greeks solved this quite a few years ago, with sortition. Under sortition (injecting noise into elections - http://en.wikipedia.org/wiki/Sortition http://en.wikipedia.org/wiki/Sortition), Bush and Gore would have been forced to pay "paper, scissors, rock" for the presidency. Under the US's more pure democracy, they would have been tempted to make all kind of Faustian bargains with sordid players to nail down the last 0.01% of votes. Randomization means that the last percent is just not worth chasing, so players in a competition won't be tempted to bend the rules for a tiny advantage. The same process could be used for tests. If you allocate places in desirable courses (say medicine) randomly to anyone above a certain score, the top students won't bother drilling as hard just to get the top score. Stocks are the same - quants wouldn't sweat timing as much if their placement in order books was randomized. It would be more efficient to pay attention to fundamental value than momentary fluctuations if they weren't guaranteed to make large profits on the momentary fluctuations. Some would still work on timing, but not as many. Patents are just bad policy badly implemented at the moment, not over-optimized.
- tolmasky 15y ago
  Do you have any links on this? I hadn't heard of it before and am quite interested in it now, but could not come up with anything after a quick Google search for "sortation". Thanks for sharing this!
  - wisty 15y ago
    I've edited in a link now. The main disadvantage with sortation seems to be that it killed Socrates. Randomly selected jurors did quite a few nasty and arbitrary things. But an elected tyrant (especially one who was willing to do anything to stay popular) might be worse. It's hard to say that sortation is bad because it allowed mistakes - every political system allows mistakes. Also, systematic slavery, misogyny, the occasional genocide and other discrimination against non-male-citizens (non-voters) was a problem, but that's not really related to sortation. Any democracy can have those problems. Fortunately, we seem to have progressed a little in the last 2000 years (despite the occasional step back).
drewcrawford 15y ago
If you pay a man by the hour, he'll work a lot of hours. If you pay him by the brick, he'll lay a lot of bricks. These "games" are basically the equivalent of counting lines of code or checkins. We're measuring poor proxies instead of the things we're actually interested in. The solution isn't an arms race to build bigger and better proxies, the solution is to measure real things instead of artificial ones. Here's just one example of what I mean by "measure real things". Electing representatives every X years to decide the laws of the land was once upon a time the fairest and best way to have the voices of the masses heard. Today it is feasible to directly poll everybody about every issue, so we no longer need the proxy. If you say everyone cannot be educated about every issue, fine, I can "follow" PG's votes on wall street reform and grellas's votes on IP tort reform and Schneier's votes on TSA etc just by copying their votes on those issues into my ballot, a permission which I can revoke at any time or on a vote-by-vote basis, as easy as unfollowing them on VoteTwitter. This is better than the proxy of professional politicians deciding every issue with fixed terms.
- sendos 15y ago
  I love the idea that electing representatives every X years to decide the laws of the land may not be the best way forward, and a new approach that takes into account modern technology may be a better way to go about this. However, based on human nature, I think that given long enough time of your proposed system in place, some people will overlearn it and find a way to exploit it, e.g. to gather tons of votes for their favorite cause. We may even see 'vote consultants/strategists' that guarantee a huge increase in the votes for the cause you want to pursue.
  - roel_v 15y ago
    Of course, you already see this in indirect form in professional PR firms whose sole job it is to influence public opinion on a narrow issue, and in direct form in places where 'direct voting' is implemented in some form, like in California's referendum system for which on almost each issue specific lobbying groups have a major influence on the outcome. I'd argue that referenda are more even susceptible to the problems described in the article than transparent representative democratic systems are; laymen are easy to influence, and groups of laymen even easier. In the end it depends on the population that is being ruled though - people get the rulers they deserve, and blaming the elected people while still participating in bribery, nepotism etc (while rational! classic prisoners dilemma!) is useless. But of course somebody needs to step up, like the Indian guy is doing now.
jacques_chester 15y ago
Systems theorists say that "structure predicts behaviour". It's a bit trite, but also deep. Here's an example. The US political system pretty much guarantees bad budgeting. Members of Congress are elected fairly independently. There is no party discipline, so each member will operate independently to maximise pork. This encourages horse-trading within and across party lines; nobody can be forced to give up something for a general good. There's more: there's no incentive to balance the budget. The Executive's separation means that Congress does not need to concern itself with proper administration; it only doles out the cash. It has every incentive to ... maximise pork. The dynamic behaviour of American government arises from the static structure. The drafters of the constitution drew on their knowledge of history and current affairs to try and avoid certain pitfalls. The US Constitution was state-of-the-art when it was written. It's less so now. Countries where the Executive is formed out of the Legislative -- the Westminster system -- tend to have much stronger party discipline, because that discipline is required to pass budgets, enact legislation and to form the Executive. This tends to almost eliminate horse-trading, except between parties and independents. It's not perfect -- whole parties can engage in pork too -- but when policy emerges that benefits the many at the cost of a few, countries with party discipline will find it easier to adopt than those without. Australia, which has the amongst the toughest party discipline in the democratic world, is also a reform leader. And I think a lot of that is explained by our constitutional arrangements.
groaner 15y ago
The cynic inside me would extend this problem to the entire economy or even human society itself. Any economic transaction is supposed to benefit both parties: the buyer offers an amount he is willing to pay for what he wants, the seller provides his goods or services at a price he is willing to accept. As PG might say, wealth is created in tandem with creating stuff that people want. The problem is that some elements in society have become extremely effective at creating a perversion of "stuff people want" for their own benefit. They exploit loopholes in the system, whether by preying upon the poorly-informed, shifting costs onto externalities that we can't price properly, engaging in corruption, or producing items of questionable actual value but very attractive perceived value. They take advantage of our desire for easy answers to our problems with minimal effort expended. Case in point: Ponzi schemes, coal power plants, Halliburton, the tobacco industry, Zynga, and even to an extent, religion We've tried to control this with laws, education, and social norms, but ultimately it seems the invisible hand reigns supreme.
Joakal 15y ago
Hey USA, your first past the post system actually does tend towards dual party governments [0]. Compare it to preferential system where voters can pick several parties or vote for one party for the preferences listed. For example, look at this: http://www.abc.net.au/elections/federal/2010/guide/deni.htm http://www.abc.net.au/elections/federal/2010/guide/deni.htm With USA's system, Australia Labor Party would have got re-elected. There's still improvements to be made for democracy that doesn't need to involve such fancy technology yet [1]. [0] http://en.wikipedia.org/wiki/Duvergers_law http://en.wikipedia.org/wiki/Duvergers_law [1] http://techpp.com/2011/06/08/homomorphic-encryption/ http://techpp.com/2011/06/08/homomorphic-encryption/
loup-vaillant 15y ago
I think "overlearning" is not a good term (edit: it is quite accurate, though), because it tend to suggest the worst solution of all: that we refrain from learning. I prefer "lost purposes"[1]. The primary purpose of a game is generally to Have Fun. This purpose is lost if you have "solved" the game. The stated purpose of patents is to foster innovation. However it doesn't work[2]. The purpose of schools is learning. However its methods are flawed [3,4]. And so on. The trick is to know your goals, and then find out means to best achieve them. The author said: > But, in real life, we need to keep "playing the game": we need to have elections, and protections for inventors, and laws that govern society, and a market where companies can raise money. But the actual goals are different: We don't need election, we need a working democratic system (which may, or may not, mandate elections). We don't need protections for inventors, we need innovations. We don't need laws, we need a fair and working society. We don't need a market where companies can raise money, we need a working economy. Well, I could attempt recurse further up until pure morality, but that would be intractable. But at least you get the idea. If something looks broken, think about its ultimate purpose before you try to fix it. [1]: http://lesswrong.com/lw/le/lost_purposes/ http://lesswrong.com/lw/le/lost_purposes/ [2]: http://www.dklevine.com/general/intellectual/against.htm http://www.dklevine.com/general/intellectual/against.htm [3]: http://vimeo.com/5513063 http://vimeo.com/5513063 (Dr Tae) [4]: http://www.khanacademy.org/ http://www.khanacademy.org/
- jorleif 15y ago
  Overlearning/Overfitting is a term from machine learning/statistics that means that a learning system fits its learning data so well that it does not generalize to new data. The examples given in the article seem like perfect examples of this, e.g. elected officials fit the election process very well, but do not "generalize" to leading a country.
  - loup-vaillant 15y ago
    Ah. I didn't see it as jargon. I recognize the term is accurate, but my main point still stands: the connotation is still there, and still does suggest we should learn less about those systems if we want to preserve their original purpose. Worse, the end of the post suggest we have to preserve things like schools, markets and elections, even though we learned enough to spoil them. Therefore, the only thing left to sacrifice is learning itself. That would really really sucks: I'm curious, and I always want to learn more. There are many things I would sacrifice before I accept to put hard limits on what I am allowed to learn.
donaq 15y ago
Regarding politics, it is interesting to note that the people have not, for their part, also overlearned the game of electing the best man for the job. I wonder why this asymmetry exists?
erikb 15y ago
Many people see this from a macroeconomic or political point of view. So I think adding a more philosophical point of view might also be of value. Of course, because it is philosophy, I can only present my own point of view. There is no right or wrong. For me, as a Zen student, the solution is to just accept that that it is, what it is. No system (shape, for Zen students) can be perfect. It is created to solve a problem and with time starts to fail badly at doing anything about the problem. Then another system is created by someone else and the cycle restarts. You might think for example that the stock market or democracy is a thing that doesn't change, because it exists longer then you live. But in the end it will change. We had different political systems before and we will develop different systems in the future. So in the end the system is one of the most important things we have, because it gives us something to base our decisions and actions on and goals to strive for. But also the system is nothing, just an illusion we create for us, maybe based on how we understand the illusions other people created for themself. The thing that is interesting to me personally is that every system itself is instable and will change or die in the end. But the life cycle of a system, what it does for us and doesn't, that all will always stay the same and even though we try to change that, we will never succeed. So while it changes a lot in one way it is totally unchangable in another. But that just as a side note.
resatori 15y ago
I think it is time we recognize that it is not the systems that shape our world but the individuals. There is no optimal system, be it political, economic or whatever. You can always find loopholes. I think it is time we take responsibility for what we do - then there is no need for a better system. The patent system is not responsible for people attacking each other - its the people.
DanielBMarkham 15y ago
We run into this situation in programming quite a bit. The answer is refactoring: taking the principles and patterns that work, streamlining them, ditching the cruft that works for a few edge cases but mostly gets in the way, simplifying the underlying metaphor of the system, and re-asssembling. The only way to do this with much larger systems, such as systems of governance, is revolution or exploring new lands. Personally I'm a bit concerned about revolution -- the assumption with refactoring is that the people refactoring understand what the "good" parts are and what the "bad" parts are. They also need to be able to generalize and simplify in order to keep the system cognitively approachable. In my experience, it's very easy to be angry-tear-down-the-system guy, very difficult to actually refactor. As the author points out, it's not that these systems are entirely useless. The hell of the thing is that the reasons for creating these systems are still very valid. I remain convinced that programmers (hackers) have a lot to add to the discussion when talking about complex, brittle systems. After all, we spend a lot of time both working with them and fixing them. To me, programming and systems architecture is applied philosophy. Very cool stuff.
- jacques_chester 15y ago
  Refactoring is what political types call 'reform'. Reformations are safer than Revolutions, because revolutions dissolve all civil structure. They allow the ruthless sociopath to prosper. What programming has taught me is humility in the face of complexity. Refactoring only works because we can, at a stroke, reverse our changes; because we can place those changes in isolation and test the heck out of them. But society, constitutions, laws: they are not software systems. A revolution can't be rolled back. A constitutional change is a hotfix applied to a running system. It's important to be careful. You might find reading Burke on the French Revolution interesting; also Oakeshott on conservatism (the intellectual kind, not the god-n-guns kind).
  - DanielBMarkham 15y ago
    I think we completely agree. I did not mean to imply anything but humility was indicated here. "Reform" is a big, fuzzy word. A lot more brittleness and complexity has been added under that rubric. Systems of governance eventually get "over learned", that is, people adapt to the system and use it in more ways than it was intended. The ability of hundreds of millions of people to adapt is much greater than any amount of complexity that we can create. Yet we persist in the idea that by continuing to "tweak" the system, somehow magic will come from it. To me this is where humility comes in. We can never create a perfect government. The best we can do is have one that balances stability and refactoring. Election cycles and constitutional amendment processes were supposed to do this. What we've found, though, is that legal precedent, statutory code, and an expansion of pure democracy and the role of career politicians, given enough time, puts us exactly where we didn't want to go. Interesting side note: overly complex and brittle systems that people execute actually give dictatorial powers to those responsible for executing them. That is, in a computer system, the complexity creates program crashes and dysfunction, because the computer has to treat each instruction cycle and piece of data the same. In a system executed by people, a subtle social goal takes over: things still appear to happen. The system still superficially looks like it is working somewhat -- those in authority just selectively apply and bend the rules depending on whatever their whims are. If you've ever worked in a large organization with too much process, you've easily observed this: most struggle under burdensome rules, never being able to get anything done. A few, however, work completely outside the system, given special permission. These folks are usually kept off the radar so as not to upset the troops. Complex people systems don't ever stop working, they just make things miserable on most everybody and give a few permission to do whatever they want. You'd think that modern tyrannies would look like old Libya: one guy in a funny hat ruling totally. But that's not the way it works any more. It's not one guy, it's a distributed oligarchy, and it's not out in the open, it's obfuscated.
aconbere 15y ago
There was a book written in the 80's that covered the overarching philosophical discussion on this kind of thinking about games. The author broke games into two categories: finite and infinite. Finite games like the one invented by the children described by the OP have end states, they have winners, and the goal within them are always framed from that perspective. While infinite games have no winners, have no end, and the goals are often framed around ensuring that the game never ends. It's easy to think of finite games in our lives, they are everywhere in our society, and the OP does a good job of pointing out some of the less obvious cases. Infinite games are less obvious, the one that I found most illuminating was "language", a game where people actively collude to extend the game to the end of time, and has no winner. Anyay, it's an interesting read, but a little bit mumbo-jumbo by the middle of the book. I would recommend it. http://www.amazon.com/Finite-Infinite-Games-Vision-Possibility/dp/0345341848/ref=sr_1_1?ie=UTF8&qid=1314621221&sr=8-1 http://www.amazon.com/Finite-Infinite-Games-Vision-Possibili...
stretchwithme 15y ago
Seems most of the things mentioned are an inevitable result of centralizing and expanding government power. Someone is being gamed alright.
Jun8 15y ago
Very interesting post! However, I got hooked on thinking about the childhood game that he mentions briefly in the introduction. Asking more and more questions about an object and wanting all of them to be consistent is a good description of mathematics. According to Godel, both teams are bound to lose, because you cannot create a system of descriptions/properties about a system that are self-consistent (as their number increases), there will always be questions to a team whose answer will be inconsistent with the previous set of questions. The game is then to see which team can push the inevitable further. IDEABOLT: It would be interesting to develop a program that plays this game. Each answer could be stored as an RDF statement in database.
One_adm12 15y ago
Great post. I face this everyday in my job at an "old media" company. There are policies, rules and constraints in place which served some purpose at some point in time, and haven't been challenged for years. Unlearning the rules of established games is almost as important as evolving them. Everything should be questioned and challenged to find the root of the point of the rule/law/restriction and if it doesn't make sense any longer, throw it out.
One_adm12 15y ago
Great post. I face this everyday in my job at an "old media" company. There are policies, rules and constraints in place which served some purpose at some point in time, and haven't been challenged for years. Unlearning the rules of established games is almost as important as evolving them. Everything should be questioned and challenged to find the root of the point of the rule/law/restriction and if it doesn't make sense any longer, throw it out.
useflyer 15y ago
Have you noticed that yesterday every comment was concise, and now that applications are open, every comment is elaborate, long-winded, and footnoted?
Greedy_Fools 15y ago
People are predators and that's what predators do, never stop searching for what they need. Predatory behavior is the problem, and there is little will in successful predators to outlaw there own behavior.
thewisedude 15y ago
Doesn't "Exploiting the System" capture the same idea? I think basically what people are doing here is understanding the core weakness that is intrinsic to the system(which probably is not perfect) and using(abusing) it to their advantage.

362 points by sendos 15y ago

deleted 15y ago

[deleted]

sien 15y ago
The original purpose of Gerrymandering was not to enhance the voting power of minorities, it was to game elections. http://en.wikipedia.org/wiki/Gerrymandering http://en.wikipedia.org/wiki/Gerrymandering Black majority electorates came decades after Gerrymandering. Not a bad article though.
- sendos 15y ago
  Thanks for the heads-up. I didn't know that Gerrymandering had not-so-pure origins. Maybe I'll remove that example from the article. EDIT: Removed Gerrymandering example.
- yahelc 15y ago
  +1. Also, while gerrymandering is indeed an abhorrent practice, its not clear it actually leads, in aggregate, to more extreme politicians. Gerrymandering is a delicate balance, and is often designed to create "just safe enough" seats (rule of thumb is 55% projected support), which, it turns out, are rarely safe at the margins in tumultuous times. Indeed, its unclear that gerrymandering is hampering the ability of voters to change their minds. The last 3 elections have all led to historic upheaval, +30D, +23D, and -44R.

kragen 15y ago

Less Wrong talks about this a lot; their term for one variant of it is "superstimuli". What's the difference between your "overlearning" and "hacking"? They sound like the same thing. This phenomenon is the reason for the Wikipedia rule, "Ignore All Rules".

sendos 15y ago
> What's the difference between your "overlearning" and "hacking"? They sound like the same thing. They are similar, but I think one difference between "hacking" and "overlearning" is that a hacker can learn to subvert/circumvent a system quite quickly, and usually does so in a way that breaks the rules. Overlearning usually takes longer to achieve, and is a more insidious problem because it achieves its goals without actually breaking any rules.
- kragen 15y ago
  It depends on what you think "the rules" are. Generally speaking, hacking happens in a framework that has unbreakable rules, whether those are rules of a game, of a CPU, or of an OS. A solution becomes a hack when it exploits those rules in a way that was previously thought impossible. Of course, there may be other, breakable, rules that are broken — "You can't render 3-D in real time on a PC!" — but those aren't the relevant rules. Many hacks are not quick at all to achieve.
  - sendos 15y ago
    I guess an example of hacking would be someone hacking the phone system to make free long distance calls. An example of overlearning would be someone learning the rules of society to such a degree that he gets free long distance calls (e.g. using lawsuits against phone companies). The former would get fined or imprisoned if caught, but the latter is fine under our legal system.
    - kragen 15y ago
      It's pretty bizarre that you're using the derogatory definition of "hack" given that you're posting on Hacker News. And it doesn't incline me to want to talk to you further.
      - sendos 15y ago
        Sorry, I was just using the colloquial meaning of hack. Did not at all mean to appear derogatory for the type of hacking exemplified by HN, of which I take part in. In any case, I do see your point about similarities between hacking and overlearning, but I think there are some differences. If I can articulate those difference better, I'll reply a little later.

drhouse_md 15y ago

This article seems to be addressing 'the spirit' of a document or system. Perhaps another way to look at the problem is to imagine creating an A.I. that you want to succeed at whatever system you present. In most cases, an A.I. will take the literal interpretation of the system and become a test-taker, an electable 'gotcha-game' politician or even an entity that finds it can maximize game theory to its own ends by complicating the rules of an existing system to the point of absurdity once it becomes powerful enough to modify and create rules. So then how do we create systems resistant to beings that take everything literally? I suppose the only way is to reward certain outcomes as opposed to rewarding the direct product of the system itself. Examples: After an election, have we elected someone who has met with a high degree of favorability in the electorate by the end of his term? After having students become proficient test-takers, do they then become excellent doctors, lawyers, engineers, etc.. ? In a game invented to be fun or fair, once overlearned, do they produce fun or fairness? If not, then a new game needs to be created or the existing one might need to be extensively modified to produced the desired product. This is where the internet shines, where everyone is welcome to take an existing system and modify it to something better. The problem with politics, law, stock market, etc... is that they have become the only method adopted in real society (there is only one game in town). If reality were allowed to adopt, incorporate or evolve from systems/games from virtual reality (i.e. internet) there might be some productive change. But first we would need to see the first step taking place, that being even flawed virtual systems are allowed to manifest in significant proportion within real life society. Don't hate the game, hate the player. Every game that profits a winner will have its cheaters.

shubber 15y ago

I'm reminded of Mechanism Design: a Nobel prize winning theory of economics that starts with the supposition that agents in any system will exploit its rules to maximize their personal gain. The corollary, which I've been quite taken by, is that where we can influence the rules of the game, we should design them such that exploitation serves a social good.

wisty 15y ago

It's related to Goodhart's Law: Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. This is often the result of attempting to overoptimize a system. You can optimize a race car to a huge degree, because you know exactly what you want it to do. You can't optimize a schooling system, because you don't know exactly what you want it to do. A little noise is a good thing, because the you want a little wiggle room for teachers to sidestep the dictums of education czars, and students to sidestep the dictums of teachers. The Greeks solved this quite a few years ago, with sortition. Under sortition (injecting noise into elections - http://en.wikipedia.org/wiki/Sortition http://en.wikipedia.org/wiki/Sortition), Bush and Gore would have been forced to pay "paper, scissors, rock" for the presidency. Under the US's more pure democracy, they would have been tempted to make all kind of Faustian bargains with sordid players to nail down the last 0.01% of votes. Randomization means that the last percent is just not worth chasing, so players in a competition won't be tempted to bend the rules for a tiny advantage. The same process could be used for tests. If you allocate places in desirable courses (say medicine) randomly to anyone above a certain score, the top students won't bother drilling as hard just to get the top score. Stocks are the same - quants wouldn't sweat timing as much if their placement in order books was randomized. It would be more efficient to pay attention to fundamental value than momentary fluctuations if they weren't guaranteed to make large profits on the momentary fluctuations. Some would still work on timing, but not as many. Patents are just bad policy badly implemented at the moment, not over-optimized.

tolmasky 15y ago
Do you have any links on this? I hadn't heard of it before and am quite interested in it now, but could not come up with anything after a quick Google search for "sortation". Thanks for sharing this!
- wisty 15y ago
  I've edited in a link now. The main disadvantage with sortation seems to be that it killed Socrates. Randomly selected jurors did quite a few nasty and arbitrary things. But an elected tyrant (especially one who was willing to do anything to stay popular) might be worse. It's hard to say that sortation is bad because it allowed mistakes - every political system allows mistakes. Also, systematic slavery, misogyny, the occasional genocide and other discrimination against non-male-citizens (non-voters) was a problem, but that's not really related to sortation. Any democracy can have those problems. Fortunately, we seem to have progressed a little in the last 2000 years (despite the occasional step back).

drewcrawford 15y ago

If you pay a man by the hour, he'll work a lot of hours. If you pay him by the brick, he'll lay a lot of bricks. These "games" are basically the equivalent of counting lines of code or checkins. We're measuring poor proxies instead of the things we're actually interested in. The solution isn't an arms race to build bigger and better proxies, the solution is to measure real things instead of artificial ones. Here's just one example of what I mean by "measure real things". Electing representatives every X years to decide the laws of the land was once upon a time the fairest and best way to have the voices of the masses heard. Today it is feasible to directly poll everybody about every issue, so we no longer need the proxy. If you say everyone cannot be educated about every issue, fine, I can "follow" PG's votes on wall street reform and grellas's votes on IP tort reform and Schneier's votes on TSA etc just by copying their votes on those issues into my ballot, a permission which I can revoke at any time or on a vote-by-vote basis, as easy as unfollowing them on VoteTwitter. This is better than the proxy of professional politicians deciding every issue with fixed terms.

sendos 15y ago
I love the idea that electing representatives every X years to decide the laws of the land may not be the best way forward, and a new approach that takes into account modern technology may be a better way to go about this. However, based on human nature, I think that given long enough time of your proposed system in place, some people will overlearn it and find a way to exploit it, e.g. to gather tons of votes for their favorite cause. We may even see 'vote consultants/strategists' that guarantee a huge increase in the votes for the cause you want to pursue.
- roel_v 15y ago
  Of course, you already see this in indirect form in professional PR firms whose sole job it is to influence public opinion on a narrow issue, and in direct form in places where 'direct voting' is implemented in some form, like in California's referendum system for which on almost each issue specific lobbying groups have a major influence on the outcome. I'd argue that referenda are more even susceptible to the problems described in the article than transparent representative democratic systems are; laymen are easy to influence, and groups of laymen even easier. In the end it depends on the population that is being ruled though - people get the rulers they deserve, and blaming the elected people while still participating in bribery, nepotism etc (while rational! classic prisoners dilemma!) is useless. But of course somebody needs to step up, like the Indian guy is doing now.

jacques_chester 15y ago

Systems theorists say that "structure predicts behaviour". It's a bit trite, but also deep. Here's an example. The US political system pretty much guarantees bad budgeting. Members of Congress are elected fairly independently. There is no party discipline, so each member will operate independently to maximise pork. This encourages horse-trading within and across party lines; nobody can be forced to give up something for a general good. There's more: there's no incentive to balance the budget. The Executive's separation means that Congress does not need to concern itself with proper administration; it only doles out the cash. It has every incentive to ... maximise pork. The dynamic behaviour of American government arises from the static structure. The drafters of the constitution drew on their knowledge of history and current affairs to try and avoid certain pitfalls. The US Constitution was state-of-the-art when it was written. It's less so now. Countries where the Executive is formed out of the Legislative -- the Westminster system -- tend to have much stronger party discipline, because that discipline is required to pass budgets, enact legislation and to form the Executive. This tends to almost eliminate horse-trading, except between parties and independents. It's not perfect -- whole parties can engage in pork too -- but when policy emerges that benefits the many at the cost of a few, countries with party discipline will find it easier to adopt than those without. Australia, which has the amongst the toughest party discipline in the democratic world, is also a reform leader. And I think a lot of that is explained by our constitutional arrangements.

groaner 15y ago

The cynic inside me would extend this problem to the entire economy or even human society itself. Any economic transaction is supposed to benefit both parties: the buyer offers an amount he is willing to pay for what he wants, the seller provides his goods or services at a price he is willing to accept. As PG might say, wealth is created in tandem with creating stuff that people want. The problem is that some elements in society have become extremely effective at creating a perversion of "stuff people want" for their own benefit. They exploit loopholes in the system, whether by preying upon the poorly-informed, shifting costs onto externalities that we can't price properly, engaging in corruption, or producing items of questionable actual value but very attractive perceived value. They take advantage of our desire for easy answers to our problems with minimal effort expended. Case in point: Ponzi schemes, coal power plants, Halliburton, the tobacco industry, Zynga, and even to an extent, religion We've tried to control this with laws, education, and social norms, but ultimately it seems the invisible hand reigns supreme.

Joakal 15y ago

Hey USA, your first past the post system actually does tend towards dual party governments [0]. Compare it to preferential system where voters can pick several parties or vote for one party for the preferences listed. For example, look at this: http://www.abc.net.au/elections/federal/2010/guide/deni.htm http://www.abc.net.au/elections/federal/2010/guide/deni.htm With USA's system, Australia Labor Party would have got re-elected. There's still improvements to be made for democracy that doesn't need to involve such fancy technology yet [1]. [0] http://en.wikipedia.org/wiki/Duvergers_law http://en.wikipedia.org/wiki/Duvergers_law [1] http://techpp.com/2011/06/08/homomorphic-encryption/ http://techpp.com/2011/06/08/homomorphic-encryption/

loup-vaillant 15y ago

I think "overlearning" is not a good term (edit: it is quite accurate, though), because it tend to suggest the worst solution of all: that we refrain from learning. I prefer "lost purposes"[1]. The primary purpose of a game is generally to Have Fun. This purpose is lost if you have "solved" the game. The stated purpose of patents is to foster innovation. However it doesn't work[2]. The purpose of schools is learning. However its methods are flawed [3,4]. And so on. The trick is to know your goals, and then find out means to best achieve them. The author said: > But, in real life, we need to keep "playing the game": we need to have elections, and protections for inventors, and laws that govern society, and a market where companies can raise money. But the actual goals are different: We don't need election, we need a working democratic system (which may, or may not, mandate elections). We don't need protections for inventors, we need innovations. We don't need laws, we need a fair and working society. We don't need a market where companies can raise money, we need a working economy. Well, I could attempt recurse further up until pure morality, but that would be intractable. But at least you get the idea. If something looks broken, think about its ultimate purpose before you try to fix it. [1]: http://lesswrong.com/lw/le/lost_purposes/ http://lesswrong.com/lw/le/lost_purposes/ [2]: http://www.dklevine.com/general/intellectual/against.htm http://www.dklevine.com/general/intellectual/against.htm [3]: http://vimeo.com/5513063 http://vimeo.com/5513063 (Dr Tae) [4]: http://www.khanacademy.org/ http://www.khanacademy.org/

jorleif 15y ago
Overlearning/Overfitting is a term from machine learning/statistics that means that a learning system fits its learning data so well that it does not generalize to new data. The examples given in the article seem like perfect examples of this, e.g. elected officials fit the election process very well, but do not "generalize" to leading a country.
- loup-vaillant 15y ago
  Ah. I didn't see it as jargon. I recognize the term is accurate, but my main point still stands: the connotation is still there, and still does suggest we should learn less about those systems if we want to preserve their original purpose. Worse, the end of the post suggest we have to preserve things like schools, markets and elections, even though we learned enough to spoil them. Therefore, the only thing left to sacrifice is learning itself. That would really really sucks: I'm curious, and I always want to learn more. There are many things I would sacrifice before I accept to put hard limits on what I am allowed to learn.

donaq 15y ago

Regarding politics, it is interesting to note that the people have not, for their part, also overlearned the game of electing the best man for the job. I wonder why this asymmetry exists?

erikb 15y ago

Many people see this from a macroeconomic or political point of view. So I think adding a more philosophical point of view might also be of value. Of course, because it is philosophy, I can only present my own point of view. There is no right or wrong. For me, as a Zen student, the solution is to just accept that that it is, what it is. No system (shape, for Zen students) can be perfect. It is created to solve a problem and with time starts to fail badly at doing anything about the problem. Then another system is created by someone else and the cycle restarts. You might think for example that the stock market or democracy is a thing that doesn't change, because it exists longer then you live. But in the end it will change. We had different political systems before and we will develop different systems in the future. So in the end the system is one of the most important things we have, because it gives us something to base our decisions and actions on and goals to strive for. But also the system is nothing, just an illusion we create for us, maybe based on how we understand the illusions other people created for themself. The thing that is interesting to me personally is that every system itself is instable and will change or die in the end. But the life cycle of a system, what it does for us and doesn't, that all will always stay the same and even though we try to change that, we will never succeed. So while it changes a lot in one way it is totally unchangable in another. But that just as a side note.

resatori 15y ago

I think it is time we recognize that it is not the systems that shape our world but the individuals. There is no optimal system, be it political, economic or whatever. You can always find loopholes. I think it is time we take responsibility for what we do - then there is no need for a better system. The patent system is not responsible for people attacking each other - its the people.

DanielBMarkham 15y ago

We run into this situation in programming quite a bit. The answer is refactoring: taking the principles and patterns that work, streamlining them, ditching the cruft that works for a few edge cases but mostly gets in the way, simplifying the underlying metaphor of the system, and re-asssembling. The only way to do this with much larger systems, such as systems of governance, is revolution or exploring new lands. Personally I'm a bit concerned about revolution -- the assumption with refactoring is that the people refactoring understand what the "good" parts are and what the "bad" parts are. They also need to be able to generalize and simplify in order to keep the system cognitively approachable. In my experience, it's very easy to be angry-tear-down-the-system guy, very difficult to actually refactor. As the author points out, it's not that these systems are entirely useless. The hell of the thing is that the reasons for creating these systems are still very valid. I remain convinced that programmers (hackers) have a lot to add to the discussion when talking about complex, brittle systems. After all, we spend a lot of time both working with them and fixing them. To me, programming and systems architecture is applied philosophy. Very cool stuff.

jacques_chester 15y ago
Refactoring is what political types call 'reform'. Reformations are safer than Revolutions, because revolutions dissolve all civil structure. They allow the ruthless sociopath to prosper. What programming has taught me is humility in the face of complexity. Refactoring only works because we can, at a stroke, reverse our changes; because we can place those changes in isolation and test the heck out of them. But society, constitutions, laws: they are not software systems. A revolution can't be rolled back. A constitutional change is a hotfix applied to a running system. It's important to be careful. You might find reading Burke on the French Revolution interesting; also Oakeshott on conservatism (the intellectual kind, not the god-n-guns kind).
- DanielBMarkham 15y ago
  I think we completely agree. I did not mean to imply anything but humility was indicated here. "Reform" is a big, fuzzy word. A lot more brittleness and complexity has been added under that rubric. Systems of governance eventually get "over learned", that is, people adapt to the system and use it in more ways than it was intended. The ability of hundreds of millions of people to adapt is much greater than any amount of complexity that we can create. Yet we persist in the idea that by continuing to "tweak" the system, somehow magic will come from it. To me this is where humility comes in. We can never create a perfect government. The best we can do is have one that balances stability and refactoring. Election cycles and constitutional amendment processes were supposed to do this. What we've found, though, is that legal precedent, statutory code, and an expansion of pure democracy and the role of career politicians, given enough time, puts us exactly where we didn't want to go. Interesting side note: overly complex and brittle systems that people execute actually give dictatorial powers to those responsible for executing them. That is, in a computer system, the complexity creates program crashes and dysfunction, because the computer has to treat each instruction cycle and piece of data the same. In a system executed by people, a subtle social goal takes over: things still appear to happen. The system still superficially looks like it is working somewhat -- those in authority just selectively apply and bend the rules depending on whatever their whims are. If you've ever worked in a large organization with too much process, you've easily observed this: most struggle under burdensome rules, never being able to get anything done. A few, however, work completely outside the system, given special permission. These folks are usually kept off the radar so as not to upset the troops. Complex people systems don't ever stop working, they just make things miserable on most everybody and give a few permission to do whatever they want. You'd think that modern tyrannies would look like old Libya: one guy in a funny hat ruling totally. But that's not the way it works any more. It's not one guy, it's a distributed oligarchy, and it's not out in the open, it's obfuscated.

aconbere 15y ago

There was a book written in the 80's that covered the overarching philosophical discussion on this kind of thinking about games. The author broke games into two categories: finite and infinite. Finite games like the one invented by the children described by the OP have end states, they have winners, and the goal within them are always framed from that perspective. While infinite games have no winners, have no end, and the goals are often framed around ensuring that the game never ends. It's easy to think of finite games in our lives, they are everywhere in our society, and the OP does a good job of pointing out some of the less obvious cases. Infinite games are less obvious, the one that I found most illuminating was "language", a game where people actively collude to extend the game to the end of time, and has no winner. Anyay, it's an interesting read, but a little bit mumbo-jumbo by the middle of the book. I would recommend it. http://www.amazon.com/Finite-Infinite-Games-Vision-Possibility/dp/0345341848/ref=sr_1_1?ie=UTF8&qid=1314621221&sr=8-1 http://www.amazon.com/Finite-Infinite-Games-Vision-Possibili...

stretchwithme 15y ago

Seems most of the things mentioned are an inevitable result of centralizing and expanding government power. Someone is being gamed alright.

Jun8 15y ago

Very interesting post! However, I got hooked on thinking about the childhood game that he mentions briefly in the introduction. Asking more and more questions about an object and wanting all of them to be consistent is a good description of mathematics. According to Godel, both teams are bound to lose, because you cannot create a system of descriptions/properties about a system that are self-consistent (as their number increases), there will always be questions to a team whose answer will be inconsistent with the previous set of questions. The game is then to see which team can push the inevitable further. IDEABOLT: It would be interesting to develop a program that plays this game. Each answer could be stored as an RDF statement in database.

One_adm12 15y ago

Great post. I face this everyday in my job at an "old media" company. There are policies, rules and constraints in place which served some purpose at some point in time, and haven't been challenged for years. Unlearning the rules of established games is almost as important as evolving them. Everything should be questioned and challenged to find the root of the point of the rule/law/restriction and if it doesn't make sense any longer, throw it out.

One_adm12 15y ago

useflyer 15y ago

Have you noticed that yesterday every comment was concise, and now that applications are open, every comment is elaborate, long-winded, and footnoted?

Greedy_Fools 15y ago

People are predators and that's what predators do, never stop searching for what they need. Predatory behavior is the problem, and there is little will in successful predators to outlaw there own behavior.

thewisedude 15y ago

Doesn't "Exploiting the System" capture the same idea? I think basically what people are doing here is understanding the core weakness that is intrinsic to the system(which probably is not perfect) and using(abusing) it to their advantage.