5 ms·
Sorry but this is extremely shoddy work. M:tG will never be "solved" like this. >> Using data from drafts carried out by humans, I trained a neural network to
by YeGoblynQueenne 8y ago
Sorry but this is extremely shoddy work. M:tG will never be "solved" like
this.
>> Using data from drafts carried out by humans, I trained a neural network to
predict what card the humans would take out of each pack. It reached 60%
accuracy at this task.
Going by what's in the linked notebook, the model was evaluated on its ability
to match the decks in its training set card-for-card.
Without any attempt to represent game semantics in the model, the fact that
the deck sometimes "predicts" different picks than the actual picks in the
dataset tells us nothing. It probably means the model has some variance that
causes it to make "mistakes" in its attempt to exactly reproduce its dataset.
It certainly doesn't say that the model can draft a good M:tG deck, certainly
not in any set other than Guilds of Ravnica.
>> The model definitely understands the concept of color. In MTG there are 5
colors, and any given draft deck will likely only play cards from 2 or 3 of
those colors. So if you’ve already taken a blue card, you should be more
likely to take blue cards in future picks. We didn’t tell the model about
this, and we also didn’t tell it which cards were which color. But it learned
anyway, by observing which cards were often drafted in combination with each
other.
This is a breathakingly brash misinterpretation of the evidence. The model's
representation of a M:tG card is its index in the Guilds of Ravnica card set.
It has no representation of any card characteristic, including colour. If it
had learned to represent "the concept of colour" in M:tg in this way, it
wouldn't be a neural net, it would be a magick spell.
The author suggests that the model "understands" colour because it drafts
decks of specific colours. Well, its dataset consists of decks with cards of
specific colours. It learned to reproduce those decks. It didn't learn
anything about why those decks pick particular cards, or what particular
cards are. All it has is a list of numbers that it has to learn to put
together in specific ways.
This is as far from "understanding the concept of colour", or anything, as can
be.
There are many more "holes" in the article's logic, that just go to show that
you can train a neural net, but you can't do much with it unless you
understand what you're doing.
Apologies to the author for the harsh critique, if he's reading this.
- YeGoblynQueenne 8y agoJust to add a bit to this: >> Using data from drafts carried out by humans, I trained a neural network to predict what card the humans would take out of each pack. It reached 60% accuracy at this task. And in the 40% when the human and the model differ, the model’s predicted choice is often better than what the human picked. How the model's pick is "better than what the humn picked" is never made clear, but since accuracy is measured by the model's ability to match its training set, I assume that's also what is meant by "better": the model was better than a human in memorising and reproducing the decks it saw during training. Well, you'd never evaluate a human's deckbuilding skills by how well they can reproduce a deck they've seen before. Given the same deck archetype, 10 humans will probably make 10 different card choices, for reasons of their own. It's like trying to evaluate how people style their hair by measuring how similar their hair looks to some examples of particular hair styles. It's a concrete measure, but it's also entirely meaningless. This effort really suffers in terms of evaluation, and so we have learned nothing about how good the model is, which is a shame.
- itsdrewmiller 8y agoThe author was saying that, in their personal playing skill-limited estimation, the model made a stronger pick than the human drafter did. They propose that individuals overrate and underrate cards, but the model collects that and appropriate rates them. But it could also just be that the author is overrating cards - that’s why they asked for other opinions.
- YeGoblynQueenne 8y agoYes, but that's an arbitrary and subjective qualification. It's eyballing - useful as a tool to evaluate your model during development, maybe, but not what you report when you claim the finished model is "better than humans".
- _raoulcousins 8y agoReproducing a deck I've seen before exactly makes no sense in limited. Maybe it's too different of an idea, but in I draft I absolutely evaluate (part of) my skill by how well I've reproduced important components of a good deck. Did I find my seat? Good curve? Enough removal? Then there are format specific things - did I include enough 1/3s for 2, knowing that I'm likely to lose to fast decks with 2/1s if I don't?
- YeGoblynQueenne 8y ago>> Reproducing a deck I've seen before exactly makes no sense in limited. Hi. From your write-up and a quick look at your notebook that's what your model is doing. And you measure its accuracy as its ability to do so. Is that incorrect?
- tveita 8y ago> The author suggests that the model "understands" colour because it drafts decks of specific colours. Well, its dataset consists of decks with cards of specific colours. It learned to reproduce those decks. It didn't learn anything about why those decks pick particular cards, or what particular cards are. All it has is a list of numbers that it has to learn to put together in specific ways. > This is as far from "understanding the concept of colour", or anything, as can be. It is very arguably bad feature engineering - if you have the information readily available, don't make the network infer it - but I think the description is fair. Word2vec uses a similar model. It starts out knowing nothing about each word except an arbitrary numeric index, and learns everything else by predicting words that appear next to each other. By the end of the training it clearly has internal representations of concepts like "color", "verb", "gender", etc. The same concept should apply here - by observing what cards are used in similar decks, with enough training data it should eventually associate concepts like card type, color and mana costs to each card. In this case there isn't enough training data for that kind of resolution, but it has learned that blue cards go with blue cards, and red cards with red cards, and there's no hard lines from there to the concept of color. Sure this isn't going to "solve" MtG, and I don't think it is a particularly good approach for the problem statement, but I think the idea is workable, and the network could already contain a proto-concept of "color" that would be refined with more training.
- deleted 8y ago[deleted]
- YeGoblynQueenne 8y ago>> In this case there isn't enough training data for that kind of resolution, but it has learned that blue cards go with blue cards, and red cards with red cards, and there's no hard lines from there to the concept of color. A card is blue (resp. red, etc) because it has a blue mana symbol in its casting cost. Not because it is found in the company of other blue cards. That is the concept of colour that a model must represent before you can say with any conviction that it "understands" the concept of colour. In terms of "hard lines"- that's the hard line you must cross. The kind of model you're talking about then would be a classifier able to label individual cards with their colours, or an end-to-end model with an internal representation of cards' charactersitics. That is not what was shown here.
- itsdrewmiller 8y agoThis is a lot of anger over a choice of vocabulary. The author certainly didn’t mean that the model had a deep understanding of the nuances of the concept of color - just that it had identified clusters that in real life correspond to the color of cards.
- YeGoblynQueenne 8y agoMy wording is stronger than I intended it (I don't feel that strongly about Magic), but the author did claim that his model "definitely understands the concept of colour", that it is "better than humans" and made several other strong claims besides.