3 ms·

OK, You’ve permanently nerd-baited me, and I wish to apply for a job at the Anthropic Factorio lab immediately. I can’t tell from the paper or these comments i

OK, You’ve permanently nerd-baited me, and I wish to apply for a job at the Anthropic Factorio lab immediately. I can’t tell from the paper or these comments if you’re sending multimodal data back — I’m guessing no, because many of these models aren’t multimodal. But some are — and of course we now have recently released Qwen 2.5 VLM which seems to be quite strong for its size. You harp on this lack of spatial ability a fair amount, which - fair enough - and you mention difficulties in both planning and spatial planning. Are you sending images back? If not, any thoughts on this? Thanks for this amazing bit of work, I really am reorganizing my day to play with it now. P.s. seems like MCP enabling the python library is a natural must-do so that all tool-enabled LLMs everywhere can play factorio.

martbakler 2y ago
Currently it's a text-only modality environment but we are planning to support vision in the future. We did run a couple of tests and saw that including screenshots of the game state did not improve performance on the off-the-shelf models. As the complexity of the game state grew and the screenshots were filled with more entities, the models got even more confused and started hallucinating directions, entities etc or weren't capable of troubleshooting factories with apparent mistakes (i.e missing transport belt, wrongly rotated inserter). We think it's because the VLMs currently aren't good at spatial reasoning in high-detailed images, likely this would improve significantly with finetuning Good point with MCP as well given it has been blowing up lately, we'll look into that!
- vessenes 2y ago
  That makes sense and it’s really interesting - it is a challenging visual test for sure; thousands of entities, either multi tier visual representations (screen, map, overview map) or a GIANT high res image. I hereby propose FLE-V a subset benchmark for visual models where they just turn a factorio image into a proper FLE description. And maybe the overview and map images as well.
  - kridsdale1 2y ago
    Such research could have hundreds of billions of dollars in downstream GDP implications when applied to real industrial settings.
    - vessenes 2y ago
      Well I better get training!
    - dismalpedigree 2y ago
      Not to mention the increased productivity of everyone not wasting their time in factorio (myself included) because the optimal solution is known.
      - lukan 2y ago
        Not wasted time, you were doing research it seems.
        dismalpedigree 2y ago
        Good point. My wife will surely understand if I explain it as “research”
  - deleted 2y ago
    [deleted]
- grayhatter 2y ago
  > As the complexity of the game state grew and the screenshots were filled with more entities, the models got even more confused and started hallucinating directions, entities etc or weren't capable of troubleshooting factories with apparent mistakes (i.e missing transport belt, wrongly rotated inserter). We think it's because [...] I think you just described a research paper that would advance sota. Less describing why, but how. (Assuming it's not just, wy finetuned the model and it worked perfectly)
  - martbakler 2y ago
    Sounds almost like a visual "needle in a haystack" type of work, that could be quite interesting!
    - pyinstallwoes 2y ago
      Where’s Waldo test for vlm
jillyboel 2y ago
Why would screenshots be necessary if a textual description of the factory state is both easier to interpret and less prone to confusion? The game is played on a grid, so converting the game state to ascii ought to be trivial.
- vessenes 2y ago
  Trivial as in only engineering work, sure. But it’s a lottt of tokens. Long context models do a number of things to get all that working context in; some of those things elide details / compress / have token segments that are harder to reason about. When a burner inserter at a location takes up like 50-100 tokens, and you want it to reason about 100 of them, this is still a pretty challenging task for any LLM.
  - jillyboel 2y ago
    Ah, I don't know much about multi modal models but I wonder what they'd think of pixel art representing the factory where each pixel is a point on the grid and each color is a specific entity, perhaps ignoring things such as bots flying about. Probably easier to comprehend than an actual screenshot?
    - kridsdale1 2y ago
      I mean at some point you compress the board state down to Dwarf Fortress with an extended ASCII representation for each grid-state (maybe 2 bytes each?)
      - vessenes 2y ago
        Lots of questions here - you need item, orientation, info about pipes (2 directions) , belts (3 or 4 colors x2 directions). Do you wish Circuits?
    - noddybear 2y ago
      The thing is that when you create a dense ASCII representation, any gain you might make from the spatial relationships is lost by: a) the tokeniser not working on characters alone (remember strawberrry), and b) the increased number of 'dead' tokens encoding not very much. Our sparse encoding seems to confuse the models less - even though it certainly isn't perfect.
deleted 2y ago
[deleted]

martbakler 2y ago

Currently it's a text-only modality environment but we are planning to support vision in the future. We did run a couple of tests and saw that including screenshots of the game state did not improve performance on the off-the-shelf models. As the complexity of the game state grew and the screenshots were filled with more entities, the models got even more confused and started hallucinating directions, entities etc or weren't capable of troubleshooting factories with apparent mistakes (i.e missing transport belt, wrongly rotated inserter). We think it's because the VLMs currently aren't good at spatial reasoning in high-detailed images, likely this would improve significantly with finetuning Good point with MCP as well given it has been blowing up lately, we'll look into that!

vessenes 2y ago
That makes sense and it’s really interesting - it is a challenging visual test for sure; thousands of entities, either multi tier visual representations (screen, map, overview map) or a GIANT high res image. I hereby propose FLE-V a subset benchmark for visual models where they just turn a factorio image into a proper FLE description. And maybe the overview and map images as well.
- kridsdale1 2y ago
  Such research could have hundreds of billions of dollars in downstream GDP implications when applied to real industrial settings.
  - vessenes 2y ago
    Well I better get training!
  - dismalpedigree 2y ago
    Not to mention the increased productivity of everyone not wasting their time in factorio (myself included) because the optimal solution is known.
    - lukan 2y ago
      Not wasted time, you were doing research it seems.
      - dismalpedigree 2y ago
        Good point. My wife will surely understand if I explain it as “research”
- deleted 2y ago
  [deleted]
grayhatter 2y ago
> As the complexity of the game state grew and the screenshots were filled with more entities, the models got even more confused and started hallucinating directions, entities etc or weren't capable of troubleshooting factories with apparent mistakes (i.e missing transport belt, wrongly rotated inserter). We think it's because [...] I think you just described a research paper that would advance sota. Less describing why, but how. (Assuming it's not just, wy finetuned the model and it worked perfectly)
- martbakler 2y ago
  Sounds almost like a visual "needle in a haystack" type of work, that could be quite interesting!
  - pyinstallwoes 2y ago
    Where’s Waldo test for vlm

jillyboel 2y ago

Why would screenshots be necessary if a textual description of the factory state is both easier to interpret and less prone to confusion? The game is played on a grid, so converting the game state to ascii ought to be trivial.

vessenes 2y ago
Trivial as in only engineering work, sure. But it’s a lottt of tokens. Long context models do a number of things to get all that working context in; some of those things elide details / compress / have token segments that are harder to reason about. When a burner inserter at a location takes up like 50-100 tokens, and you want it to reason about 100 of them, this is still a pretty challenging task for any LLM.
- jillyboel 2y ago
  Ah, I don't know much about multi modal models but I wonder what they'd think of pixel art representing the factory where each pixel is a point on the grid and each color is a specific entity, perhaps ignoring things such as bots flying about. Probably easier to comprehend than an actual screenshot?
  - kridsdale1 2y ago
    I mean at some point you compress the board state down to Dwarf Fortress with an extended ASCII representation for each grid-state (maybe 2 bytes each?)
    - vessenes 2y ago
      Lots of questions here - you need item, orientation, info about pipes (2 directions) , belts (3 or 4 colors x2 directions). Do you wish Circuits?
  - noddybear 2y ago
    The thing is that when you create a dense ASCII representation, any gain you might make from the spatial relationships is lost by: a) the tokeniser not working on characters alone (remember strawberrry), and b) the increased number of 'dead' tokens encoding not very much. Our sparse encoding seems to confuse the models less - even though it certainly isn't perfect.

deleted 2y ago

[deleted]