noddybear

6 ms·

31.
by noddybear 2y ago
These models were not post-trained - all off-the-shelf. We can fit about 128 pairs maximum in the context, but this performed the same as 32, which we ultimately decided on (for cost, latency purposes). Encoding the input/outputs to ma
32.
by noddybear 2y ago
We were able to pass all the early lab tasks manually - although it took a lot longer than using the UI!
33.
by noddybear 2y ago
We’re still waiting for arXiv, but the pdf is at the top.
34.
by noddybear 2y ago
This is exactly how FLE works, the agent writes a program that executes its policy. I think you bring up a good point, we could create tasks where the goal is to optimise a static factory, starting from a kernel of functionality like '
35.
by noddybear 2y ago
In FLE, you have access to milestones representing the first time a new entity was created, but coming up with a stratification of rewards for different degrees of automation would be really interesting. Join us!
36.
by noddybear 2y ago
Regarding the 2d image - the issue is that these frontier models don't tend to support supplemental image inputs, and the ones that do aren't sufficiently well trained on (high precision) Factorio visuals to add that much informat
37.
by noddybear 2y ago
This is really interesting, do you have a repo or anything describing the approach? I would be particularly interested in trying your approach in FLE to see how it affects layout design. How are you performing the spatial reasoning?
38.
by noddybear 2y ago
Thats really cool! Please shoot me an email if you get around to doing this, always happy to assist however I can.
39.
by noddybear 2y ago
I think generating the scenarios as you suggest (in text) is easy, but creating correct factory game states to start from is a lot harder. AFAIK it reduces into the same manual task of designing an init state and a task to complete.
40.
by noddybear 2y ago
The thing is that when you create a dense ASCII representation, any gain you might make from the spatial relationships is lost by: a) the tokeniser not working on characters alone (remember strawberrry), and b) the increased number of &#x27
41.
by noddybear 2y ago
The models have some understanding of the game and the initial build orders that they should adopt. What they don't remember is: 1) How many resources each item costs to make 2) Spatial gotchas such as a belt requiring an inserter to l
42.
by noddybear 2y ago
True - although it might be interesting to benchmark them both, as (1) is more about debugging (something that these agents spend a lot of time doing).
43.
by noddybear 2y ago
It certainly did with me.
44.
by noddybear 2y ago
Checkout the 'ObserveAll' tool in the repo - its deprecated now, but it pipes all the raw entities on the map back to the agent. You could procedurally convert it to unicode format given a pre-defined codebook (which you give to t
45.
by noddybear 2y ago
Someone created this Terraform provider for Factorio a few years ago: https://registry.terraform.io/providers/efokschaner/factorio... I think this could be a good starting point for what you describe! This stuff i
46.
by noddybear 2y ago
I think your intuition is correct about the amount of information that needs to be encoded into an ASCII char. You could potentially use unicode to pack more more into each char, e.g direction, type, status etc. Or make each representation
47.
by noddybear 2y ago
The pipe placement seems to confuse the models especially - as you can't place pipes carrying different fluids next to each other.
48.
by noddybear 2y ago
I experimented a bit using deep search trees to find better Factorio trajectories (MCTS), which worked somewhat well. Unfortunately, it's very computationally expensive, and probably only makes sense in a training context (i.e gatherin
49.
by noddybear 2y ago
Not seen one yet, but the ASCII representation of the game would be ideal for an LLM benchmark.
50.
by noddybear 2y ago
Thank you. Spent too many hours procrastinating by pushing-pixels for those diagrams!
51.
by noddybear 2y ago
One thing we've been talking about is creating tasks that are a bit more 'tower defence', where biters are released every X steps / seconds. The idea would be to test agents in building a military-industrial complex. One
52.
by noddybear 2y ago
We do actually track this. Each action an agent takes consumes 'ticks', which is how long it would take a human player to do it. For example, moving 10 tiles to the right takes something like 300 ticks (5 seconds). However, our re
53.
by noddybear 2y ago
While I agree that busses incentivise over-production, my only argument in their favour stems from SWE design best-practices of maximising cohesion and minimising coupling. The benefits of a bus is that it makes it easier to create factory
54.
by noddybear 2y ago
Agents don't have access to screenshots, as we are purely evaluating text-only models. All reasoning is conducted over object representations of the game (with positions etc). I have anecdotally tried using screenshots to help models d
55.
by noddybear 2y ago
Feel free to create an issue in the repo - am totally happy to help however I am able! I think that the only change you'll have to make is to expose your GoFAI framework in the 'Namespace' object which the agents have access
56.
by noddybear 2y ago
I'm reminded of Hitchhikers Guide to the Galaxy, where the whole planet is one big computer. I would expect that any model directly implemented in Factorio would take up most of the game-world.
57.
by noddybear 2y ago
Not yet, but starting the runs for 3.7 later today! The cost for running all the evals (across all models) was about $10k. Simply giving the agents access to the tool descriptions and API schema is like 20k tokens from the outset. It would
58.
by noddybear 2y ago
I have some pictures of bigger factories - but they tend to be filled with artefacts and general nonsense. I'll dig them out and add them to the appendix. The zig-zag into plastic production was the best 'lab' result, as its
59.
by noddybear 2y ago
FLE agents technically can implement their own Python libraries to leverage GOFAI to do the heavy lifting. None has actually attempted this yet though. It would be interesting to see if this can be achieved just by modifying the manual give
60.
by noddybear 2y ago
I definitely agree that planning is essential to perform well in Factorio - my hope is that we can create agents in FLE that can better front-load the planning part, as well as create utility functions for future use - such that as the agen

More ›