5 ms·
Hey! Robotopia looks awesome, I'm excited to try it out when it launches. How do you convert the LLM output to actions? Is there more broad actions available (i
by Charmunk 9mo ago
Hey! Robotopia looks awesome, I'm excited to try it out when it launches. How do you convert the LLM output to actions? Is there more broad actions available (ie like creating any object, moving anything anywhere) exposed to the LLM or is it more specific tools it can call?
- tom_0 9mo agoThanks :) It may sound insane but we convert actions to Python functions then ask the LLM to write a python script that actually runs in IronPython inside the game. Then we have a visual Behavior Tree system to let our designer define the actions. So yeah, they got a bunch of general actions like walk, talk, follow, interact etc. PS: I think MCP/Tool Calls are a boondoggle and LLMs yearn to just run code. It's crazy how much better this works than JSON schema etc.
- woodrowbarlow 9mo agouhhh... you're running generated code on your customers' PCs? what kind of sandboxing do you have?
- tom_0 9mo agoFair reaction tbh. Right now there's a time watchdog + I'm entirely disabling all I/O and import, But going forward I want to replace it with a proper sandboxing tech... things I looked into are V8 isolates, compilation to WASM, implementing our own gutted python interpreter, spinning up a locked down process, and others. I'm definitely aware of the risk here. The good news is that unless we get pwned, LLMs are very unlikely to write malicious code for the user.
- vannevar 9mo ago>...LLMs are very unlikely to write malicious code for the user. Do you have any idea what the actual probability is? Because if millions of people start using the system, 'very unlikely' can turn into 'virtual certainty' pretty quickly.
- woodrowbarlow 9mo agoyikes