5 ms·

Show HN: TutoriaLLM – AI Integrated programming tutorials

128 points by Soumame 2y ago

Hello! I’m So Tokumaru ( https://tokumaru.work https://tokumaru.work ), high school student from Japan working on ways to leverage large language models (LLMs) for K-12 programming education. Recently, I built a project called "TutoriaLLM," a self-hosted application designed to support educators and students through a multi-modal LLM that offers guided tutorials. TutoriaLLM integrates voice (voice-to-voice) and text-based AI interactions into Blockly, a block editor widely known for its use in Scratch. It enables server-side code execution and allows students to test their code directly in Minecraft (Bedrock or Education Edition) without any setup. I’m actively developing this project and aim to bring it to a level ready for use in programming education by next year. I’d love to hear any feedback! Documentation: https://tutoriallm.com https://tutoriallm.com Demo: https://demo.tutoriallm.com https://demo.tutoriallm.com Repository: https://github.com/TutoriaLLM/TutoriaLLM https://github.com/TutoriaLLM/TutoriaLLM

esperent 2y ago
Best of luck! You'll probably get a better response here if the readme also has English. It's typically the first thing other software developers will look at in a project.
- Soumame 2y ago
  Thank you! Now, I have updated readme files with English.
namanyayg 2y ago
It's so cool that you've done so much work in high school already! This project looks great, do you have any active users yet?
- Soumame 2y ago
  As this service is under development, so there are no active users. But user testing has already started in programming schools in Japan with 10~20 kids from primary school. In the future, the aim is to make the software available to programming schools and primary schools worldwide.
zbyforgotp 2y ago
How do you run code in Bedrock? I don’t use computer games at all, but my son plays Minecraft a lot and is into programming. I investigated options for programming in Minecraft, there is Tinker, an app that lets you program in scratch, but it is not very usable, and there are Minecraft servers where you can program in Java, but the learning process is rather steep for that one and also it works only for Java Edition Minecraft, while most of my sons friends use Bedrock. Supposedly there is also Education Edition, but I have no idea how to install it - it seems that it is only available for schools.
- Soumame 2y ago
  When I teach people to learn to code in Minecraft, I basically recommend using Minecraft Education Edition. This combines the Bedrock edition with the original editor called Makecode. They run as two separate apps within the application and are connected via Websockets. Websockets are supported in both the Bedrock and Education Editions, but the editor is only included in the Education.Java Edition does not support Websockets and requires the use of a mod or server. By the way, TutoriaLLM hacks this mechanism of Minecrafft to create code in the browser and run it in Minecraft Bedrock without installation on a small VM thread created on the server. Currently only some of the functionality is available as it is still under development, but in the future it will offer the same functionality as Minecraft Education for free (at least until Microsoft finds it and block them...).
  - Soumame 2y ago
    https://education.minecraft.net/en-us https://education.minecraft.net/en-us This is Minecraft Education. You can try it with purchase an year license.
    - zbyforgotp 2y ago
      I cannot use it - it requires a login from school or other institution.
      - 2y ago
amrrs 2y ago
This is impressive, the fact the dev is high school student is mind blowing! Curious what LLM is used and what kind of role LLM plays?
- amrrs 2y ago
  Looks like it uses OpenAI - https://github.com/TutoriaLLM/TutoriaLLM/blob/bac952b288ededed40fad83483a5953d243cd07c/src/server/session/llm/index.ts https://github.com/TutoriaLLM/TutoriaLLM/blob/bac952b288eded... Would be interesting to see if we can replace it with openai-compatible local llm end point to run this locally
  - Soumame 2y ago
    Correct, I am using OpenAI through the software, includes, gpt-4o, 4o-audio-preview(for multi-modal response, includes audio and text), and whisper(speech to text for generating transcript for user).
bastloing 2y ago
Wow! This is fantastic!
Lerc 2y ago
I tried it out on my phone and was very impressed with how well it worked on mobile. Cramped, but usable. I would love to see simple turtle graphics and canvas output modes. And maybe a retro style serial terminal with glowing text.
- Soumame 2y ago
  Currently, all code is executed server-side (on a Node VM Module created on Worker Threads, using Hono+Websocket to provide the ability to run it in Minecraft), so it is not possible to run Turtle inside the screen. However, if there is a lot of desire for such functionality, it may be possible to use a VM (such as scratch-vm) within the browser to run the code.
Soumame 2y ago
You can join discord community from here: https://discord.gg/zxuREnWVXC https://discord.gg/zxuREnWVXC

128 points by Soumame 2y ago

esperent 2y ago

Best of luck! You'll probably get a better response here if the readme also has English. It's typically the first thing other software developers will look at in a project.

Soumame 2y ago
Thank you! Now, I have updated readme files with English.

namanyayg 2y ago

It's so cool that you've done so much work in high school already! This project looks great, do you have any active users yet?

Soumame 2y ago
As this service is under development, so there are no active users. But user testing has already started in programming schools in Japan with 10~20 kids from primary school. In the future, the aim is to make the software available to programming schools and primary schools worldwide.

zbyforgotp 2y ago

How do you run code in Bedrock? I don’t use computer games at all, but my son plays Minecraft a lot and is into programming. I investigated options for programming in Minecraft, there is Tinker, an app that lets you program in scratch, but it is not very usable, and there are Minecraft servers where you can program in Java, but the learning process is rather steep for that one and also it works only for Java Edition Minecraft, while most of my sons friends use Bedrock. Supposedly there is also Education Edition, but I have no idea how to install it - it seems that it is only available for schools.

Soumame 2y ago
When I teach people to learn to code in Minecraft, I basically recommend using Minecraft Education Edition. This combines the Bedrock edition with the original editor called Makecode. They run as two separate apps within the application and are connected via Websockets. Websockets are supported in both the Bedrock and Education Editions, but the editor is only included in the Education.Java Edition does not support Websockets and requires the use of a mod or server. By the way, TutoriaLLM hacks this mechanism of Minecrafft to create code in the browser and run it in Minecraft Bedrock without installation on a small VM thread created on the server. Currently only some of the functionality is available as it is still under development, but in the future it will offer the same functionality as Minecraft Education for free (at least until Microsoft finds it and block them...).
- Soumame 2y ago
  https://education.minecraft.net/en-us https://education.minecraft.net/en-us This is Minecraft Education. You can try it with purchase an year license.
  - zbyforgotp 2y ago
    I cannot use it - it requires a login from school or other institution.
    - 2y ago

amrrs 2y ago

This is impressive, the fact the dev is high school student is mind blowing! Curious what LLM is used and what kind of role LLM plays?

amrrs 2y ago
Looks like it uses OpenAI - https://github.com/TutoriaLLM/TutoriaLLM/blob/bac952b288ededed40fad83483a5953d243cd07c/src/server/session/llm/index.ts https://github.com/TutoriaLLM/TutoriaLLM/blob/bac952b288eded... Would be interesting to see if we can replace it with openai-compatible local llm end point to run this locally
- Soumame 2y ago
  Correct, I am using OpenAI through the software, includes, gpt-4o, 4o-audio-preview(for multi-modal response, includes audio and text), and whisper(speech to text for generating transcript for user).

bastloing 2y ago

Wow! This is fantastic!

Lerc 2y ago

I tried it out on my phone and was very impressed with how well it worked on mobile. Cramped, but usable. I would love to see simple turtle graphics and canvas output modes. And maybe a retro style serial terminal with glowing text.

Soumame 2y ago
Currently, all code is executed server-side (on a Node VM Module created on Worker Threads, using Hono+Websocket to provide the ability to run it in Minecraft), so it is not possible to run Turtle inside the screen. However, if there is a lot of desire for such functionality, it may be possible to use a VM (such as scratch-vm) within the browser to run the code.

Soumame 2y ago

You can join discord community from here: https://discord.gg/zxuREnWVXC https://discord.gg/zxuREnWVXC