7 ms·

Show HN: ChessBoss – enhancing physical chessboards with computer vision

83 points by yeldarb 7y ago

We built ChessBoss at the TechCrunch Disrupt hackathon this week. It didn’t make the top 10 but I thought Hacker News might think it’s pretty cool. https://devpost.com/software/chess-boss There are these really cool smart chessboards that can suggest moves and track your games... but they’re $400 and weigh 19 pounds. And of course there are apps that can analyze games but tracking and inputting games by hand is a huge pain. Or fully-digital chess apps... but board games are way more fun in real life! We wondered: “why can’t you just do that in software and bring the best parts of chess apps into the real world?” So we did! A camera passes a feed of the board through our machine learning model which interprets the state of the board and passes it off to Stockfish to display move suggestions in real time. We didn’t quite get to recording the state over time in PGN but we hope to continue this project and add that soon! Would love to know what you think. We’re working on enhancing other board games with computer vision as well; if you want to help us beta test sign up at https://boardboss.com

yeldarb 7y ago
We built ChessBoss at the TechCrunch Disrupt hackathon this week. It didn’t make the top 10 but I thought Hacker News might think it’s pretty cool. There are these really cool smart chessboards that can suggest moves and track your games... but they’re $400 and weigh 19 pounds. And of course there are apps that can analyze games but tracking and inputting games by hand is a huge pain. Or fully-digital chess apps... but board games are way more fun in real life! We wondered: “why can’t you just do that in software and bring the best parts of chess apps into the real world?” So we did! A camera passes a feed of the board through our machine learning model which interprets the state of the board and passes it off to Stockfish to display move suggestions in real time. We didn’t quite get to recording the state over time in PGN but we hope to continue this project and add that soon! Would love to know what you think. We’re working on enhancing other board games with computer vision as well; if you want to help us beta test sign up at https://boardboss.com https://boardboss.com Also I live tweeted about our progress during the hackathon so if you’re interested in how the sausage is made you can check out the blow-by-blow here: https://threadreaderapp.com/thread/1179424684502388736.html https://threadreaderapp.com/thread/1179424684502388736.html
- psuter 7y ago
  Congrats! We tried something similar a few years ago (also at the TC Disrupt Hackathon [1]), but had to take a lot of shortcuts to get to something working. I'm impressed you had the time to train a proper model (we went with old school CV hacks). Looking forward to seeing what BoardBoss could become. These days I've been wanting a CV app to track backgammon games. Those dice can be pretty tiny though :) [1] https://devpost.com/software/chesseye https://devpost.com/software/chesseye
  - yeldarb 7y ago
    Thanks! ML tools have definitely come a long way in the past few years! We used CreateML for our first pass which is great for prototyping; you just give it your training data and hit go. Unfortunately it’s not particularly transparent or tweakable. If it doesn’t do the job well enough you you’re out of luck and have to switch to another tool completely. Edit: cool project!
  - rocauc 7y ago
    Great minds! Love that you deployed to a Pi – I’ve thought about the same to complement or replace smartphones. Can you shed some insight into your ML process? One thing we did to simplify the vision problem is capture images from the same perspective (hence our tripod). We labeled 2894 objects across 292 images. We had 12 objects to detect: each piece for black and white. We struggled with occlusion, especially if a pawn is behind a queen.
    - psuter 7y ago
      I described some of the process in a previous HN thread: https://news.ycombinator.com/item?id=19567549 https://news.ycombinator.com/item?id=19567549 There is also a presentation we had prepared for an informal talk: https://github.com/chesseye/chesseye/blob/master/presentation/slides/slides.pdf https://github.com/chesseye/chesseye/blob/master/presentatio... Hope this helps! I always enjoy talking about these things, so feel free to reach out if you want to discuss it more.
ngcc_hk 7y ago
Sorry no mood but that seems to be a project I would like to do but for WeiQi/Go ...
- yeldarb 7y ago
  Nice! What software features would you like to see added to those games?
  - karmakaze 7y ago
    I also want a Go game recorder. No features just a game recording to have the enjoyment of playing with a real board and still have the game file for later review.
thom 7y ago
Really glad to see a few projects like this recently. For teaching it's _so_ good to be able to quickly help kids analyze a position (even when you have no real right to be teaching people chess) without blundering. I do think the loop needs to be closed with better, more explainable chess engine analyses on the other side though, but this is fantastic for a hackathon project!
- rocauc 7y ago
  So, as we think about making this into a production app, a feature you’d request is the ability for a user to add input as to why a given move is being suggested? At present, we’re simply outputting the recommended move from StockFish, which also does give a “Why.” Perhaps letting users add that commentary in the app sufficiently solves that need.
  - thom 7y ago
    Oh I'm not sure I'm making a practical suggestion here, I think it's a tremendously hard problem to humanize engine outputs (quite often the answer is just "because otherwise in five moves you lose your knight" etc). Some people have attempted this: https://decodechess.com/ https://decodechess.com/
malux85 7y ago
Wow this is cool! Any plans to open source it? This is something I'd love to hack on. If not, anyone else wanna work on this together? email me (in my profile)
- yeldarb 7y ago
  We’re thinking of doing a kickstarter to fund making it into an actual product if there’s enough interest. But I’d love to chat more about what you’d be interested in building! I’ll follow up with you.
  - malux85 7y ago
    Reach out, I’m connected to funding. We may be able to help.
ospohngellert 7y ago
This would be an awesome tool for streaming :) You should also consider posting this on the chess subreddit: http://reddit.com/r/chess http://reddit.com/r/chess
- rocauc 7y ago
  We think streaming and creating game logs will be great features to add. We’ll likely share to the 147k /r/chess when we have an app others can demo. Good call.
Aspos 7y ago
Would suggest using two cameras for stereo photogrammetry. Using mini-projector to highlight clues would also be cool. I see a micro-projector with two cameras just 10 cm apart on a single tripod. With stereo the tripod does not need to be that big for reliable detection.
- TylerE 7y ago
  At that point why bother with a physical board? There are tons of free chess apps that will happily run on a tablet or PC with out all this faffing about.
  - yeldarb 7y ago
    It’s way more fun to play board games in real life than on apps; trying to combine the best of software with the best of physical games.
    - TylerE 7y ago
      Sounds more like the worst of both ... have to set up all those connections, power, still need a PC, but lacking the convenience.
- yeldarb 7y ago
  Ooo cool idea! Have you seen Tilt Five? It’s an AR headset that operates off a similar concept. Instead of trying to figure out the passthrough optics they put projectors in the glasses that reflect back at the viewer. https://www.kickstarter.com/projects/tiltfive/holographic-tabletop-gaming https://www.kickstarter.com/projects/tiltfive/holographic-ta...
  - Aspos 7y ago
    Having to wear bulky glasses would ruin boardgames for me. But I see your point.
levischoen 7y ago
Where’s the code?
- mhuffman 7y ago
  > We plan to launch a Kickstarter to get funding so we can turn this into a real product to enhance Chess for people playing in the real world. I expect it is not available!
_jayhack_ 7y ago
Very cool! I did something similar a school project a while back, shameless promotion here: https://www.youtube.com/watch?v=iZOA1ew-zYc&t=256s https://www.youtube.com/watch?v=iZOA1ew-zYc&t=256s
- rocauc 7y ago
  Nice. We’ll dive into your repo, too. How did you handle a queen occluding a smaller pawn behind it? Simply more training data?
lern_too_spel 7y ago
Similar project from 2015. https://medium.com/@daylenyang/building-chess-id-99afa57326cd https://medium.com/@daylenyang/building-chess-id-99afa57326c...
- rocauc 7y ago
  Nice - I like how your project demonstrates a Bird’s Eye View angle works well. We’ll aim to support the view from a seated player on each side of the game.
billforsternz 7y ago
This is very cool. One very important application would be simply recording competitive games. There are electronic boards that can do this, and they are ubiquitous at top level events. But they are infeasibly expensive. At our chess club one selfless and heroic volunteer inputs all the scoresheets into a PGN every week (shameless plug: he uses my chess GUI, Tarrasch https://triplehappy.com https://triplehappy.com). But it takes something like 5 minutes a game, and gets tedious after a while. I have thought about trying to build a phone app to compete with the electronic boards, but who has the time? You guys do!
- yeldarb 7y ago
  Yes! Absolutely on our road map. We were hoping to get this done at the hackathon actually but it turned out our v1 model wasn’t stable enough to track the game state over time. Hopefully v2 of the model will let us do this. An added bonus over tracking it by hand is that a computer vision powered version would be able to track the time taken for each move as well. If we built this would your club be willing to help us beta test?
  - billforsternz 7y ago
    Yes!
    - yeldarb 7y ago
      Woot! Shoot me an email: brad at roboflow dot ai
dangero 7y ago
Would love to see something like this done in augmented reality so you point it at a chess board and it gives you info about the game in progress.
- yeldarb 7y ago
  Check in later this week :) (Not chess but I think we’re on the same page!)
mustacheemperor 7y ago
This inspires some nifty applications for headset/eyeglass AR. I second the request for source code, it’d be great to play with this on something like the vuzix or HoloLens.
tonyinthehouse 7y ago
This could be very useful for building your own chess bot! I am definitely going try this out when I have some free time.

83 points by yeldarb 7y ago

yeldarb 7y ago

We built ChessBoss at the TechCrunch Disrupt hackathon this week. It didn’t make the top 10 but I thought Hacker News might think it’s pretty cool. There are these really cool smart chessboards that can suggest moves and track your games... but they’re $400 and weigh 19 pounds. And of course there are apps that can analyze games but tracking and inputting games by hand is a huge pain. Or fully-digital chess apps... but board games are way more fun in real life! We wondered: “why can’t you just do that in software and bring the best parts of chess apps into the real world?” So we did! A camera passes a feed of the board through our machine learning model which interprets the state of the board and passes it off to Stockfish to display move suggestions in real time. We didn’t quite get to recording the state over time in PGN but we hope to continue this project and add that soon! Would love to know what you think. We’re working on enhancing other board games with computer vision as well; if you want to help us beta test sign up at https://boardboss.com https://boardboss.com Also I live tweeted about our progress during the hackathon so if you’re interested in how the sausage is made you can check out the blow-by-blow here: https://threadreaderapp.com/thread/1179424684502388736.html https://threadreaderapp.com/thread/1179424684502388736.html

psuter 7y ago
Congrats! We tried something similar a few years ago (also at the TC Disrupt Hackathon [1]), but had to take a lot of shortcuts to get to something working. I'm impressed you had the time to train a proper model (we went with old school CV hacks). Looking forward to seeing what BoardBoss could become. These days I've been wanting a CV app to track backgammon games. Those dice can be pretty tiny though :) [1] https://devpost.com/software/chesseye https://devpost.com/software/chesseye
- yeldarb 7y ago
  Thanks! ML tools have definitely come a long way in the past few years! We used CreateML for our first pass which is great for prototyping; you just give it your training data and hit go. Unfortunately it’s not particularly transparent or tweakable. If it doesn’t do the job well enough you you’re out of luck and have to switch to another tool completely. Edit: cool project!
- rocauc 7y ago
  Great minds! Love that you deployed to a Pi – I’ve thought about the same to complement or replace smartphones. Can you shed some insight into your ML process? One thing we did to simplify the vision problem is capture images from the same perspective (hence our tripod). We labeled 2894 objects across 292 images. We had 12 objects to detect: each piece for black and white. We struggled with occlusion, especially if a pawn is behind a queen.
  - psuter 7y ago
    I described some of the process in a previous HN thread: https://news.ycombinator.com/item?id=19567549 https://news.ycombinator.com/item?id=19567549 There is also a presentation we had prepared for an informal talk: https://github.com/chesseye/chesseye/blob/master/presentation/slides/slides.pdf https://github.com/chesseye/chesseye/blob/master/presentatio... Hope this helps! I always enjoy talking about these things, so feel free to reach out if you want to discuss it more.

ngcc_hk 7y ago

Sorry no mood but that seems to be a project I would like to do but for WeiQi/Go ...

yeldarb 7y ago
Nice! What software features would you like to see added to those games?
- karmakaze 7y ago
  I also want a Go game recorder. No features just a game recording to have the enjoyment of playing with a real board and still have the game file for later review.

thom 7y ago

Really glad to see a few projects like this recently. For teaching it's _so_ good to be able to quickly help kids analyze a position (even when you have no real right to be teaching people chess) without blundering. I do think the loop needs to be closed with better, more explainable chess engine analyses on the other side though, but this is fantastic for a hackathon project!

rocauc 7y ago
So, as we think about making this into a production app, a feature you’d request is the ability for a user to add input as to why a given move is being suggested? At present, we’re simply outputting the recommended move from StockFish, which also does give a “Why.” Perhaps letting users add that commentary in the app sufficiently solves that need.
- thom 7y ago
  Oh I'm not sure I'm making a practical suggestion here, I think it's a tremendously hard problem to humanize engine outputs (quite often the answer is just "because otherwise in five moves you lose your knight" etc). Some people have attempted this: https://decodechess.com/ https://decodechess.com/

malux85 7y ago

Wow this is cool! Any plans to open source it? This is something I'd love to hack on. If not, anyone else wanna work on this together? email me (in my profile)

yeldarb 7y ago
We’re thinking of doing a kickstarter to fund making it into an actual product if there’s enough interest. But I’d love to chat more about what you’d be interested in building! I’ll follow up with you.
- malux85 7y ago
  Reach out, I’m connected to funding. We may be able to help.

ospohngellert 7y ago

This would be an awesome tool for streaming :) You should also consider posting this on the chess subreddit: http://reddit.com/r/chess http://reddit.com/r/chess

rocauc 7y ago
We think streaming and creating game logs will be great features to add. We’ll likely share to the 147k /r/chess when we have an app others can demo. Good call.

Aspos 7y ago

Would suggest using two cameras for stereo photogrammetry. Using mini-projector to highlight clues would also be cool. I see a micro-projector with two cameras just 10 cm apart on a single tripod. With stereo the tripod does not need to be that big for reliable detection.

TylerE 7y ago
At that point why bother with a physical board? There are tons of free chess apps that will happily run on a tablet or PC with out all this faffing about.
- yeldarb 7y ago
  It’s way more fun to play board games in real life than on apps; trying to combine the best of software with the best of physical games.
  - TylerE 7y ago
    Sounds more like the worst of both ... have to set up all those connections, power, still need a PC, but lacking the convenience.
yeldarb 7y ago
Ooo cool idea! Have you seen Tilt Five? It’s an AR headset that operates off a similar concept. Instead of trying to figure out the passthrough optics they put projectors in the glasses that reflect back at the viewer. https://www.kickstarter.com/projects/tiltfive/holographic-tabletop-gaming https://www.kickstarter.com/projects/tiltfive/holographic-ta...
- Aspos 7y ago
  Having to wear bulky glasses would ruin boardgames for me. But I see your point.

levischoen 7y ago

Where’s the code?

mhuffman 7y ago
> We plan to launch a Kickstarter to get funding so we can turn this into a real product to enhance Chess for people playing in the real world. I expect it is not available!

_jayhack_ 7y ago

Very cool! I did something similar a school project a while back, shameless promotion here: https://www.youtube.com/watch?v=iZOA1ew-zYc&t=256s https://www.youtube.com/watch?v=iZOA1ew-zYc&t=256s

rocauc 7y ago
Nice. We’ll dive into your repo, too. How did you handle a queen occluding a smaller pawn behind it? Simply more training data?

lern_too_spel 7y ago

Similar project from 2015. https://medium.com/@daylenyang/building-chess-id-99afa57326cd https://medium.com/@daylenyang/building-chess-id-99afa57326c...

rocauc 7y ago
Nice - I like how your project demonstrates a Bird’s Eye View angle works well. We’ll aim to support the view from a seated player on each side of the game.

billforsternz 7y ago

This is very cool. One very important application would be simply recording competitive games. There are electronic boards that can do this, and they are ubiquitous at top level events. But they are infeasibly expensive. At our chess club one selfless and heroic volunteer inputs all the scoresheets into a PGN every week (shameless plug: he uses my chess GUI, Tarrasch https://triplehappy.com https://triplehappy.com). But it takes something like 5 minutes a game, and gets tedious after a while. I have thought about trying to build a phone app to compete with the electronic boards, but who has the time? You guys do!

yeldarb 7y ago
Yes! Absolutely on our road map. We were hoping to get this done at the hackathon actually but it turned out our v1 model wasn’t stable enough to track the game state over time. Hopefully v2 of the model will let us do this. An added bonus over tracking it by hand is that a computer vision powered version would be able to track the time taken for each move as well. If we built this would your club be willing to help us beta test?
- billforsternz 7y ago
  Yes!
  - yeldarb 7y ago
    Woot! Shoot me an email: brad at roboflow dot ai

dangero 7y ago

Would love to see something like this done in augmented reality so you point it at a chess board and it gives you info about the game in progress.

yeldarb 7y ago
Check in later this week :) (Not chess but I think we’re on the same page!)

mustacheemperor 7y ago

This inspires some nifty applications for headset/eyeglass AR. I second the request for source code, it’d be great to play with this on something like the vuzix or HoloLens.

tonyinthehouse 7y ago

This could be very useful for building your own chess bot! I am definitely going try this out when I have some free time.