5 ms·

Hi, thanks for submitting. I'm the developer of the app! Happy to answer any questions (I was going to write up a medium post about the technical side of build

Hi, thanks for submitting. I'm the developer of the app! Happy to answer any questions (I was going to write up a medium post about the technical side of building the app but haven't gotten a chance yet)

olympus 9y ago
Hi yeldarb! This is a really neat app. I don't want to diminish your creation, so please don't think I'm trying to put you down with this question: How much of this was just glue-code to integrate a few modules, vs. how much did you have to do from scratch? I'm asking to try to get an idea to see if ARkit is within the reach of an average joe- sudoku solver code could be a copy/paste, digit recognition could be a copy/paste, and ARKit should provide you with some environment/location info.
- yeldarb 9y ago
  It's mostly glue (but some of it is special glue). Oddly enough, digit recognition was one of the harder parts. iOS11's Vision API has character recognition but it only gives you a bounding box, it doesn't do the OCR! So for that I ended up ripping up a whole bunch of Sudoku puzzle books from the second hand book store and scanning them in to create a dataset. Then I trained my own nerual network in Keras and converted it to CoreML. The actual ARKit part was really easy, even for me who has never done iOS programming before.
generic_T 9y ago
Your app is awesome! What type of neural network did you use? Please write a post, I’m teaching myself machine learning and iOS development, and would love to learn from a successful developer!
- yeldarb 9y ago
  Thanks! It's using a convolutional neural network which I minimally adapted from an MNIST tutorial (one of the reasons I picked this project to start with is that there are so many MNIST tutorials for how to do digit recognition out there and it seemed like a good way to dip my toe into ML). Converting it to CoreML after it was trained was really easy. The hardest part was actually creating the dataset and wrestling with pixel buffers to get the image data into a format CoreML liked.
sdrothrock 9y ago
Were there any particular advantages in using a neural network to solve the sudoku as opposed to an existing algorithm?
- yeldarb 9y ago
  Actually the solving of the puzzle itself is just done with a simple recursive algorithm. I trained a neural network to do the computer vision part to interpret the puzzles. And there are a bunch of pre-trained networks built into iOS11's Vision API and ARKit that the app just uses black box.
  - sdrothrock 9y ago
    ahh, sorry for the misunderstanding. Does the network do well with handwriting, e.g. puzzles that have been half-solved?
    - yeldarb 9y ago
      Well I wasn't going to spill the beans just yet but if you promise to keep a secret.. working on that for v2 :) Edit: Although so far I've learned that some people have VERY bad handwriting. I'm having a hard time sanitizing a dataset because I can't even tell what digit they were trying to write
      - ianamartin 9y ago
        I'm that guy. Sorry :( I have the worst handwriting. People are like, "Dude, wtf?" when I try to whiteboard stuff.
      - yeldarb 9y ago
        All good; validates that I'm at least on the right track! Had to pare back my ambitions quite a bit to get _something_ finished before the iOS11 launch. I didn't expect the reception to be so enthusiastic for v1 tbh; but I guess I chose a good place to call it an "MVP"
M4v3R 9y ago
Hey, congrats on shipping! Unfortunately your app doesn't work for me :(. I've tried it on few sudoku designs on my laptop screen (don't have anything on hand right now) and there were two problems - the camera would not focus on the screen which resulted in blurry image, and second - it didn't recognize the puzzle correctly and returned incorrect results. I did put my screen flat on the table to simulate a paper laying on it, because I noticed the app doesn't work when the screen was upright - it rendered the results horizontally anyway.
- yeldarb 9y ago
  Thanks for giving it a shot! Sorry it's not working for you. I'm working on a hack to get non-horizontal puzzles working right now. Unfortunately ARKit can only detect horizontal planes at this point so I'm trying to find a workaround since I now realize most people won't have a paper sudoku laying around to try it out and will logically go to the web to find one. I suspect what's happening with the bad reads is that my neural network for digit classification is trained only on paper puzzles (I chopped up a bunch of books and scanned them to make my dataset). I'll have to get some scans of computer screen puzzles to get better accuracy. Can you let me know what site you tried to use so I can make sure to get that font included in the dataset as well? Hoping to get v1.1 submitted tonight yet that will fix the horizontal plane deficiency. Collecting, labeling, and retraining the network may take a bit longer than that.
krilnon 9y ago
Are you the same yeldarb as the one from kirupaForum?
- yeldarb 9y ago
  I am! Good to see you :) I miss that place; I feel like we grew up there.
  - krilnon 9y ago
    Awesome! It's surreal to see a familiar name making such a neat app. I saw this earlier today and had a small moment of triumph since I'm terrible at solving sudokus myself. A few people visit kF every once and a while; feel free to stop by!
    - kirupac 9y ago
      Small world! Congrats on the app launch :-)
pveierland 9y ago
Have you tried to render individual digits in the existing empty squares instead of rendering a whole new grid? Coupled with a font detection algorithm this could end up looking more magical.
- yeldarb 9y ago
  Originally that was the goal. Unfortunately it wouldn't align exactly perfect (either the size was slightly off due to whether the rectangle detection snagged the inside or outside of the borders or the depth was slightly off so there was slippage when moving around). Didn't quite work right. I do have some things on my to-do list to experiment with though that might help the magic
vegardx 9y ago
Just tried it out, and it looks very good. It was a perfect way to demonstrate ARKit to colleagues. But for some reason it flips the board upside down.
- yeldarb 9y ago
  Sorry about that! That was happening when the phone traveled to the other side of the scene's origin from where it started. The angle calculation got confused. I just got that fixed and submitted to Apple in v1.1 this morning. Should be inbound today or tomorrow as soon as they approve it.

olympus 9y ago

Hi yeldarb! This is a really neat app. I don't want to diminish your creation, so please don't think I'm trying to put you down with this question: How much of this was just glue-code to integrate a few modules, vs. how much did you have to do from scratch? I'm asking to try to get an idea to see if ARkit is within the reach of an average joe- sudoku solver code could be a copy/paste, digit recognition could be a copy/paste, and ARKit should provide you with some environment/location info.

yeldarb 9y ago
It's mostly glue (but some of it is special glue). Oddly enough, digit recognition was one of the harder parts. iOS11's Vision API has character recognition but it only gives you a bounding box, it doesn't do the OCR! So for that I ended up ripping up a whole bunch of Sudoku puzzle books from the second hand book store and scanning them in to create a dataset. Then I trained my own nerual network in Keras and converted it to CoreML. The actual ARKit part was really easy, even for me who has never done iOS programming before.

generic_T 9y ago

Your app is awesome! What type of neural network did you use? Please write a post, I’m teaching myself machine learning and iOS development, and would love to learn from a successful developer!

yeldarb 9y ago
Thanks! It's using a convolutional neural network which I minimally adapted from an MNIST tutorial (one of the reasons I picked this project to start with is that there are so many MNIST tutorials for how to do digit recognition out there and it seemed like a good way to dip my toe into ML). Converting it to CoreML after it was trained was really easy. The hardest part was actually creating the dataset and wrestling with pixel buffers to get the image data into a format CoreML liked.

sdrothrock 9y ago

Were there any particular advantages in using a neural network to solve the sudoku as opposed to an existing algorithm?

yeldarb 9y ago
Actually the solving of the puzzle itself is just done with a simple recursive algorithm. I trained a neural network to do the computer vision part to interpret the puzzles. And there are a bunch of pre-trained networks built into iOS11's Vision API and ARKit that the app just uses black box.
- sdrothrock 9y ago
  ahh, sorry for the misunderstanding. Does the network do well with handwriting, e.g. puzzles that have been half-solved?
  - yeldarb 9y ago
    Well I wasn't going to spill the beans just yet but if you promise to keep a secret.. working on that for v2 :) Edit: Although so far I've learned that some people have VERY bad handwriting. I'm having a hard time sanitizing a dataset because I can't even tell what digit they were trying to write
    - ianamartin 9y ago
      I'm that guy. Sorry :( I have the worst handwriting. People are like, "Dude, wtf?" when I try to whiteboard stuff.
    - yeldarb 9y ago
      All good; validates that I'm at least on the right track! Had to pare back my ambitions quite a bit to get _something_ finished before the iOS11 launch. I didn't expect the reception to be so enthusiastic for v1 tbh; but I guess I chose a good place to call it an "MVP"

M4v3R 9y ago

Hey, congrats on shipping! Unfortunately your app doesn't work for me :(. I've tried it on few sudoku designs on my laptop screen (don't have anything on hand right now) and there were two problems - the camera would not focus on the screen which resulted in blurry image, and second - it didn't recognize the puzzle correctly and returned incorrect results. I did put my screen flat on the table to simulate a paper laying on it, because I noticed the app doesn't work when the screen was upright - it rendered the results horizontally anyway.

yeldarb 9y ago
Thanks for giving it a shot! Sorry it's not working for you. I'm working on a hack to get non-horizontal puzzles working right now. Unfortunately ARKit can only detect horizontal planes at this point so I'm trying to find a workaround since I now realize most people won't have a paper sudoku laying around to try it out and will logically go to the web to find one. I suspect what's happening with the bad reads is that my neural network for digit classification is trained only on paper puzzles (I chopped up a bunch of books and scanned them to make my dataset). I'll have to get some scans of computer screen puzzles to get better accuracy. Can you let me know what site you tried to use so I can make sure to get that font included in the dataset as well? Hoping to get v1.1 submitted tonight yet that will fix the horizontal plane deficiency. Collecting, labeling, and retraining the network may take a bit longer than that.

krilnon 9y ago

Are you the same yeldarb as the one from kirupaForum?

yeldarb 9y ago
I am! Good to see you :) I miss that place; I feel like we grew up there.
- krilnon 9y ago
  Awesome! It's surreal to see a familiar name making such a neat app. I saw this earlier today and had a small moment of triumph since I'm terrible at solving sudokus myself. A few people visit kF every once and a while; feel free to stop by!
  - kirupac 9y ago
    Small world! Congrats on the app launch :-)

pveierland 9y ago

Have you tried to render individual digits in the existing empty squares instead of rendering a whole new grid? Coupled with a font detection algorithm this could end up looking more magical.

yeldarb 9y ago
Originally that was the goal. Unfortunately it wouldn't align exactly perfect (either the size was slightly off due to whether the rectangle detection snagged the inside or outside of the borders or the depth was slightly off so there was slippage when moving around). Didn't quite work right. I do have some things on my to-do list to experiment with though that might help the magic

vegardx 9y ago

Just tried it out, and it looks very good. It was a perfect way to demonstrate ARKit to colleagues. But for some reason it flips the board upside down.

yeldarb 9y ago
Sorry about that! That was happening when the phone traveled to the other side of the scene's origin from where it started. The angle calculation got confused. I just got that fixed and submitted to Apple in v1.1 this morning. Should be inbound today or tomorrow as soon as they approve it.