The biggest thing that would probably help is not changing the features to SIFT, but to make the recognition part using a classifier rather than the set of rule

The biggest thing that would probably help is not changing the features to SIFT, but to make the recognition part using a classifier rather than the set of rules he's using. Thus, instead of having to join edges into lines, and then applying rules about parallelism, etc., one could take a set of training examples and train a classifier (e.g., support vector machine) on it. But I agree, cool demo.