3 ms·

It's a problem in the middle between CV and NLP. It used to be solved with LSTM sequence tagging models, now solved with combinations of CNN and transformer wit

It's a problem in the middle between CV and NLP. It used to be solved with LSTM sequence tagging models, now solved with combinations of CNN and transformer with 2d coordinate embeddings. It's far from being solved, not even receipts work 99% correct, real scores between 89..92%. Manual programming solutions are very bad, they usually do regex matching or template matching. If you want to extract 5-10 fields from a document then you'll have at least one or more errors per page, unfortunately not "automating" too many documents out of 100 yet, with any method.

YeGoblynQueenne 4y ago
To be honest, 89-92% sounds too optimistic for a real-world application. But I was talking about applications of the lego assembly model. My intuition is that lego is a special case: the pieces are made to interlock with each other and there are only a small number of ways in which two pieces can lock together. Compare this with building a prefab house, for example, or a car, where elements must be bound together with nails, mortar, soldering, or glue etc, in essentially unpredictable ways. I say that the approach described is pointless because I can't see any real-world applications of extracting assembly plans from a visualisation of an already-known assembly plan. Even less so when the object assembled is just a toy. And it's been possible to assemble toys using machine vision, given a pre-determined set of instructions, from the 1970's already: (Donald Michie presenting the Edinburgh Research Purpose General Assembly Robot, a.k.a. "Freddy"): https://youtu.be/03p2CADwGF8?t=1668 https://youtu.be/03p2CADwGF8?t=1668 More info of Freddy here https://www.aiai.ed.ac.uk/project/freddy/ https://www.aiai.ed.ac.uk/project/freddy/ Freddy was tediously hand-programmed with a full plan but it used vision and touch to find the pieces of the toy it assembles in the video.

YeGoblynQueenne 4y ago

To be honest, 89-92% sounds too optimistic for a real-world application. But I was talking about applications of the lego assembly model. My intuition is that lego is a special case: the pieces are made to interlock with each other and there are only a small number of ways in which two pieces can lock together. Compare this with building a prefab house, for example, or a car, where elements must be bound together with nails, mortar, soldering, or glue etc, in essentially unpredictable ways. I say that the approach described is pointless because I can't see any real-world applications of extracting assembly plans from a visualisation of an already-known assembly plan. Even less so when the object assembled is just a toy. And it's been possible to assemble toys using machine vision, given a pre-determined set of instructions, from the 1970's already: (Donald Michie presenting the Edinburgh Research Purpose General Assembly Robot, a.k.a. "Freddy"): https://youtu.be/03p2CADwGF8?t=1668 https://youtu.be/03p2CADwGF8?t=1668 More info of Freddy here https://www.aiai.ed.ac.uk/project/freddy/ https://www.aiai.ed.ac.uk/project/freddy/ Freddy was tediously hand-programmed with a full plan but it used vision and touch to find the pieces of the toy it assembles in the video.