9 ms·

Building a license plate reader from scratch with deep learning

230 points by ole_gooner 7y ago

ole_gooner 7y ago
Hey, This is an a DIY Article on building a license plate detector using attention OCR. We deep dive in with this article on how you can automate your data entry work with the help of deep learning based OCR. It speaks about attention mechanisms, spatial transformer networks and how they are applied for any text recognition task.
- eggoa 7y ago
  This seems like an item for "Show HN". https://news.ycombinator.com/show https://news.ycombinator.com/show (Not trying to scold or anything, but seems like this would fit well with that convention.)
  - capableweb 7y ago
    I always seen Show HN for interactive things, not articles (which I would say this is), thinking specifically about this part from the Show HN Guidelines: > Show HN is for something you've made that other people can play with. HN users can try it out, give you feedback, and ask questions in the thread.
- waterfirezero 7y ago
  Hi, Could you tell me how to get the charset-labels.txt file? I am running the code
  - anuj123 7y ago
    it's in the attention-ocr fork of the same account. the link is mentioned in the README file (the link to number_plates.py file). go into models/research/attention_ocr/python/datasets/data/number_plates. you'll find the charset-labels.txt file there.
rpmisms 7y ago
Nice primer on how the tech works. The visuals for weighted relationships especially made it click for me.
hackermang 7y ago
Maybe for your next lesson you can teach us how to build a social credit score model? You can train us all to be good obedient citizens.
- criddell 7y ago
  Sharing information on this kind of technology isn't a bad thing, is it?
- sterlind 7y ago
  Hey, training is the machine's job! First, we need our data set. We want: credit scores (possibly the Equifax breach?), a corpus of political views (scrape Twitter), and a way of linking the two (e.g. filter for verified profiles, extract name and look for town references.) Once we have that, identify several sets: the "nobles" are hand-picked by us as shining examples of $COUNTRY values, and assigned value 1, "undesirables" are ranked in steps from [-1,0], and the "serfs" are the ones we're monitoring. Labeling nobles and undesirables the most time-consuming part of this exercise, it's recommended to outsource to some political instructors. Once you have that, mix the sets together and create a deep neural net with a compact feature encoding layer at the top, say 128 neurons, which maps to a scalar value in range [-1,1] to determine social credit. Argmax for a few thousand iters. Once you're done, you'll have a feature encoding network you can deploy on your country's firewall and search engines, while you can hide the credit calculator in a Party-controlled central server. Enjoy! EDIT: D'oh, totally forgot about clique analysis. Honestly that can be handled without ML, you can just assign credit' = min(credit, friend_credit) where friends are one edge away. Run that iteratively with an exponential discount factor in the delta change from credit->credit' and you'll converge on a good answer.
- i_am_nomad 7y ago
  You’re the kind of person who would have opposed license plates themselves when they were introduced.
  - quickthrower2 7y ago
    That's just plain ad-hominem.
dpflan 7y ago
Musings: I've seen more articles on license plate readers (or perhaps been more aware/paying more attention to these). I understand YC has invested in a LPR-tech company Flock Safety. I'm intrigued by the rise in LPR related articles. Such technology has been around for a while, but perhaps properly modularizing it (each camera unit is wireless, etc) and making it SaaS-backed is innovative. Has anyone else noticed this trend in activity for LPR ML tech (care to explain or to discuss the trend)?
- angry_octet 7y ago
  Text OCR is the cannonical neural nets challenge (MNIST) so it seems natural to extend it to be something practical like LPR, without the insane difficulty of other real world shape recognition problems. The easy ubiquity of LPR is essentially the death knell of privacy of movement. We'd have to move to encrypted transponders that only respond to queries with the right codes, but of course the police would still be able to know where you were whenever they liked. Rolling QR code digital number plates would work. But it won't happen.
  - dpflan 7y ago
    Yes, what I found interesting the growth in articles discussing the technology and even open-source projects for LPRs. It's like vehicle facial recognition. Because of this, I've always been intrigued by subversive/defense techniques that fit into the realm of ML - like adversarial patches. Essentially, surveillance in some form is becoming ubiquitous, and private companies especially in the US will probably lead the way. It seems necessary to keep these systems in check at the citizen/individual level because political systems are out-paced by the rate of technological developments. I like the "Rolling QR code digital number plates" concept; are you aware of rolling QR codes used elsewhere?
    - angry_octet 7y ago
      Recognizing individual vehicles is actually really difficult from a distance, because all the features are small -- scratches and dints at the 0.1-50mm scale, mm scale paint irregularities visible in other wavelenghts, misalignment of panels, window stickers etc. So licence plates are a huge reduction in privacy. Other digital transponders, either for tolling or just your devices broadcasting their Wifi/Bluetooth addresses, certainly cause problems but they are within your scope to control. It would be interesting to see if you could design an adversarial LPR jammer, that would not look to a cop like a licence plate. When I wrote 'rolling code' I was being a bit off the cuff. A PKI based system might be something like: {nonce,E(rego_pub,(nonce,car ID, date, hour)),S(car_priv,(nonce,car ID, date, hour))}, where rego_pub is the public key of the local authority, car_priv is the private key of the car, E is encrypt, S is sign. This is already quite complicated, requires central PKI etc. And unfortunately QR codes themselves are not a great fit, because you need to be close to read them, but you could have a variant designed for shorter strings. Another possibility might be to use a (prefix, TOTP code). With a fixed prefix indexed on car model/year (which is already visible from looking at it), the set of all TOTP codes from that fleet at that time could easily be searched by the registration authority to identify which vehicle it was. So, e.g., all blue Toyota Camry 2018s would have prefix 'P9J' and then a 10 minute changing code, like 'Y3KE'. So if a cop needs to look up a car the LPR reads P9JY3KE. The local terminal says it should be a blue Camry, and then sends the string to the validator, which computes the TOTP for all vehicles in the class (like, 2000 in a group) and see if any of them match. If none match, pull them over. There are a few problems with this still, e.g. replay attacks. But possibly solved by using IFF techniques, i.e. the police car can send a signed query forcing the named plate to show more digits of the TOTP code. Criminals just steal the plates off another vehicle (of the same model/color if they are smarter), so maybe they can just steal the whole digital licence plate computer module? Unless the smart licence plate is part of the vehicle security system, on the car bus, it is still going to happen. --- The whole IOV (Internet of Vehicles), V2V (Vehicle to Vehicle) space is going to have to deal with these problems, and there is no shortage of protocols but I don't think it is remotely solved. Lots of companies pushing 5G approaches for this, but I wonder if transponders like for aircraft (ADS-B) or ships (AIS) won't be a simpler (and much cheaper) way, even if they are still using mm wave radios.
colechristensen 7y ago
I cannot think of a use of license-plate reader technology which is not immoral. With the rise of this kind of tech and its prevalence, I think the privacy-focused among us should start pushing to outlaw the use of this tech and eliminate the need for external identification on vehicles.
- capableweb 7y ago
  > I cannot think of a use of license-plate reader technology which is not immoral. The parking garage I use every weekday reads my plate, and opens the gate as I have a subscription. Same thing happens when I want to leave. Is that immoral? It's less about the technology about license-plate reading and more about what people do with the data afterwards. I would restructure the argument as " I cannot think of a usage for storing the data of a license-plate reader for purpose of selling it which is not immoral." or something like that.
  - godelski 7y ago
    > It's less about the technology about license-plate reading and more about what people do with the data afterwards. As someone who cares very much about probably I think this is a point that cannot be stressed enough. It is mostly about storage. Storing things is the major danger. Of course there are use cases too, and I would feel uncomfortable with the police tracking every car on the 5 freeway, but tracking requires storage. Honestly I think your parking garage example is a great use. I can think of others. A notification of your friend or family member pulling into your driveway. Have your house do things like turn on the lights or give you a notification. It could be a good way to do parking meters. I'm sure people could think of more. Technology always has two sides to the coin, it's always a balance of using it only for ethical things. But that requires nuance. Edit: a cool way to use these might be like we use passwords. Your license plate is a password (or let's say username). OCR is used to identify the plate, then it hashes it, checks the hash with that in the database and bingo, door opens. I think that's a level of privacy most of us are comfortable with. There's no retention of the license plate, no good way to identify who's it is, and if the database is hacked the attacker doesn't get your plate.
bredren 7y ago
I work on EasyALPR which I’m putting through Startup School. One thing often missed in license plate reading is that the tech is really well solved now using an open source stack. There is always room for improvement so I welcome new technological approaches like this. However when thinking about license plate data, the real trouble is in what you do with the data, how you handle duplication, and create rules and interface to make the collection useful.
- TuringNYC 7y ago
  I'd love to hear about your stack. I tried this with -- Object Detection for Plate localization; -- OpenCV for de-obliqueing angled images -- Character separation from background; -- OCR around individual characters via Convnet Classifier I tried this with decent results, but I wonder what state of the art open-source is?
  - m463 7y ago
    probably this? https://github.com/openalpr/openalpr https://github.com/openalpr/openalpr
- aidenn0 7y ago
  My uncle is a cop in DC and he says >90% of the hits on plates are for the same number but a different state. Is there any work in identifying the state a license plate is from? Just the style is often enough to narrow it down, despite the fact that most license plate frames partially or totally cover up the state name.
  - bredren 7y ago
    I’m. It aware if efforts to identify same plate, different state. If the data quality is so good you’re getting exact same plate different states, that is impressive. Instead, I’d look at vehicle type identification. There is at least one startup selling an API that determines make / model info that when combined with license plate you would very likely find unique data.
  - 7y ago
jlarocco 7y ago
Great article, misleading title. It's not "from scratch" if it's using OpenCV, TensorFlow, and a web service to do all of the work. Imagine, "Bake Cookies From Scratch," and step one is remove the Pillsbury cookie dough from the freezer...
- phelmig 7y ago
  or you just don’t have to invent the oven, the overall ingredients and how to make a dough, but still need to learn how to get from the basic ideas to something tasty which you want to show to the world...
- zahreeley 7y ago
  le liya author ki. Bindaas
- markdown 7y ago
  "If you wish to make an apple pie from scratch, you must first buy Pillsbury Deep Dish Pie Crust" - Carl Sagan
  - kjeetgill 7y ago
    For those not in the know, the joke here is a play on the real quote from Carl Sagan: If you wish to make an apple pie from scratch, you must first invent the universe.
- wyldfire 7y ago
  I guess it's a bit relative. You're not growing the wheat, growing the cane, milling the flour, etc either.
- kranner 7y ago
  > It's not "from scratch" if it's using OpenCV, TensorFlow, and a web service to do all of the work. To be fair, the web service (for which this article appears to be inbound marketing) is optional and presented as an alternative to building anything at all.

230 points by ole_gooner 7y ago

Hey, This is an a DIY Article on building a license plate detector using attention OCR. We deep dive in with this article on how you can automate your data entry work with the help of deep learning based OCR. It speaks about attention mechanisms, spatial transformer networks and how they are applied for any text recognition task.

eggoa 7y ago
This seems like an item for "Show HN". https://news.ycombinator.com/show https://news.ycombinator.com/show (Not trying to scold or anything, but seems like this would fit well with that convention.)
- capableweb 7y ago
  I always seen Show HN for interactive things, not articles (which I would say this is), thinking specifically about this part from the Show HN Guidelines: > Show HN is for something you've made that other people can play with. HN users can try it out, give you feedback, and ask questions in the thread.
waterfirezero 7y ago
Hi, Could you tell me how to get the charset-labels.txt file? I am running the code
- anuj123 7y ago
  it's in the attention-ocr fork of the same account. the link is mentioned in the README file (the link to number_plates.py file). go into models/research/attention_ocr/python/datasets/data/number_plates. you'll find the charset-labels.txt file there.

rpmisms 7y ago

Nice primer on how the tech works. The visuals for weighted relationships especially made it click for me.

hackermang 7y ago

Maybe for your next lesson you can teach us how to build a social credit score model? You can train us all to be good obedient citizens.

criddell 7y ago
Sharing information on this kind of technology isn't a bad thing, is it?
sterlind 7y ago
Hey, training is the machine's job! First, we need our data set. We want: credit scores (possibly the Equifax breach?), a corpus of political views (scrape Twitter), and a way of linking the two (e.g. filter for verified profiles, extract name and look for town references.) Once we have that, identify several sets: the "nobles" are hand-picked by us as shining examples of $COUNTRY values, and assigned value 1, "undesirables" are ranked in steps from [-1,0], and the "serfs" are the ones we're monitoring. Labeling nobles and undesirables the most time-consuming part of this exercise, it's recommended to outsource to some political instructors. Once you have that, mix the sets together and create a deep neural net with a compact feature encoding layer at the top, say 128 neurons, which maps to a scalar value in range [-1,1] to determine social credit. Argmax for a few thousand iters. Once you're done, you'll have a feature encoding network you can deploy on your country's firewall and search engines, while you can hide the credit calculator in a Party-controlled central server. Enjoy! EDIT: D'oh, totally forgot about clique analysis. Honestly that can be handled without ML, you can just assign credit' = min(credit, friend_credit) where friends are one edge away. Run that iteratively with an exponential discount factor in the delta change from credit->credit' and you'll converge on a good answer.
i_am_nomad 7y ago
You’re the kind of person who would have opposed license plates themselves when they were introduced.
- quickthrower2 7y ago
  That's just plain ad-hominem.

dpflan 7y ago

Musings: I've seen more articles on license plate readers (or perhaps been more aware/paying more attention to these). I understand YC has invested in a LPR-tech company Flock Safety. I'm intrigued by the rise in LPR related articles. Such technology has been around for a while, but perhaps properly modularizing it (each camera unit is wireless, etc) and making it SaaS-backed is innovative. Has anyone else noticed this trend in activity for LPR ML tech (care to explain or to discuss the trend)?

angry_octet 7y ago
Text OCR is the cannonical neural nets challenge (MNIST) so it seems natural to extend it to be something practical like LPR, without the insane difficulty of other real world shape recognition problems. The easy ubiquity of LPR is essentially the death knell of privacy of movement. We'd have to move to encrypted transponders that only respond to queries with the right codes, but of course the police would still be able to know where you were whenever they liked. Rolling QR code digital number plates would work. But it won't happen.
- dpflan 7y ago
  Yes, what I found interesting the growth in articles discussing the technology and even open-source projects for LPRs. It's like vehicle facial recognition. Because of this, I've always been intrigued by subversive/defense techniques that fit into the realm of ML - like adversarial patches. Essentially, surveillance in some form is becoming ubiquitous, and private companies especially in the US will probably lead the way. It seems necessary to keep these systems in check at the citizen/individual level because political systems are out-paced by the rate of technological developments. I like the "Rolling QR code digital number plates" concept; are you aware of rolling QR codes used elsewhere?
  - angry_octet 7y ago
    Recognizing individual vehicles is actually really difficult from a distance, because all the features are small -- scratches and dints at the 0.1-50mm scale, mm scale paint irregularities visible in other wavelenghts, misalignment of panels, window stickers etc. So licence plates are a huge reduction in privacy. Other digital transponders, either for tolling or just your devices broadcasting their Wifi/Bluetooth addresses, certainly cause problems but they are within your scope to control. It would be interesting to see if you could design an adversarial LPR jammer, that would not look to a cop like a licence plate. When I wrote 'rolling code' I was being a bit off the cuff. A PKI based system might be something like: {nonce,E(rego_pub,(nonce,car ID, date, hour)),S(car_priv,(nonce,car ID, date, hour))}, where rego_pub is the public key of the local authority, car_priv is the private key of the car, E is encrypt, S is sign. This is already quite complicated, requires central PKI etc. And unfortunately QR codes themselves are not a great fit, because you need to be close to read them, but you could have a variant designed for shorter strings. Another possibility might be to use a (prefix, TOTP code). With a fixed prefix indexed on car model/year (which is already visible from looking at it), the set of all TOTP codes from that fleet at that time could easily be searched by the registration authority to identify which vehicle it was. So, e.g., all blue Toyota Camry 2018s would have prefix 'P9J' and then a 10 minute changing code, like 'Y3KE'. So if a cop needs to look up a car the LPR reads P9JY3KE. The local terminal says it should be a blue Camry, and then sends the string to the validator, which computes the TOTP for all vehicles in the class (like, 2000 in a group) and see if any of them match. If none match, pull them over. There are a few problems with this still, e.g. replay attacks. But possibly solved by using IFF techniques, i.e. the police car can send a signed query forcing the named plate to show more digits of the TOTP code. Criminals just steal the plates off another vehicle (of the same model/color if they are smarter), so maybe they can just steal the whole digital licence plate computer module? Unless the smart licence plate is part of the vehicle security system, on the car bus, it is still going to happen. --- The whole IOV (Internet of Vehicles), V2V (Vehicle to Vehicle) space is going to have to deal with these problems, and there is no shortage of protocols but I don't think it is remotely solved. Lots of companies pushing 5G approaches for this, but I wonder if transponders like for aircraft (ADS-B) or ships (AIS) won't be a simpler (and much cheaper) way, even if they are still using mm wave radios.

colechristensen 7y ago

I cannot think of a use of license-plate reader technology which is not immoral. With the rise of this kind of tech and its prevalence, I think the privacy-focused among us should start pushing to outlaw the use of this tech and eliminate the need for external identification on vehicles.

capableweb 7y ago
> I cannot think of a use of license-plate reader technology which is not immoral. The parking garage I use every weekday reads my plate, and opens the gate as I have a subscription. Same thing happens when I want to leave. Is that immoral? It's less about the technology about license-plate reading and more about what people do with the data afterwards. I would restructure the argument as " I cannot think of a usage for storing the data of a license-plate reader for purpose of selling it which is not immoral." or something like that.
- godelski 7y ago
  > It's less about the technology about license-plate reading and more about what people do with the data afterwards. As someone who cares very much about probably I think this is a point that cannot be stressed enough. It is mostly about storage. Storing things is the major danger. Of course there are use cases too, and I would feel uncomfortable with the police tracking every car on the 5 freeway, but tracking requires storage. Honestly I think your parking garage example is a great use. I can think of others. A notification of your friend or family member pulling into your driveway. Have your house do things like turn on the lights or give you a notification. It could be a good way to do parking meters. I'm sure people could think of more. Technology always has two sides to the coin, it's always a balance of using it only for ethical things. But that requires nuance. Edit: a cool way to use these might be like we use passwords. Your license plate is a password (or let's say username). OCR is used to identify the plate, then it hashes it, checks the hash with that in the database and bingo, door opens. I think that's a level of privacy most of us are comfortable with. There's no retention of the license plate, no good way to identify who's it is, and if the database is hacked the attacker doesn't get your plate.

bredren 7y ago

I work on EasyALPR which I’m putting through Startup School. One thing often missed in license plate reading is that the tech is really well solved now using an open source stack. There is always room for improvement so I welcome new technological approaches like this. However when thinking about license plate data, the real trouble is in what you do with the data, how you handle duplication, and create rules and interface to make the collection useful.

TuringNYC 7y ago
I'd love to hear about your stack. I tried this with -- Object Detection for Plate localization; -- OpenCV for de-obliqueing angled images -- Character separation from background; -- OCR around individual characters via Convnet Classifier I tried this with decent results, but I wonder what state of the art open-source is?
- m463 7y ago
  probably this? https://github.com/openalpr/openalpr https://github.com/openalpr/openalpr
aidenn0 7y ago
My uncle is a cop in DC and he says >90% of the hits on plates are for the same number but a different state. Is there any work in identifying the state a license plate is from? Just the style is often enough to narrow it down, despite the fact that most license plate frames partially or totally cover up the state name.
- bredren 7y ago
  I’m. It aware if efforts to identify same plate, different state. If the data quality is so good you’re getting exact same plate different states, that is impressive. Instead, I’d look at vehicle type identification. There is at least one startup selling an API that determines make / model info that when combined with license plate you would very likely find unique data.
- 7y ago

jlarocco 7y ago

Great article, misleading title. It's not "from scratch" if it's using OpenCV, TensorFlow, and a web service to do all of the work. Imagine, "Bake Cookies From Scratch," and step one is remove the Pillsbury cookie dough from the freezer...

phelmig 7y ago
or you just don’t have to invent the oven, the overall ingredients and how to make a dough, but still need to learn how to get from the basic ideas to something tasty which you want to show to the world...
zahreeley 7y ago
le liya author ki. Bindaas
markdown 7y ago
"If you wish to make an apple pie from scratch, you must first buy Pillsbury Deep Dish Pie Crust" - Carl Sagan
- kjeetgill 7y ago
  For those not in the know, the joke here is a play on the real quote from Carl Sagan: If you wish to make an apple pie from scratch, you must first invent the universe.
wyldfire 7y ago
I guess it's a bit relative. You're not growing the wheat, growing the cane, milling the flour, etc either.
kranner 7y ago
> It's not "from scratch" if it's using OpenCV, TensorFlow, and a web service to do all of the work. To be fair, the web service (for which this article appears to be inbound marketing) is optional and presented as an alternative to building anything at all.