7 ms·

Parsing the Infamous Japanese Postal CSV

168 points by polm23 6y ago

deleted 6y ago
[deleted]
innocenat 6y ago
I remembered a few years I ago I tried to parse this monster of a CSV file to try to extract as much area name as possible. I was trying to create a new Japanese town name list for OpenTTD, since the existing one only have ~600 names. Since I was just scraping for names, I just ignored all the lines that are hard to parse. Kudos to the author for actually parsing it properly.
Cloudef 6y ago
This sounds like a good task for ragel http://www.colm.net/open-source/ragel/ http://www.colm.net/open-source/ragel/
- jacobush 6y ago
  Why?
  - Cloudef 6y ago
    Ragel (or more like finite state machines) excel at parsing streams with all sorts of nuances while still being rigid and bailing on invalid input. Considering the ken_all.csv isn't really standard csv anymore, it's better to write tailored parser for it, than to approach it as a typical csv file.
    - jacobush 6y ago
      I thought you meant using Ragel would be better than https://github.com/polm/posuto https://github.com/polm/posuto what Posuto is currently doing.
      - Cloudef 6y ago
        Nah, didn't imply that. Just that ragel would be suitable to generate the parser for this kind of data.
renewiltord 6y ago
Jesus Christ. What a nightmare! Glad to see you've done all the work. By the way, why did you choose to have invalid identifiers in the project README example and then say the example will work if you rename them? Surely that's better if you just renamed them. It is a bit whimsical, though, so I understand marking Tokyo Tower by its Unicode emoji (what a thing to be in Unicode) for amusement.
- polm23 6y ago
  I originally wrote that bit of code for a code screenshot, and actually thought it would work since I heard Python had added support for Unicode variable names. Shortly afterwards I realized it was invalid, but people seemed to like it so I kept it around. I probably should change it so people have something to copy/paste though...
  - oblvious-earth 6y ago
    Python does support Unicode Variable names but only for characters which represent written languages: https://python-3-for-scientists.readthedocs.io/en/latest/python3_features.html#unicode-variable-names https://python-3-for-scientists.readthedocs.io/en/latest/pyt... Specifics here: https://www.python.org/dev/peps/pep-3131/#specification-of-language-changes https://www.python.org/dev/peps/pep-3131/#specification-of-l... So no emojis alas.
m463 6y ago
This sounds like the flip side of Postel's law. "Be conservative in what you do, be liberal in what you accept from others" https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle And here we have a hero cutting through japanese red tape like Ikiru
nullsense 6y ago
Fun fact: a number of years back I wrote a bash one-liner that attempted to parse this beast. Sadly it wasn't 100% accurate, but it was pretty darn close! This thing truly is a gem.
aikinai 6y ago
My address also always completes to "XXX-borough (except the following buildings)." I hope one day more sites migrate to Posuto on the backend to clean up the mess.
howlgarnish 6y ago
Props to the author for putting in the hard yards here. I used to work with a major geocoding service, and Japanese addresses were a nightmare, both in the original Japanese and when translated to English. In many places in Japan (Kyushu, Okinawa and Hokkaido are particularly bad), the kanji readings of names are completely impenetrable even to native Japanese speakers: https://soranews24.com/2016/12/01/w-t-f-japan-top-5-most-insane-kanji-place-names-in-japan%E3%80%90weird-top-five%E3%80%91/ https://soranews24.com/2016/12/01/w-t-f-japan-top-5-most-ins... And then you get to deal with this kind of extra weirdness: https://en.wikipedia.org/wiki/Japanese_addressing_system#Special_cases https://en.wikipedia.org/wiki/Japanese_addressing_system#Spe...
- jeastwood 6y ago
  Wow this is something else. I thought we had it bad in Scotland, but this sounds worse. In Scotland there are quite a few places where the council use a different tenement flat numbering scheme from the post office. Edinburgh council for example https://www.edinburgh.gov.uk/downloads/file/24357/statutory-addressing-charter#:~:text=Edinburgh%20has%20two%20main%20flat,e.g.%20Flat%201%2C%20Flat%202.&text=For%20the%20traditional%20tenement%20numbering,form%201F1%2C%201F2%2C%20etc https://www.edinburgh.gov.uk/downloads/file/24357/statutory-.... > It is recognised that Edinburgh has a unique character which also translates into the flat numbering systems used. > Edinburgh has two main flat numbering systems in operation; the traditional tenement numbering system e.g. > GF1,1F1 and the modern flat numbering conventions e.g. Flat 1, Flat 2. Where development takes place within > properties with the traditional tenement numbering, this numbering system will be retained. New development will > be allocated the modern flat numbering convention. > Properties in common stairs must be allocated a main street number. Numbers are then allocated internally to each > flat for example, Flat 1, Flat 2. For the traditional tenement numbering system, flats are allocated numbers in the form > 1F1, 1F2, etc. 1F1 should be interpreted as 1st Floor, Flat 1. > The rotation of the internal numbers follows the rotation of the staircase, with the highest number being located at > the door furthest from the last riser on the stair. But the post office (by default) uses a simple numbering scheme, which is the de facto standard even though the "legal" address is the one set by the council. Absolute nightmare.
peterburkimsher 6y ago
>> An example of another comment is 一円. Normally this would mean "one yen", but it also means "the area surrounding", and is a note in the CSV that should be removed from neighborhood names, except for exactly one neighborhood in Shiga where that's actually the name (〒522-0317). Reminds me of the town called Street. https://www.google.co.uk/maps?q=Street,+Somerset https://www.google.co.uk/maps?q=Street,+Somerset There's more great examples of Japanese addressing quirks in Falsehoods Programmers Believe About Addresses. https://www.mjt.me.uk/posts/falsehoods-programmers-believe-about-addresses/ https://www.mjt.me.uk/posts/falsehoods-programmers-believe-a...
- kyuudou 6y ago
  円 is also the kanji for "round" and "circle"
deleted 6y ago
[deleted]
tuukkah 6y ago
Here's the resulting JSON (80 MB!): https://github.com/polm/posuto/blob/master/posuto/postaldata.json https://github.com/polm/posuto/blob/master/posuto/postaldata...
numpad0 6y ago
Also consider: https://info.gbiz.go.jp/tools/imi_tools/ https://info.gbiz.go.jp/tools/imi_tools/
Aeolun 6y ago
I recently bought a house here (in Japan), and our new address is [something] 3-45-6 But apparently, the same is true for all the other people that bought their house on the piece of land where two old houses were torn down to make place for 6 new ones (parcels and houses). I cannot wrap my head around the fact that they have a whole new building and land record, but it doesn’t have a unique address. Uber eats is seriously confused about our address, and is guaranteed to send someone to the other end of the street instead.
WildParser 6y ago
My impression in Japan was that people use phone numbers to identify destinations. Translating phone-numbers into GPS-coordinates seemed to be common.
- wodenokoto 6y ago
  I've never heard of that. Is it a recent thing (past five years) or a really old thing?
  - WildParser 6y ago
    ~2 years ago. Navigation of rental cars worked best with phone numbers. Some cab drivers also preferred phone numbers. Most likely some service to lookup phone-number->GPS became popular - I haven't seen this outside Japan so far.
  - fomine3 6y ago
    It's still common to use phone number to input location for car navigation because its Japanese input system is not good. Phone number is easiest but sometimes points wrong position if you use older maps. We won't use it on smartphone's map app.
- polm23 6y ago
  I had no idea what you were talking about, but looking it up it does look like it's possible to use phone numbers in navigation systems, primarily numbers of businesses. I was able to find several accounts of the feature using the keywords カーナビ電話番号検索 - using just ナビ gets a lot of hits for directories of phone numbers (mainly used for identifying spam callers), and using GPS instead of ナビ gets hits for tracking cell phones. I have never used this and didn't realize it was possible, but I don't drive and rarely take taxis. I can see why this would be popular, especially with tourists - the language isn't a barrier to conveying a number. It also avoids issues with address formatting.
paulintrognon 6y ago
Off topic but I got curious about the Kopyleft mention at the bottom of the article. I was familiar with the notion of Copyleft, but not with the variant with a "K" ; so I googled it and found this article: https://en.wikipedia.org/wiki/Wikipedia:Anti-Wikipedianism https://en.wikipedia.org/wiki/Wikipedia:Anti-Wikipedianism > One of the main sources of Anti-Wikipedianism is the radical far copyLeft (also referred to as Kopyleft, with the Communist K, i.e. Das Kapital). They expect Wikipedia to forbid content from being used for commercial purposes. But on the article, it is written "Do as you like.". So I am a bit confused. Can we really do "as we like", or can we use this content for commercial purpuses..?
- jstarfish 6y ago
  The Kopyleft thing started in the 60s with the Principia Discordia. Same with the "do as you like." It has nothing to do with Wikipedia.
darksaints 6y ago
This has a lot of similarities to the infamous ULS database published by the FCC, though this actually looks a little bit easier to work with. The ULS database doesn't even follow a CSV-like format, because despite all of its variations, one of the two newline forms are ultimately what separates records, and with the ULS, you can't make that assumption (unescaped newline characters are valid values). Nothing less than a customized parser will suffice.
merkul204 6y ago
coming from a non-tech and Japanese background, can someone ELI5?

168 points by polm23 6y ago

deleted 6y ago

[deleted]

I remembered a few years I ago I tried to parse this monster of a CSV file to try to extract as much area name as possible. I was trying to create a new Japanese town name list for OpenTTD, since the existing one only have ~600 names. Since I was just scraping for names, I just ignored all the lines that are hard to parse. Kudos to the author for actually parsing it properly.

Cloudef 6y ago

This sounds like a good task for ragel http://www.colm.net/open-source/ragel/ http://www.colm.net/open-source/ragel/

jacobush 6y ago
Why?
- Cloudef 6y ago
  Ragel (or more like finite state machines) excel at parsing streams with all sorts of nuances while still being rigid and bailing on invalid input. Considering the ken_all.csv isn't really standard csv anymore, it's better to write tailored parser for it, than to approach it as a typical csv file.
  - jacobush 6y ago
    I thought you meant using Ragel would be better than https://github.com/polm/posuto https://github.com/polm/posuto what Posuto is currently doing.
    - Cloudef 6y ago
      Nah, didn't imply that. Just that ragel would be suitable to generate the parser for this kind of data.

renewiltord 6y ago

Jesus Christ. What a nightmare! Glad to see you've done all the work. By the way, why did you choose to have invalid identifiers in the project README example and then say the example will work if you rename them? Surely that's better if you just renamed them. It is a bit whimsical, though, so I understand marking Tokyo Tower by its Unicode emoji (what a thing to be in Unicode) for amusement.

polm23 6y ago
I originally wrote that bit of code for a code screenshot, and actually thought it would work since I heard Python had added support for Unicode variable names. Shortly afterwards I realized it was invalid, but people seemed to like it so I kept it around. I probably should change it so people have something to copy/paste though...
- oblvious-earth 6y ago
  Python does support Unicode Variable names but only for characters which represent written languages: https://python-3-for-scientists.readthedocs.io/en/latest/python3_features.html#unicode-variable-names https://python-3-for-scientists.readthedocs.io/en/latest/pyt... Specifics here: https://www.python.org/dev/peps/pep-3131/#specification-of-language-changes https://www.python.org/dev/peps/pep-3131/#specification-of-l... So no emojis alas.

m463 6y ago

This sounds like the flip side of Postel's law. "Be conservative in what you do, be liberal in what you accept from others" https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle And here we have a hero cutting through japanese red tape like Ikiru

nullsense 6y ago

Fun fact: a number of years back I wrote a bash one-liner that attempted to parse this beast. Sadly it wasn't 100% accurate, but it was pretty darn close! This thing truly is a gem.

aikinai 6y ago

My address also always completes to "XXX-borough (except the following buildings)." I hope one day more sites migrate to Posuto on the backend to clean up the mess.

howlgarnish 6y ago

Props to the author for putting in the hard yards here. I used to work with a major geocoding service, and Japanese addresses were a nightmare, both in the original Japanese and when translated to English. In many places in Japan (Kyushu, Okinawa and Hokkaido are particularly bad), the kanji readings of names are completely impenetrable even to native Japanese speakers: https://soranews24.com/2016/12/01/w-t-f-japan-top-5-most-insane-kanji-place-names-in-japan%E3%80%90weird-top-five%E3%80%91/ https://soranews24.com/2016/12/01/w-t-f-japan-top-5-most-ins... And then you get to deal with this kind of extra weirdness: https://en.wikipedia.org/wiki/Japanese_addressing_system#Special_cases https://en.wikipedia.org/wiki/Japanese_addressing_system#Spe...

jeastwood 6y ago
Wow this is something else. I thought we had it bad in Scotland, but this sounds worse. In Scotland there are quite a few places where the council use a different tenement flat numbering scheme from the post office. Edinburgh council for example https://www.edinburgh.gov.uk/downloads/file/24357/statutory-addressing-charter#:~:text=Edinburgh%20has%20two%20main%20flat,e.g.%20Flat%201%2C%20Flat%202.&text=For%20the%20traditional%20tenement%20numbering,form%201F1%2C%201F2%2C%20etc https://www.edinburgh.gov.uk/downloads/file/24357/statutory-.... > It is recognised that Edinburgh has a unique character which also translates into the flat numbering systems used. > Edinburgh has two main flat numbering systems in operation; the traditional tenement numbering system e.g. > GF1,1F1 and the modern flat numbering conventions e.g. Flat 1, Flat 2. Where development takes place within > properties with the traditional tenement numbering, this numbering system will be retained. New development will > be allocated the modern flat numbering convention. > Properties in common stairs must be allocated a main street number. Numbers are then allocated internally to each > flat for example, Flat 1, Flat 2. For the traditional tenement numbering system, flats are allocated numbers in the form > 1F1, 1F2, etc. 1F1 should be interpreted as 1st Floor, Flat 1. > The rotation of the internal numbers follows the rotation of the staircase, with the highest number being located at > the door furthest from the last riser on the stair. But the post office (by default) uses a simple numbering scheme, which is the de facto standard even though the "legal" address is the one set by the council. Absolute nightmare.

peterburkimsher 6y ago

>> An example of another comment is 一円. Normally this would mean "one yen", but it also means "the area surrounding", and is a note in the CSV that should be removed from neighborhood names, except for exactly one neighborhood in Shiga where that's actually the name (〒522-0317). Reminds me of the town called Street. https://www.google.co.uk/maps?q=Street,+Somerset https://www.google.co.uk/maps?q=Street,+Somerset There's more great examples of Japanese addressing quirks in Falsehoods Programmers Believe About Addresses. https://www.mjt.me.uk/posts/falsehoods-programmers-believe-about-addresses/ https://www.mjt.me.uk/posts/falsehoods-programmers-believe-a...

kyuudou 6y ago
円 is also the kanji for "round" and "circle"

deleted 6y ago

[deleted]

tuukkah 6y ago

Here's the resulting JSON (80 MB!): https://github.com/polm/posuto/blob/master/posuto/postaldata.json https://github.com/polm/posuto/blob/master/posuto/postaldata...

numpad0 6y ago

Also consider: https://info.gbiz.go.jp/tools/imi_tools/ https://info.gbiz.go.jp/tools/imi_tools/

Aeolun 6y ago

I recently bought a house here (in Japan), and our new address is [something] 3-45-6 But apparently, the same is true for all the other people that bought their house on the piece of land where two old houses were torn down to make place for 6 new ones (parcels and houses). I cannot wrap my head around the fact that they have a whole new building and land record, but it doesn’t have a unique address. Uber eats is seriously confused about our address, and is guaranteed to send someone to the other end of the street instead.

WildParser 6y ago

My impression in Japan was that people use phone numbers to identify destinations. Translating phone-numbers into GPS-coordinates seemed to be common.

wodenokoto 6y ago
I've never heard of that. Is it a recent thing (past five years) or a really old thing?
- WildParser 6y ago
  ~2 years ago. Navigation of rental cars worked best with phone numbers. Some cab drivers also preferred phone numbers. Most likely some service to lookup phone-number->GPS became popular - I haven't seen this outside Japan so far.
- fomine3 6y ago
  It's still common to use phone number to input location for car navigation because its Japanese input system is not good. Phone number is easiest but sometimes points wrong position if you use older maps. We won't use it on smartphone's map app.
polm23 6y ago
I had no idea what you were talking about, but looking it up it does look like it's possible to use phone numbers in navigation systems, primarily numbers of businesses. I was able to find several accounts of the feature using the keywords カーナビ電話番号検索 - using just ナビ gets a lot of hits for directories of phone numbers (mainly used for identifying spam callers), and using GPS instead of ナビ gets hits for tracking cell phones. I have never used this and didn't realize it was possible, but I don't drive and rarely take taxis. I can see why this would be popular, especially with tourists - the language isn't a barrier to conveying a number. It also avoids issues with address formatting.

paulintrognon 6y ago

Off topic but I got curious about the Kopyleft mention at the bottom of the article. I was familiar with the notion of Copyleft, but not with the variant with a "K" ; so I googled it and found this article: https://en.wikipedia.org/wiki/Wikipedia:Anti-Wikipedianism https://en.wikipedia.org/wiki/Wikipedia:Anti-Wikipedianism > One of the main sources of Anti-Wikipedianism is the radical far copyLeft (also referred to as Kopyleft, with the Communist K, i.e. Das Kapital). They expect Wikipedia to forbid content from being used for commercial purposes. But on the article, it is written "Do as you like.". So I am a bit confused. Can we really do "as we like", or can we use this content for commercial purpuses..?

jstarfish 6y ago
The Kopyleft thing started in the 60s with the Principia Discordia. Same with the "do as you like." It has nothing to do with Wikipedia.

darksaints 6y ago

This has a lot of similarities to the infamous ULS database published by the FCC, though this actually looks a little bit easier to work with. The ULS database doesn't even follow a CSV-like format, because despite all of its variations, one of the two newline forms are ultimately what separates records, and with the ULS, you can't make that assumption (unescaped newline characters are valid values). Nothing less than a customized parser will suffice.

merkul204 6y ago

coming from a non-tech and Japanese background, can someone ELI5?