GoKawiilLate last year I released posuto, a package presenting Japanese postal code data in an easy-to-use format. It's based on data released by Japan Post, which is infamous for being widely used but hard to parse.
This adorable character by Irasutoya is cute, but the raw postal CSV data is not.
I first became aware of the postal data when I entered my postal code in an online form and it auto-completed my address as "XXX-borough (except the following buildings)". I had no idea what that parenthetical was referring to, so I looked for a common source of postal data, found the CSV, and found the issue. It turns out the CSV file contains parenthetical notes for anyone reading the CSV file and makes reference to the order of the rows.
This causes problems. The data is mainly useful one row at a time, where the parenthetical is meaningless. Since CSV is a field-delimited format, there's also no need for parentheticals - you could just add a note field.
This is only one of many issues with ken_all.csv . You can find people complaining about it regularly on Twitter, and there was even briefly a blog just collecting posts from all over the web about it. A particularly amusing tweet describes people who expect computers to bend to the will of humans being punished in Hell by having to parse ken_all.csv forever.
The README for the file explains that lines with overly long fields will be broken up into multiple lines. Specifically, if the neighborhood name is over 38 characters, or if the half-width katakana (half-width katakana) pronunciation field is over 76 characters, the line will be split into two lines. The overly-long neighborhood field will be continued and all other fields will be duplicated. This is an abbreviated sample of what that looks like:
12345,Tokyo,Minato,This place name is really 12345,Tokyo,Minato,very long it didn't fit in 12345,Tokyo,Minato,a single line so we had to 12345,Tokyo,Minato,split it
The motivation for this is not explained. Maybe there was a fixed-width buffer for storing a line somewhere thirty years ago. I used to process CSV and other files from hundreds of different providers at an old job and I saw many horrors, but I've never seen this particular formatting choice anywhere else. It should also be noted that while the length limits are as stated, the location where line breaks are inserted in long lines appears random, occurring neither at the character limit nor at normal word boundaries.
It's worth noting not all the issues with the CSV are inherently technical; postal codes are always complicated. The postal code with the most rows in the CSV - a stunning 66 - is 〒452-0961, which refers to the Haruhi region of Kiyosu City in Aichi Prefecture. This has that many lines because every neighborhood gets a separate line. (This particular case may be related to Haruhi having been the smallest town by area in Japan from 2006 until 2009, when it was incorporated into Kiyosu City.)
In contrast, the longest continued line, using the line break rules above, is the entry for 〒602-8368 or 〒602-8374, both with eight lines. These are both in one of a few areas in Kyoto that uses a unique, bizarre system of intersection-based addressing. The entry looks a bit like this:
... continue reading