31 ms·

Has anyone ever successfully converted a large legacy Apache/PHP/MySQL site to UTF-8? If so you deserve an award. utf8mb4 is just one issue and not the worst.

usbqk 5y ago
Sure, you just prepend mb_ to all php string calls >:)
onli 5y ago
https://github.com/s9y/Serendipity/blob/05f58f90d743fe9ade24f3fdfe9a934d0b87c6a1/include/db/mysqli.inc.php#L320 https://github.com/s9y/Serendipity/blob/05f58f90d743fe9ade24... is just the detection function I wrote for a PHP blog engine to detect whether it would even be possible to use utf8mb4 on the system. We completely ran into this issue and didn't know how to handle it for literally years. Lots of blogs to migrate, and who knows whether it worked for all of them...
- mwattsun 5y ago
  The craziest issue I had was I couldn't predict what char encoding the text in my database was in. Most users entered Windows-1252, some text blobs were UTF-16, others were European character sets, and some were UTF-8. Some were Japanese SHIFT_JIS. Don't ask me how any of this happened. I retrospect, I should have dumped all the tables from MySQL and used the excellent PyPy Chardet [1] library to see what I was dealing with, do the conversions and then re-import the data. But then someone could copy UTF-16 from a Windows document and paste it in, so you have to convert going in to the database. You have set Apache to UTF-8, PHP to UTF-8, MySQL to UTF-8, and the MySQL driver you are using to UTF-8. It's not clear how these setting interact. Are there silent conversions happening or do you always have to detect the encoding on data coming from the server? HTML pages have a character encoding specifier, but the BOM at the start of the file takes precedence (I think.) I got it to work by always detecting encoding for any text coming from the database and using iconv, but this turned out to be really slow and unreliable. It was truly the biggest mess by an order of magnitude than any other programming problem I faced in my career. Would not attempt again. [1] https://github.com/chardet/chardet https://github.com/chardet/chardet
sam_lowry_ 5y ago
Yes, but I have chosen utf8_bin as encoding and lived happily with this choice for a dozen years.
ospzfmbbzr 5y ago
Yes. I had the exact same issue with '<' in HTML content stored in the database -- cms pages and also some JS that was stored in the DB and injected into the HEAD of all pages.
charrondev 5y ago
Technical lead for vanillaforums. It’s a huge PHP MySQL app. Probably 10b+ rows in production DBs. We did the the migration in 2017 if I remember correctly.
- mwattsun 5y ago
  Not to take anything away but by legacy I meant late last century sites. VanillaForums was released in July 2006 according to Wikipedia, which makes it Web 2.0. Wikipedia also says "UTF-8 has been the most common encoding for the World Wide Web since 2008" I'm glad you posted though because now I know that sites are ok starting in that time frame, not before and then never touch the old ones again.
ars 5y ago
Been there, done that, with a phpbb installation. I manually converted all the non-UTF data in the database to UTF8, and updated the PHP code and database settings. It worked fine, no major issues, just some work.
fleddr 5y ago
At least once, but it's been a while. From the top of my head: - set encoding at HTML head - also at any HTML form element - Set database collation correctly - I believe it also needs to be done per table - The PHP -> MySQL connection itself may need to have the encoding set - All string manipulation to be done with mb_ - Even ensure your code files themselves are in UTF8

usbqk 5y ago

Sure, you just prepend mb_ to all php string calls >:)

onli 5y ago

https://github.com/s9y/Serendipity/blob/05f58f90d743fe9ade24f3fdfe9a934d0b87c6a1/include/db/mysqli.inc.php#L320 https://github.com/s9y/Serendipity/blob/05f58f90d743fe9ade24... is just the detection function I wrote for a PHP blog engine to detect whether it would even be possible to use utf8mb4 on the system. We completely ran into this issue and didn't know how to handle it for literally years. Lots of blogs to migrate, and who knows whether it worked for all of them...

mwattsun 5y ago
The craziest issue I had was I couldn't predict what char encoding the text in my database was in. Most users entered Windows-1252, some text blobs were UTF-16, others were European character sets, and some were UTF-8. Some were Japanese SHIFT_JIS. Don't ask me how any of this happened. I retrospect, I should have dumped all the tables from MySQL and used the excellent PyPy Chardet [1] library to see what I was dealing with, do the conversions and then re-import the data. But then someone could copy UTF-16 from a Windows document and paste it in, so you have to convert going in to the database. You have set Apache to UTF-8, PHP to UTF-8, MySQL to UTF-8, and the MySQL driver you are using to UTF-8. It's not clear how these setting interact. Are there silent conversions happening or do you always have to detect the encoding on data coming from the server? HTML pages have a character encoding specifier, but the BOM at the start of the file takes precedence (I think.) I got it to work by always detecting encoding for any text coming from the database and using iconv, but this turned out to be really slow and unreliable. It was truly the biggest mess by an order of magnitude than any other programming problem I faced in my career. Would not attempt again. [1] https://github.com/chardet/chardet https://github.com/chardet/chardet

sam_lowry_ 5y ago

Yes, but I have chosen utf8_bin as encoding and lived happily with this choice for a dozen years.

ospzfmbbzr 5y ago

Yes. I had the exact same issue with '<' in HTML content stored in the database -- cms pages and also some JS that was stored in the DB and injected into the HEAD of all pages.

charrondev 5y ago

Technical lead for vanillaforums. It’s a huge PHP MySQL app. Probably 10b+ rows in production DBs. We did the the migration in 2017 if I remember correctly.

mwattsun 5y ago
Not to take anything away but by legacy I meant late last century sites. VanillaForums was released in July 2006 according to Wikipedia, which makes it Web 2.0. Wikipedia also says "UTF-8 has been the most common encoding for the World Wide Web since 2008" I'm glad you posted though because now I know that sites are ok starting in that time frame, not before and then never touch the old ones again.

ars 5y ago

Been there, done that, with a phpbb installation. I manually converted all the non-UTF data in the database to UTF8, and updated the PHP code and database settings. It worked fine, no major issues, just some work.

fleddr 5y ago

At least once, but it's been a while. From the top of my head: - set encoding at HTML head - also at any HTML form element - Set database collation correctly - I believe it also needs to be done per table - The PHP -> MySQL connection itself may need to have the encoding set - All string manipulation to be done with mb_ - Even ensure your code files themselves are in UTF8