nesdev.com
http://forums.nesdev.com/

Can't post with non-BMP characters such as emoji
http://forums.nesdev.com/viewtopic.php?f=13&t=13702
Page 1 of 1

Author:  tepples [ Tue Jan 05, 2016 5:17 pm ]
Post subject:  Can't post with non-BMP characters such as emoji

In this post, I was trying to make a distinction between SNROM with battery and SNROM without battery by including the Unicode character BATTERY (U+1F50B). I can preview the post just fine:
Attachment:
File comment: Preview works fine.
sgemoji.png
sgemoji.png [ 756 Bytes | Viewed 3737 times ]

But when I try to post I get this error:
General Error
SQL ERROR [ mysqli ]

Incorrect string value: '\xF0\x9F\x94\x8B\x0A\x0A...' for column 'post_text' at row 1 [1366]


it appears MySQL by default does not support UTF-8 code unit sequences that correspond to Unicode code points outside the Basic Multilingual Plane (U+0000 through U+FFFD). In GTK+ applications, this character can be typed with Ctrl-Shift-U 1f50b Space. A related question on Stack Overflow is “Incorrect string value” when trying to insert UTF-8 into MySQL via JDBC? which implies that certain settings will need to be flipped from utf8 (BMP-only UTF-8) to utf8mb4 (UTF-8 including extra planes, the NES 2.0 of Unicode), which was introduced in MySQL 5.5.

Author:  Dwedit [ Wed Jan 06, 2016 6:41 pm ]
Post subject:  Re: Can't post with non-BMP characters such as emoji

I would have mistaken the battery graphic for a logic gate or something.

Author:  tepples [ Fri Mar 18, 2016 12:32 pm ]
Post subject:  Re: Can't post with non-BMP characters such as emoji

It's not just the battery. Other characters, such as the emoji that inspired several emblems in the NES homebrew game Concentration Room, can't be posted because the column for the text of a post is set to CHARACTER SET utf8 instead of CHARACTER SET utf8mb4. Part of the reason for this is that forums.nesdev.com runs MySQL 5.1.62, not 5.5.3 or later.

Step 2: Upgrade the MySQL server

Upgrade the MySQL server to v5.5.3+, or ask your server administrator to do it for you.

I hereby request that the server administrator do it for me.

Author:  tepples [ Wed Feb 21, 2018 8:17 am ]
Post subject:  Re: Can't post with non-BMP characters such as emoji

Today I discovered that MySQL had been upgraded to 5.5.53, but the tables still had not been upgraded from utf8 to utf8mb4.

Author:  koitsu [ Wed Feb 21, 2018 2:35 pm ]
Post subject:  Re: Can't post with non-BMP characters such as emoji

tepples, please remember: just because a Unicode glyph exists doesn't mean everyone can view it. Not everyone's devices have every version of Unicode on them. For example, my mobile phone is from 2013 and there are many present-day glyphs that show up as [X]. Battery symbol is Unicode 6.0, so my phone has it, but just because *my* device has it doesn't mean it's wise to use. It also can cause problems when such content is copy-pasted into non-Unicode mediums.

In other words: use of said glyph (vs. an actual word) brings nothing to the table (pun not intended) content-wise. You're just being pedantic / obsessed.

That said: upgrading of the MySQL tables from utf8 to utf8mb4 should probably happen anyways, but you should do some digging to see if there are negative ramifications of that. Examples include: bugs in MySQL client, bugs in MySQL server, performance implications in MySQL server, disk space concerns, incompatibility/breakage with future phpBB upgrades (these upgrades often do ALTER TABLE or other table mangling), and so on. It's always best to follow what the software (phpBB) recommends, even if that means giving up something you personally want. One must be practical.

Page 1 of 1 All times are UTC - 7 hours
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/