Unicode Regular Expressions

I have long been familiar with processing Unicode characters with RegExp (Regular Expressions). I was also aware that RegExp could be used to match Unicode characters based upon their Unicode assigned character properties. I had not yet though coded such property based RegExp. A few days ago I decided to explore this area.

An interesting property, for example, is the script to which a character belongs. e.g. \p{Hangul} will match with any character which belongs to the Hangul script. Hangul is the script used to write Korean.

I started with Perl and here is my simple Perl program:

#!/usr/bin/perl
if("노팅엄"=~/^\p{Hangul}+$/){print "korean\n";}else{print "not korean\n";}

...and this code did not work. I know that 노팅엄 is Korean hangul but my code disagreed. After much searching I discovered I needed to include the statement use utf8 which instructs Perl to use Unicode UTF8 encoding. So my working version of the code is:

#!/usr/bin/perl
use utf8;
if("노팅엄"=~/^\p{Hangul}+$/){print "korean\n";}else{print "not korean\n";}

...and now onto PHP using PCRE Perl Compatible RegExp. My initial RegExp was:

preg_match('/^\p{Hangul}+$/','노팅엄')

...and this did not work! We have already established that 노팅엄 is Korean hangul but my PHP code disagreed with me. After I investigated further I discovered there is a u modifier which directs the code to use Unicode UTF8 encoding. So add the u modifier and we now have a working code!

preg_match('/^\p{Hangul}+$/u','노팅엄')

For several years now, my standard practice is to save text files as Unicode UTF8 encoded files. This includes code files. One still, though, has to repeatedly and explicitly tell systems, programs, functions, utilities, processes to use Unicode. We seem still to be a long way from a total Unicode environment with everything being seamlessly and natively Unicode.

Environments: Perl v5.18.2; PHP v5.5.29