UTF8 / UTF16 / Unicode 3.2 / RFC 3491 - Internationalization of Strings (Framework oversite?)

[Chris Mullins]

I'm implementing RFC 3491 in .NET, and running into a strange issue.

Step 1 of RFC 3491 is performing a set of mappings dicated by tables B.1 and
B.2.

I'm having trouble with the following mappings though, and it seems like a
shortcoming of the .NET framework:

When I see Unicode value 0x10400, I'm supposed to map it to value 0x10428.
This list goes on (the left colulmn is the existing value, the right column
is the replacement value):
(values are in HEX)

10400; 10428; Case map
10401; 10429; Case map
10402; 1042A; Case map
10403; 1042B; Case map
10404; 1042C; Case map
10405; 1042D; Case map
10406; 1042E; Case map
10407; 1042F; Case map
10408; 10430; Case map

(... and on for another few thousand lines...)

I've got the strings loaded into a StringBuilder, and am iterating through
it one character at a time, and comparing the character value to the mapping
values. The problem is that a Character cannot have a value greater than
0xFFFF. Both UTF8 and UTF16 encodings of Unicode 3.2 allow for values
larger than 0xFFFF.

Is there a workaround to this approach that I can use, or do I have to
convert everything to Bytes and do this the hard way?

 

[Jason Smith]

Have you thought about using an array of "long" values? All the string
libraries in .NET assume Unicode, which is 2 bytes per character.

Alternately, you might use a "struct" containing a "long" in place of a
"long." That would just make it easier to group your character conversion
routines.

 

[Chris Mullins]

Unfortunatly, 2 bytes per character - which is what much of the libraries in
..NET assume - is not sufficient. The .NET "char" value, is only good for the
assymetric range -32768 to +65535 (this is sufficient for almost
eveything... except for surrogate pairs). Because everything is based off
"Chars", I can't figure out how to get an arbitrary Unicode Code Point to
properly encode into any of the encodings. The problem is one of Unicode
surrogate pairs, which are supported, but I can't figure out how to properly
encode one...

If only there were a UTF.Encoder method that encoded a true Unicode Code
Point (any value from 0 to 10FFFF), rather than a a char() array. There's
got to be a simple way around this, but it's not evident to me...

I suppose I could manually encode my value into a series of UTF8 bytes, but
that sure seems ugly.

 

[Jon Skeet]

Unfortunatly, 2 bytes per character - which is what much of the libraries in
.NET assume - is not sufficient. The .NET "char" value, is only good for the
assymetric range -32768 to +65535 (this is sufficient for almost
eveything... except for surrogate pairs).

Char is actually 0-65535. The range -32768 to 65535 couldn't be stored
in 16 bits.

Because everything is based off
"Chars", I can't figure out how to get an arbitrary Unicode Code Point to
properly encode into any of the encodings. The problem is one of Unicode
surrogate pairs, which are supported, but I can't figure out how to properly
encode one...

See my recent post - and
http://uk.geocities.com/BabelStone1357/Unicode/surrogates.html
(amongst other pages - a google search for
Unicode "surrogate pairs"
finds a lot of pages.)

 

[Chris Mullins]

See my recent post - and
http://uk.geocities.com/BabelStone1357/Unicode/surrogates.html

I've read and read on surrogate pairs, and I understand what they are at
this point. My problem is how to encode a surrogate pair from an arbitrary
Unicode point. There doesn't seem to be any support in the .NET framework
for doing this.

I suppose I can manually encode the value I'm looking for using UTF8 or
UTF16 encoding, but that seems like the wrong approach.

..NET Encoders have to convert char arrays into a particular byte encoding,
and to turn a byte encoding into a character array. The problem is that I
don't see any mechanism for encoding a value that won't fit in a char array.

How do I, using the .NET Framework, get U-10FF8 into a UTF-8 encoded string?
This is driving me batty.

There is a fantastic mechanism in the framework for iterating over a string
and pull out all the graphemes, but I can't find the encoding side of this
equation....

 

[John Saunders]

I've read and read on surrogate pairs, and I understand what they are at
this point. My problem is how to encode a surrogate pair from an arbitrary
Unicode point. There doesn't seem to be any support in the .NET framework
for doing this.

I suppose I can manually encode the value I'm looking for using UTF8 or
UTF16 encoding, but that seems like the wrong approach.

.NET Encoders have to convert char arrays into a particular byte encoding,
and to turn a byte encoding into a character array. The problem is that I
don't see any mechanism for encoding a value that won't fit in a char array.

How do I, using the .NET Framework, get U-10FF8 into a UTF-8 encoded string?
This is driving me batty.

There is a fantastic mechanism in the framework for iterating over a string
and pull out all the graphemes, but I can't find the encoding side of this
equation....

Since the "char" type is 16 bits, and since strings consist of "char"s, I
don't think you're going to be doing _anything_ with strings and chars and
Unicode code points > 0xffff.

 

[Chris Mullins]

[Unicode Surrogate Pairs]

Since the "char" type is 16 bits, and since strings consist of "char"s, I
don't think you're going to be doing _anything_ with strings and chars and
Unicode code points > 0xffff.

That was originally my though as well, but further reading has proved both
of us wrong.

..NET strings actually have full support for Unicode Surrogate pairs built
into them. I can iterate over the graphemes in the string (rather than all
the characters in the string), with no trouble at all. This functionality is
provided by the StringInfo class (and related family of classes).

It's just the encoding side that I still haven't figured out...

 

[John Saunders]

[Unicode Surrogate Pairs]

Since the "char" type is 16 bits, and since strings consist of "char"s, I
don't think you're going to be doing _anything_ with strings and chars and
Unicode code points > 0xffff.

That was originally my though as well, but further reading has proved both
of us wrong.

.NET strings actually have full support for Unicode Surrogate pairs built
into them. I can iterate over the graphemes in the string (rather than all
the characters in the string), with no trouble at all. This functionality is
provided by the StringInfo class (and related family of classes).

Ok, but it's 16-bit surrogate pairs in the string, not 32-bit characters,
right?

 

[Chris Mullins]

[Unicode Surrogate Pairs]

.NET strings actually have full support for Unicode Surrogate pairs built
into them. I can iterate over the graphemes in the string (rather than all
the characters in the string), with no trouble at all. This

functionality

is
provided by the StringInfo class (and related family of classes).

Ok, but it's 16-bit surrogate pairs in the string, not 32-bit characters,
right?

True.

But I need to figure out how to encode a 32 bit character (0x10FFA) into a
UTF-8 encoded string. This is a legit thing to do, I just don't know how to
do it....

 

[Jon Skeet]

True.

But I need to figure out how to encode a 32 bit character (0x10FFA) into a
UTF-8 encoded string. This is a legit thing to do, I just don't know how to
do it....

The UTF-8 encoding of the 32-bit character should be treated the same
as the UTF-8 encoding of the equivalent surrogate pair. In other words,
you shouldn't need to worry too much.

 

[Peter Huang [MSFT]]

Hi Chris,

I think you may need to do it yourself.
Here is a link, you may have a look.

Regards,
Peter Huang
Microsoft Online Partner Support
Get Secure! www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.

--------------------

From: "Chris Mullins" <[email protected]>
References: <eeBe0u#[email protected]>

<[email protected]>
<[email protected]>
<[email protected]>
<#[email protected]>
<[email protected]>

Subject: Re: UTF8 / UTF16 / Unicode 3.2 / RFC 3491 - Internationalization

of Strings (Framework oversite?)

Date: Sun, 21 Sep 2003 22:16:14 -0700
Lines: 29
X-Priority: 3
X-MSMail-Priority: Normal
X-Newsreader: Microsoft Outlook Express 6.00.2800.1158
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165
Message-ID: <[email protected]>
Newsgroups: microsoft.public.dotnet.framework,microsoft.public.dotnet.general
NNTP-Posting-Host: dcn242-16.dcn.davis.ca.us 168.150.242.16
Path: cpmsftngxa06.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP10.phx.gbl
Xref: cpmsftngxa06.phx.gbl microsoft.public.dotnet.general:109350 microsoft.public.dotnet.framework:54354
X-Tomcat-NG: microsoft.public.dotnet.general

[Unicode Surrogate Pairs]

.NET strings actually have full support for Unicode Surrogate pairs built
into them. I can iterate over the graphemes in the string (rather than all
the characters in the string), with no trouble at all. This

functionality

is
provided by the StringInfo class (and related family of classes).

Ok, but it's 16-bit surrogate pairs in the string, not 32-bit characters,
right?

True.

But I need to figure out how to encode a 32 bit character (0x10FFA) into a
UTF-8 encoded string. This is a legit thing to do, I just don't know how to
do it....

 

[Chris Mullins]

[Encoding a Unicode surrogate pair into a UTF8 string]

Hi Chris,

I think you may need to do it yourself.
Here is a link, you may have a look.

How about that link?

 

[Peter Huang [MSFT]]

Hi Chris,

I am sorry for missing the link.
http://www.cis.ohio-state.edu/cgi-bin/rfc/rfc2279.html
It is about the Encoding from UCS-4 to UTF-8. You may have a check.

Regards,
Peter Huang
Microsoft Online Partner Support
Get Secure! www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.

--------------------

From: "Chris Mullins" <[email protected]>
References: <eeBe0u#[email protected]>

<[email protected]>
<[email protected]>
<[email protected]>
<#[email protected]>
<[email protected]>
<uo#[email protected]>
<#[email protected]>

Subject: Re: UTF8 / UTF16 / Unicode 3.2 / RFC 3491 - Internationalization

of Strings (Framework oversite?)

Date: Mon, 22 Sep 2003 09:59:48 -0700
Lines: 15
X-Priority: 3
X-MSMail-Priority: Normal
X-Newsreader: Microsoft Outlook Express 6.00.2800.1158
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165
Message-ID: <[email protected]>
Newsgroups: microsoft.public.dotnet.general
NNTP-Posting-Host: dcn242-16.dcn.davis.ca.us 168.150.242.16
Path: cpmsftngxa06.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP10.phx.gbl
Xref: cpmsftngxa06.phx.gbl microsoft.public.dotnet.general:109460
X-Tomcat-NG: microsoft.public.dotnet.general


[Encoding a Unicode surrogate pair into a UTF8 string]

Hi Chris,

I think you may need to do it yourself.
Here is a link, you may have a look.

How about that link?