ToUpper()

[Ornette]

Hello,

I'm trying to convert strings to upper without the accents. For the moment,
ToUpper converts é to E with an accent...
I tried to set up english culture (en) but it's the same...

Any ideas ?

Ornette.

 

[Ornette]

Ok, finally I did it like this :

private string ReplaceAccents(string chaine)
{

string strAccents=
"ÀÁÂÃÄÅàáâãäåÒÓÔÕÖØòóôõöøÈÉÊËèéêëÌÍÎÏìíîïÙÚÛÜùúûüÿÑñÇç";
string strNoAccents =
"AAAAAAaaaaaaOOOOOOooooooEEEEeeeeIIIIiiiiUUUUuuuuyNnCc";

char[] tAccent = strAccents.ToCharArray();
char[] tNoAccent = strNoAccents .ToCharArray();

for(int i=0; i<strAccents.Length; i++)
{
chaine = chaine.Replace(tAccent .ToString(), tNoAccent
.ToString());
}
return chaine;
}

J'ai pas trouvé mieux, même si ça boucle un peu pour rien...

Ornette.

 

[Ornette]

Hello,

This is better :

byte[] bString =
System.Text.Encoding.GetEncoding(1251).GetBytes(StringAvecAccents);
string stringSansAccent = System.Text.Encoding.ASCII.GetString(bString );

Reference CodePage :
http://www.microsoft.com/globaldev/reference/sbcs/1251.mspx

Ornette.

Ok, finally I did it like this :

private string ReplaceAccents(string chaine)
{

string strAccents=
"ÀÁÂÃÄÅàáâãäåÒÓÔÕÖØòóôõöøÈÉÊËèéêëÌÍÎÏìíîïÙÚÛÜùúûüÿÑñÇç";
string strNoAccents =
"AAAAAAaaaaaaOOOOOOooooooEEEEeeeeIIIIiiiiUUUUuuuuyNnCc";

char[] tAccent = strAccents.ToCharArray();
char[] tNoAccent = strNoAccents .ToCharArray();

for(int i=0; i<strAccents.Length; i++)
{
chaine = chaine.Replace(tAccent .ToString(), tNoAccent
.ToString());
}
return chaine;
}

J'ai pas trouvé mieux, même si ça boucle un peu pour rien...

Ornette.

Hello,

I'm trying to convert strings to upper without the accents. For the
moment, ToUpper converts é to E with an accent...
I tried to set up english culture (en) but it's the same...

Any ideas ?

Ornette.

 

[Jon Skeet [C# MVP]]

This is better :

byte[] bString =
System.Text.Encoding.GetEncoding(1251).GetBytes(StringAvecAccents);
string stringSansAccent = System.Text.Encoding.ASCII.GetString(bString );

Well, that's assuming that the encoding will find the closest match
letter. It may work now, but there's no guarantee that it will in the
future.

 

[Ornette]

Hello,

So how would you do ?

Ornette.

This is better :

byte[] bString =
System.Text.Encoding.GetEncoding(1251).GetBytes(StringAvecAccents);
string stringSansAccent = System.Text.Encoding.ASCII.GetString(bString );

Well, that's assuming that the encoding will find the closest match
letter. It may work now, but there's no guarantee that it will in the
future.

 

[Jon Skeet [C# MVP]]

So how would you do ?

The mapping table idea you had before looked best to me, although I
wouldn't quite implement it the same way. I'd have a look up table for
every possible character, where it defaults to the Unicode character,
but for all the accented characters you care about, you specify the
non-accented version.

You'd then call ToCharArray() on the string in question, go through
each character replacing the original with the mapped character, and
then create a new string with the char array.

It does require you to manually map all the accented characters you
care about though.

My guess is that there are libraries around to do this somewhere, but I
don't know of any myself.

 

[Ornette]

Ok, thank you for your point of view.
I really agree.

For the the librairies, I also didn't find any one.

Have a nice day and thanks again.

Ornette.

 

[Chris Mullins [MVP]]

The closest thing that comes to mind is an RFC called stringprep. There are
a wide variety of stringprep profiles, and while they don't quite do what
you're looking for, they're close. Included in stringprep is a set of
mapping tables for Uppder->Lower case conversions. These are (in that
context) called case-foldings, are are found in table B.2. Unfortunatly,
they're Upper->Lower, not the other way around.

Stringprep:
http://www.faqs.org/rfcs/rfc3454.html

There are a number of profiles:
[Profile for Internaional Domain Names]
http://www.rfc-editor.org/rfc/rfc3491.txt

[Profile for iSCSI names]
http://tools.ietf.org/html/draft-ietf-ips-iscsi-string-prep-01

[Profile for SASL UserNames & Passwords]
http://www.ietf.org/rfc/rfc4013.txt

[Profile for XMPP Resources]
http://www.xmpp.org/internet-drafts/attic/draft-ietf-xmpp-resourceprep-02.html

There's a C# implementation of this RFC that's part of the libidn library.
There's also a C++ & Java version.
http://www.gnu.org/software/libidn/

We've actually got a full implemention of stringprep as well - it's much
more .Net 2.0 ish than the libidn one, which is just a native C++ app that
was then ported to Java & .Net. It's found in our open-source SoapBox
Framework.

 

[JR]

In a more general way:

There is a Unicode database at

http://www.unicode.org/Public/UNIDATA/

You could do what you want in two steps: decompose the string to base
characters followed by accent (NFKD normalization), then remove the accents.

JR

The closest thing that comes to mind is an RFC called stringprep. There
are a wide variety of stringprep profiles, and while they don't quite do
what you're looking for, they're close. Included in stringprep is a set of
mapping tables for Uppder->Lower case conversions. These are (in that
context) called case-foldings, are are found in table B.2. Unfortunatly,
they're Upper->Lower, not the other way around.

Stringprep:
http://www.faqs.org/rfcs/rfc3454.html

There are a number of profiles:
[Profile for Internaional Domain Names]
http://www.rfc-editor.org/rfc/rfc3491.txt

[Profile for iSCSI names]
http://tools.ietf.org/html/draft-ietf-ips-iscsi-string-prep-01

[Profile for SASL UserNames & Passwords]
http://www.ietf.org/rfc/rfc4013.txt

[Profile for XMPP Resources]
http://www.xmpp.org/internet-drafts/attic/draft-ietf-xmpp-resourceprep-02.html

There's a C# implementation of this RFC that's part of the libidn library.
There's also a C++ & Java version.
http://www.gnu.org/software/libidn/

We've actually got a full implemention of stringprep as well - it's much
more .Net 2.0 ish than the libidn one, which is just a native C++ app that
was then ported to Java & .Net. It's found in our open-source SoapBox
Framework.

 

[Chris Mullins]

I hadn't thought of that, but it's certainly an option.

Doing the normalization in .Net 2.0 is easy enough:

string s = "test";
string normalized = s.Normalize(NormalizationForm.FormKC);

Then you can iterate over the normalized string looking for (and
removing) the accents.

--
Chris Mullins

In a more general way:

There is a Unicode database at

http://www.unicode.org/Public/UNIDATA/

You could do what you want in two steps: decompose the string to base
characters followed by accent (NFKD normalization), then remove the accents.

JR

The closest thing that comes to mind is an RFC called stringprep. There
are a wide variety of stringprep profiles, and while they don't quite do
what you're looking for, they're close. Included in stringprep is a setof
mapping tables for Uppder->Lower case conversions. These are (in that
context) called case-foldings, are are found in table B.2. Unfortunatly,
they're Upper->Lower, not the other way around.

Stringprep:
http://www.faqs.org/rfcs/rfc3454.html

There are a number of profiles:
[Profile for Internaional Domain Names]
http://www.rfc-editor.org/rfc/rfc3491.txt

[Profile for iSCSI names]
http://tools.ietf.org/html/draft-ietf-ips-iscsi-string-prep-01

[Profile for SASL UserNames & Passwords]
http://www.ietf.org/rfc/rfc4013.txt

[Profile for XMPP Resources]
http://www.xmpp.org/internet-drafts/attic/draft-ietf-xmpp-resourcepre...

There's a C# implementation of this RFC that's part of the libidn library.
There's also a C++ & Java version.
http://www.gnu.org/software/libidn/

We've actually got a full implemention of stringprep as well - it's much
more .Net 2.0 ish than the libidn one, which is just a native C++ app that
was then ported to Java & .Net. It's found in our open-source SoapBox
Framework.

 

[Jon Skeet [C# MVP]]

I hadn't thought of that, but it's certainly an option.

Doing the normalization in .Net 2.0 is easy enough:

string s = "test";
string normalized = s.Normalize(NormalizationForm.FormKC);

Then you can iterate over the normalized string looking for (and
removing) the accents.

Cool - I hadn't seen that 2.0 had normalization stuff. Fantastic!

 

[Chris Mullins [MVP]]

Yea, .Net 2.0 also added the IDN stuff (which includes the PunyCode
algorithm), which came as a complete surprise to me:

System.Globalization.IdnMapping mapping = new
System.Globalization.IdnMapping();
normalized = mapping.GetAscii(normalized);

(just don't run an empty string through the IdnMapping class, or you'll get
an exception)

Now, if only they exposed the BiDirectional stuff...

 

[JR]

You meant NormalizationForm.FormKD.

Looking into it, I see a simpler method: After normalization, use
ASCIIEncoding with DecoderReplacementFallback replacing invalid ASCII
characters (which will be the accents) with the empty string.

JR

"Chris Mullins" <[email protected]> ???
??????:[email protected]...
I hadn't thought of that, but it's certainly an option.

Doing the normalization in .Net 2.0 is easy enough:

string s = "test";
string normalized = s.Normalize(NormalizationForm.FormKC);

Then you can iterate over the normalized string looking for (and
removing) the accents.

--
Chris Mullins

In a more general way:

There is a Unicode database at

http://www.unicode.org/Public/UNIDATA/

You could do what you want in two steps: decompose the string to base
characters followed by accent (NFKD normalization), then remove the
accents.

JR

The closest thing that comes to mind is an RFC called stringprep. There
are a wide variety of stringprep profiles, and while they don't quite do
what you're looking for, they're close. Included in stringprep is a set
of
mapping tables for Uppder->Lower case conversions. These are (in that
context) called case-foldings, are are found in table B.2. Unfortunatly,
they're Upper->Lower, not the other way around.

Stringprep:
http://www.faqs.org/rfcs/rfc3454.html

There are a number of profiles:
[Profile for Internaional Domain Names]
http://www.rfc-editor.org/rfc/rfc3491.txt

[Profile for iSCSI names]
http://tools.ietf.org/html/draft-ietf-ips-iscsi-string-prep-01

[Profile for SASL UserNames & Passwords]
http://www.ietf.org/rfc/rfc4013.txt

[Profile for XMPP Resources]
http://www.xmpp.org/internet-drafts/attic/draft-ietf-xmpp-resourcepre...

There's a C# implementation of this RFC that's part of the libidn
library.
There's also a C++ & Java version.
http://www.gnu.org/software/libidn/

We've actually got a full implemention of stringprep as well - it's much
more .Net 2.0 ish than the libidn one, which is just a native C++ app
that
was then ported to Java & .Net. It's found in our open-source SoapBox
Framework.

 

[Chris Mullins [MVP]]

Opps. Definatly KD!

We want to do the decomposition & make our changes. For KC would decompose &
then perform a canonical recompose - which would defeat the purpose!

I've never used (or even seen) the DecoderReplacementFallback - that's
another good idea. By now the original poster has probably given up and will
never try any of these solutions, but I think they would very cleanly do the
trick.

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

You meant NormalizationForm.FormKD.

Looking into it, I see a simpler method: After normalization, use
ASCIIEncoding with DecoderReplacementFallback replacing invalid ASCII
characters (which will be the accents) with the empty string.

JR

"Chris Mullins" <[email protected]> ???
??????:[email protected]...
I hadn't thought of that, but it's certainly an option.

Doing the normalization in .Net 2.0 is easy enough:

string s = "test";
string normalized = s.Normalize(NormalizationForm.FormKC);

Then you can iterate over the normalized string looking for (and
removing) the accents.

--
Chris Mullins

In a more general way:

There is a Unicode database at

http://www.unicode.org/Public/UNIDATA/

You could do what you want in two steps: decompose the string to base
characters followed by accent (NFKD normalization), then remove the
accents.

JR

The closest thing that comes to mind is an RFC called stringprep. There
are a wide variety of stringprep profiles, and while they don't quite
do
what you're looking for, they're close. Included in stringprep is a set
of
mapping tables for Uppder->Lower case conversions. These are (in that
context) called case-foldings, are are found in table B.2.
Unfortunatly,
they're Upper->Lower, not the other way around.

Stringprep:
http://www.faqs.org/rfcs/rfc3454.html

There are a number of profiles:
[Profile for Internaional Domain Names]
http://www.rfc-editor.org/rfc/rfc3491.txt

[Profile for iSCSI names]
http://tools.ietf.org/html/draft-ietf-ips-iscsi-string-prep-01

[Profile for SASL UserNames & Passwords]
http://www.ietf.org/rfc/rfc4013.txt

[Profile for XMPP Resources]
http://www.xmpp.org/internet-drafts/attic/draft-ietf-xmpp-resourcepre...

There's a C# implementation of this RFC that's part of the libidn
library.
There's also a C++ & Java version.
http://www.gnu.org/software/libidn/

We've actually got a full implemention of stringprep as well - it's
much
more .Net 2.0 ish than the libidn one, which is just a native C++ app
that
was then ported to Java & .Net. It's found in our open-source SoapBox
Framework.

So how would you do ?

The mapping table idea you had before looked best to me, although I
wouldn't quite implement it the same way. I'd have a look up table for
every possible character, where it defaults to the Unicode character,
but for all the accented characters you care about, you specify the
non-accented version.

You'd then call ToCharArray() on the string in question, go through
each character replacing the original with the mapped character, and
then create a new string with the char array.

It does require you to manually map all the accented characters you
care about though.

My guess is that there are libraries around to do this somewhere, but
I
don't know of any myself.

 

[Chris Mullins [MVP]]

For anyone still paying attention, the complete (and working + tested) code
is:

string s = "áäåãòä:usdBDlGXHHA";
string normalized = s.Normalize(NormalizationForm.FormKD);

Encoding ascii = Encoding.GetEncoding(
"us-ascii",
new EncoderReplacementFallback(string.Empty),
new DecoderReplacementFallback(string.Empty));

byte[] encodedBytes = new byte[ascii.GetByteCount(normalized)];
int numberOfEncodedBytes = ascii.GetBytes(normalized, 0, normalized.Length,
encodedBytes, 0);

string newString = ascii.GetString(encodedBytes).ToUpper();
MessageBox.Show(newString);

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

You meant NormalizationForm.FormKD.

Looking into it, I see a simpler method: After normalization, use
ASCIIEncoding with DecoderReplacementFallback replacing invalid ASCII
characters (which will be the accents) with the empty string.

JR

"Chris Mullins" <[email protected]> ???
??????:[email protected]...
I hadn't thought of that, but it's certainly an option.

Doing the normalization in .Net 2.0 is easy enough:

string s = "test";
string normalized = s.Normalize(NormalizationForm.FormKC);

Then you can iterate over the normalized string looking for (and
removing) the accents.

--
Chris Mullins

In a more general way:

There is a Unicode database at

http://www.unicode.org/Public/UNIDATA/

You could do what you want in two steps: decompose the string to base
characters followed by accent (NFKD normalization), then remove the
accents.

JR

The closest thing that comes to mind is an RFC called stringprep. There
are a wide variety of stringprep profiles, and while they don't quite
do
what you're looking for, they're close. Included in stringprep is a set
of
mapping tables for Uppder->Lower case conversions. These are (in that
context) called case-foldings, are are found in table B.2.
Unfortunatly,
they're Upper->Lower, not the other way around.

Stringprep:
http://www.faqs.org/rfcs/rfc3454.html

There are a number of profiles:
[Profile for Internaional Domain Names]
http://www.rfc-editor.org/rfc/rfc3491.txt

[Profile for iSCSI names]
http://tools.ietf.org/html/draft-ietf-ips-iscsi-string-prep-01

[Profile for SASL UserNames & Passwords]
http://www.ietf.org/rfc/rfc4013.txt

[Profile for XMPP Resources]
http://www.xmpp.org/internet-drafts/attic/draft-ietf-xmpp-resourcepre...

There's a C# implementation of this RFC that's part of the libidn
library.
There's also a C++ & Java version.
http://www.gnu.org/software/libidn/

We've actually got a full implemention of stringprep as well - it's
much
more .Net 2.0 ish than the libidn one, which is just a native C++ app
that
was then ported to Java & .Net. It's found in our open-source SoapBox
Framework.

So how would you do ?

The mapping table idea you had before looked best to me, although I
wouldn't quite implement it the same way. I'd have a look up table for
every possible character, where it defaults to the Unicode character,
but for all the accented characters you care about, you specify the
non-accented version.

You'd then call ToCharArray() on the string in question, go through
each character replacing the original with the mapped character, and
then create a new string with the char array.

It does require you to manually map all the accented characters you
care about though.

My guess is that there are libraries around to do this somewhere, but
I
don't know of any myself.

 

[Ornette]

Hello,

I following the thread which is very interesting.
I's thinking about wrinting a "StringHelper" class to do the jos and re-use


Thanks a lot for your participation to this subject !!!

Ornette.

Opps. Definatly KD!

We want to do the decomposition & make our changes. For KC would decompose
& then perform a canonical recompose - which would defeat the purpose!

I've never used (or even seen) the DecoderReplacementFallback - that's
another good idea. By now the original poster has probably given up and
will never try any of these solutions, but I think they would very cleanly
do the trick.

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

You meant NormalizationForm.FormKD.

Looking into it, I see a simpler method: After normalization, use
ASCIIEncoding with DecoderReplacementFallback replacing invalid ASCII
characters (which will be the accents) with the empty string.

JR

"Chris Mullins" <[email protected]> ???
??????:[email protected]...
I hadn't thought of that, but it's certainly an option.

Doing the normalization in .Net 2.0 is easy enough:

string s = "test";
string normalized = s.Normalize(NormalizationForm.FormKC);

Then you can iterate over the normalized string looking for (and
removing) the accents.

--
Chris Mullins

In a more general way:

There is a Unicode database at

http://www.unicode.org/Public/UNIDATA/

You could do what you want in two steps: decompose the string to base
characters followed by accent (NFKD normalization), then remove the
accents.

JR

"Chris Mullins [MVP]" <[email protected]> ëúá
áäåãòä:[email protected]...

The closest thing that comes to mind is an RFC called stringprep.
There
are a wide variety of stringprep profiles, and while they don't quite
do
what you're looking for, they're close. Included in stringprep is a
set of
mapping tables for Uppder->Lower case conversions. These are (in that
context) called case-foldings, are are found in table B.2.
Unfortunatly,
they're Upper->Lower, not the other way around.

Stringprep:
http://www.faqs.org/rfcs/rfc3454.html

There are a number of profiles:
[Profile for Internaional Domain Names]
http://www.rfc-editor.org/rfc/rfc3491.txt

[Profile for iSCSI names]
http://tools.ietf.org/html/draft-ietf-ips-iscsi-string-prep-01

[Profile for SASL UserNames & Passwords]
http://www.ietf.org/rfc/rfc4013.txt

[Profile for XMPP Resources]
http://www.xmpp.org/internet-drafts/attic/draft-ietf-xmpp-resourcepre...

There's a C# implementation of this RFC that's part of the libidn
library.
There's also a C++ & Java version.
http://www.gnu.org/software/libidn/

We've actually got a full implemention of stringprep as well - it's
much
more .Net 2.0 ish than the libidn one, which is just a native C++ app
that
was then ported to Java & .Net. It's found in our open-source SoapBox
Framework.

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

So how would you do ?

The mapping table idea you had before looked best to me, although I
wouldn't quite implement it the same way. I'd have a look up table
for
every possible character, where it defaults to the Unicode character,
but for all the accented characters you care about, you specify the
non-accented version.

You'd then call ToCharArray() on the string in question, go through
each character replacing the original with the mapped character, and
then create a new string with the char array.

It does require you to manually map all the accented characters you
care about though.

My guess is that there are libraries around to do this somewhere, but
I
don't know of any myself.

 

[Chris Mullins [MVP]]

I'm sitting in front of my computer, and not feeling much like working (too
busy sneezing and being sick!), so I broke down and turned the solution to
this problem into a little blog entry.

http://www.coversant.com/Default.aspx?tabid=88&EntryID=30

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

Hello,

I following the thread which is very interesting.
I's thinking about wrinting a "StringHelper" class to do the jos and
re-use

Thanks a lot for your participation to this subject !!!

Ornette.

Opps. Definatly KD!

We want to do the decomposition & make our changes. For KC would
decompose & then perform a canonical recompose - which would defeat the
purpose!

I've never used (or even seen) the DecoderReplacementFallback - that's
another good idea. By now the original poster has probably given up and
will never try any of these solutions, but I think they would very
cleanly do the trick.

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

You meant NormalizationForm.FormKD.

Looking into it, I see a simpler method: After normalization, use
ASCIIEncoding with DecoderReplacementFallback replacing invalid ASCII
characters (which will be the accents) with the empty string.

JR

"Chris Mullins" <[email protected]> ???
??????:[email protected]...
I hadn't thought of that, but it's certainly an option.

Doing the normalization in .Net 2.0 is easy enough:

string s = "test";
string normalized = s.Normalize(NormalizationForm.FormKC);

Then you can iterate over the normalized string looking for (and
removing) the accents.

--
Chris Mullins


In a more general way:

There is a Unicode database at

http://www.unicode.org/Public/UNIDATA/

You could do what you want in two steps: decompose the string to base
characters followed by accent (NFKD normalization), then remove the
accents.

JR

"Chris Mullins [MVP]" <[email protected]> ëúá
áäåãòä:[email protected]...

The closest thing that comes to mind is an RFC called stringprep.
There
are a wide variety of stringprep profiles, and while they don't quite
do
what you're looking for, they're close. Included in stringprep is a
set of
mapping tables for Uppder->Lower case conversions. These are (in that
context) called case-foldings, are are found in table B.2.
Unfortunatly,
they're Upper->Lower, not the other way around.

Stringprep:
http://www.faqs.org/rfcs/rfc3454.html

There are a number of profiles:
[Profile for Internaional Domain Names]
http://www.rfc-editor.org/rfc/rfc3491.txt

[Profile for iSCSI names]
http://tools.ietf.org/html/draft-ietf-ips-iscsi-string-prep-01

[Profile for SASL UserNames & Passwords]
http://www.ietf.org/rfc/rfc4013.txt

[Profile for XMPP Resources]
http://www.xmpp.org/internet-drafts/attic/draft-ietf-xmpp-resourcepre...

There's a C# implementation of this RFC that's part of the libidn
library.
There's also a C++ & Java version.
http://www.gnu.org/software/libidn/

We've actually got a full implemention of stringprep as well - it's
much
more .Net 2.0 ish than the libidn one, which is just a native C++ app
that
was then ported to Java & .Net. It's found in our open-source SoapBox
Framework.

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

So how would you do ?

The mapping table idea you had before looked best to me, although I
wouldn't quite implement it the same way. I'd have a look up table
for
every possible character, where it defaults to the Unicode
character,
but for all the accented characters you care about, you specify the
non-accented version.

You'd then call ToCharArray() on the string in question, go through
each character replacing the original with the mapped character, and
then create a new string with the char array.

It does require you to manually map all the accented characters you
care about though.

My guess is that there are libraries around to do this somewhere,
but I
don't know of any myself.

 

[Ornette]

Hello,

Thank you for this really understandable article (even for french people
Nice & clear !!

Ornette

I'm sitting in front of my computer, and not feeling much like working
(too busy sneezing and being sick!), so I broke down and turned the
solution to this problem into a little blog entry.

http://www.coversant.com/Default.aspx?tabid=88&EntryID=30

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

Hello,

I following the thread which is very interesting.
I's thinking about wrinting a "StringHelper" class to do the jos and
re-use

Thanks a lot for your participation to this subject !!!

Ornette.

Opps. Definatly KD!

We want to do the decomposition & make our changes. For KC would
decompose & then perform a canonical recompose - which would defeat the
purpose!

I've never used (or even seen) the DecoderReplacementFallback - that's
another good idea. By now the original poster has probably given up and
will never try any of these solutions, but I think they would very
cleanly do the trick.

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

You meant NormalizationForm.FormKD.

Looking into it, I see a simpler method: After normalization, use
ASCIIEncoding with DecoderReplacementFallback replacing invalid ASCII
characters (which will be the accents) with the empty string.

JR

"Chris Mullins" <[email protected]> ???
??????:[email protected]...
I hadn't thought of that, but it's certainly an option.

Doing the normalization in .Net 2.0 is easy enough:

string s = "test";
string normalized = s.Normalize(NormalizationForm.FormKC);

Then you can iterate over the normalized string looking for (and
removing) the accents.

--
Chris Mullins


In a more general way:

There is a Unicode database at

http://www.unicode.org/Public/UNIDATA/

You could do what you want in two steps: decompose the string to base
characters followed by accent (NFKD normalization), then remove the
accents.

JR

"Chris Mullins [MVP]" <[email protected]> ëúá
áäåãòä:[email protected]...

The closest thing that comes to mind is an RFC called stringprep.
There
are a wide variety of stringprep profiles, and while they don't
quite do
what you're looking for, they're close. Included in stringprep is a
set of
mapping tables for Uppder->Lower case conversions. These are (in
that
context) called case-foldings, are are found in table B.2.
Unfortunatly,
they're Upper->Lower, not the other way around.

Stringprep:
http://www.faqs.org/rfcs/rfc3454.html

There are a number of profiles:
[Profile for Internaional Domain Names]
http://www.rfc-editor.org/rfc/rfc3491.txt

[Profile for iSCSI names]
http://tools.ietf.org/html/draft-ietf-ips-iscsi-string-prep-01

[Profile for SASL UserNames & Passwords]
http://www.ietf.org/rfc/rfc4013.txt

[Profile for XMPP Resources]
http://www.xmpp.org/internet-drafts/attic/draft-ietf-xmpp-resourcepre...

There's a C# implementation of this RFC that's part of the libidn
library.
There's also a C++ & Java version.
http://www.gnu.org/software/libidn/

We've actually got a full implemention of stringprep as well - it's
much
more .Net 2.0 ish than the libidn one, which is just a native C++
app that
was then ported to Java & .Net. It's found in our open-source
SoapBox
Framework.

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

So how would you do ?

The mapping table idea you had before looked best to me, although I
wouldn't quite implement it the same way. I'd have a look up table
for
every possible character, where it defaults to the Unicode
character,
but for all the accented characters you care about, you specify the
non-accented version.

You'd then call ToCharArray() on the string in question, go through
each character replacing the original with the mapped character,
and
then create a new string with the char array.

It does require you to manually map all the accented characters you
care about though.

My guess is that there are libraries around to do this somewhere,
but I
don't know of any myself.

 

[Cor Ligthert [MVP]]

Chris,

Maybe for your blog, be aware that there is a Microsoft VisualBasic method
StrConv(mystring, VbStrConv.Narrow)

However I saw that it only is converting 16bit Asian languages to 8 bit not
European languages (Latin characters).

Cor

I'm sitting in front of my computer, and not feeling much like working
(too busy sneezing and being sick!), so I broke down and turned the
solution to this problem into a little blog entry.

http://www.coversant.com/Default.aspx?tabid=88&EntryID=30

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

Hello,

I following the thread which is very interesting.
I's thinking about wrinting a "StringHelper" class to do the jos and
re-use

Thanks a lot for your participation to this subject !!!

Ornette.

Opps. Definatly KD!

We want to do the decomposition & make our changes. For KC would
decompose & then perform a canonical recompose - which would defeat the
purpose!

I've never used (or even seen) the DecoderReplacementFallback - that's
another good idea. By now the original poster has probably given up and
will never try any of these solutions, but I think they would very
cleanly do the trick.

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

You meant NormalizationForm.FormKD.

Looking into it, I see a simpler method: After normalization, use
ASCIIEncoding with DecoderReplacementFallback replacing invalid ASCII
characters (which will be the accents) with the empty string.

JR

"Chris Mullins" <[email protected]> ???
??????:[email protected]...
I hadn't thought of that, but it's certainly an option.

Doing the normalization in .Net 2.0 is easy enough:

string s = "test";
string normalized = s.Normalize(NormalizationForm.FormKC);

Then you can iterate over the normalized string looking for (and
removing) the accents.

--
Chris Mullins


In a more general way:

There is a Unicode database at

http://www.unicode.org/Public/UNIDATA/

You could do what you want in two steps: decompose the string to base
characters followed by accent (NFKD normalization), then remove the
accents.

JR

"Chris Mullins [MVP]" <[email protected]> ëúá
áäåãòä:[email protected]...

The closest thing that comes to mind is an RFC called stringprep.
There
are a wide variety of stringprep profiles, and while they don't
quite do
what you're looking for, they're close. Included in stringprep is a
set of
mapping tables for Uppder->Lower case conversions. These are (in
that
context) called case-foldings, are are found in table B.2.
Unfortunatly,
they're Upper->Lower, not the other way around.

Stringprep:
http://www.faqs.org/rfcs/rfc3454.html

There are a number of profiles:
[Profile for Internaional Domain Names]
http://www.rfc-editor.org/rfc/rfc3491.txt

[Profile for iSCSI names]
http://tools.ietf.org/html/draft-ietf-ips-iscsi-string-prep-01

[Profile for SASL UserNames & Passwords]
http://www.ietf.org/rfc/rfc4013.txt

[Profile for XMPP Resources]
http://www.xmpp.org/internet-drafts/attic/draft-ietf-xmpp-resourcepre...

There's a C# implementation of this RFC that's part of the libidn
library.
There's also a C++ & Java version.
http://www.gnu.org/software/libidn/

We've actually got a full implemention of stringprep as well - it's
much
more .Net 2.0 ish than the libidn one, which is just a native C++
app that
was then ported to Java & .Net. It's found in our open-source
SoapBox
Framework.

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

So how would you do ?

The mapping table idea you had before looked best to me, although I
wouldn't quite implement it the same way. I'd have a look up table
for
every possible character, where it defaults to the Unicode
character,
but for all the accented characters you care about, you specify the
non-accented version.

You'd then call ToCharArray() on the string in question, go through
each character replacing the original with the mapped character,
and
then create a new string with the char array.

It does require you to manually map all the accented characters you
care about though.

My guess is that there are libraries around to do this somewhere,
but I
don't know of any myself.

 

[Chris Mullins]

As was pointed out to me in a blog comment, my original solution to
this problem - using the ASCII encoder to avoid the string iteration -
was pretty silly.

The code in my blog has been updated to use the correct method - the
Unicode Character Info class. More details at:
http://www.coversant.com/Default.aspx?tabid=88&EntryID=30