CryptoStream/MemoryStream problems

  • Thread starter Thread starter Stingray
  • Start date Start date
S

Stingray

Are there any know problems with using a MemoryStream as a backing store for
a CryptoStream? I'm trying to simply encrypt and decrypt text in memory.
I'd like to create some simple methods to encrypt text before writing to a
database and decrypt it when reading it back. I figured I'd use a
MemoryStream that I can either store as a blob or convert to a string and
store as a varchar. Anyway, I can't get the CryptoStream to write to the
MemoryStream. The buffer remains empty. If I use a FileStream it works
great. But I don't want to write to a file.

Is there something I need to do differently when using a MemoryStream? Are
there any know issues/incompatiblities with CryptoStream and MemoryStream?
Is there an easier way for me to do what I want to do?

TIA,
John
 
Hi John,

There are no such issues... it's just you not doing it right.

If you search VS help, keywords "cryptostream memorystream", will find several samples using MemoryStreams.

I should warn you for a different problem you will face after getting your buffer full of ciphertext.

It's a common (newbie) misunderstanding, about storing ciphertext as strings. The point is you shouldn't. Converting a buffer into a string and vice-versa is called Text encoding and decoding. There are several encoding schemes (Ascii, Unicode, UTF8, etc). Each ciphertext byte can range from 0 to 255. Encoding schemes do not preserve all those unique values. And a single wrong byte is enough to make decryption impossible.

You should always store and handle ciphertext as a buffer (array of bytes). If you really need to store it as a string then you need to get a Base encoding representation of the buffer (hexadecimal, or Base64/Mime). You can create your own Base encoding functions or use framework System.Convert.ToBase64(buffer).

Regards,
Mario
 
Thanks Mario,

The search on my VS help didn't return anything useful but I searched on MSDN and found a good example. I was missing a FlushFinalBlock() on the CryptoStream (evidently a Flush() isn't enough). Anyway it works now!

Regarding conversion to/from string, is the following code "safe"?

byte[] buffer = ... // buffer contains encrypted byte array
// convert byte array to string
string encryptedString = Convert.ToBase64String (buffer, 0, buffer.Length);
// store encrypted string in an nvarchar in database
....
// read from database
....
// convert back to byte array
byte[] buffer = Convert.FromBase64String (encryptedString);

This seems to work perfectly, but let me know if there's something that will bite me. I can store it in the database as varbinary directly from the byte array or as an nvarchar (converted as above) and they both seem to work. I know you recommend keeping it in a byte array, but text is easier to work with, especially when it needs to be exported to xml and back. But I can deal with this if it is necessary. Is there a "standard" way that most people store encrypted values?

Also, do you have a recommendation on which algrorithm is best (Rijndael, DES, etc)? I'm using Rijndael right now. Thanks again.

John

Hi John,

There are no such issues... it's just you not doing it right.

If you search VS help, keywords "cryptostream memorystream", will find several samples using MemoryStreams.

I should warn you for a different problem you will face after getting your buffer full of ciphertext.

It's a common (newbie) misunderstanding, about storing ciphertext as strings. The point is you shouldn't. Converting a buffer into a string and vice-versa is called Text encoding and decoding. There are several encoding schemes (Ascii, Unicode, UTF8, etc). Each ciphertext byte can range from 0 to 255. Encoding schemes do not preserve all those unique values. And a single wrong byte is enough to make decryption impossible.

You should always store and handle ciphertext as a buffer (array of bytes). If you really need to store it as a string then you need to get a Base encoding representation of the buffer (hexadecimal, or Base64/Mime). You can create your own Base encoding functions or use framework System.Convert.ToBase64(buffer).

Regards,
Mario
 
Hi John,



Glad to know it's working now. Yes, FlushFinalBlock() is needed for padding. Symmetric algorithms are block ciphers, and many times plaintext length don't fill the last block completely (for instance, a 25 bytes buffer don't fill the second block completely; because Rijndael block size is 16 bytes, 7 bytes are missing to fill the last block). So we need to pad it with zeros or other byte value. Zero padding cannot be removed unambiguously because last plaintext bytes could be zeros as well. That's why we use a padding scheme called PKCS7 that has rules to remove padded bytes and get the same exact text. Framework default padding mode is PKCS7. As long as you don't change the default padding mode, you don't need to worry about padding at all. It's all done on your back by framework classes. I'm just saying this for curiosity.



Yes, correct, your Base64 encoding/decoding code is safe.



You can store safe Base64 encoded strings in database, no problem at all.

Base64 encoded strings take a little bit more space in the database, but are easier to use.



Most people today use Base64, taking advantage of that Framework Convert class.

In the old days, Hexadecimal format was popular, because Hex encoding is simpler than Base64; and as there was none, people implemented Hex instead of Base64.

I recommend you use Base64 strings.



You made a good choice picking Rijndael. Rijndael is AES, the new NIST standard that replaces DES.

Rijndael can work with several key sizes (128 bits, 192, 256); larger keys are safer, and take the same CPU time to encrypt/decrypt. On the other side, 128 bits (16 bytes) are safe enough. So, if you don't mind handling 32 bytes keys instead of 16 bytes keys, you can choose 256 bits keysize.



Best Regards,

Mario


"Stingray" <nospam> wrote in message Thanks Mario,

The search on my VS help didn't return anything useful but I searched on MSDN and found a good example. I was missing a FlushFinalBlock() on the CryptoStream (evidently a Flush() isn't enough). Anyway it works now!

Regarding conversion to/from string, is the following code "safe"?

byte[] buffer = ... // buffer contains encrypted byte array
// convert byte array to string
string encryptedString = Convert.ToBase64String (buffer, 0, buffer.Length);
// store encrypted string in an nvarchar in database
...
// read from database
...
// convert back to byte array
byte[] buffer = Convert.FromBase64String (encryptedString);

This seems to work perfectly, but let me know if there's something that will bite me. I can store it in the database as varbinary directly from the byte array or as an nvarchar (converted as above) and they both seem to work. I know you recommend keeping it in a byte array, but text is easier to work with, especially when it needs to be exported to xml and back. But I can deal with this if it is necessary. Is there a "standard" way that most people store encrypted values?

Also, do you have a recommendation on which algrorithm is best (Rijndael, DES, etc)? I'm using Rijndael right now. Thanks again.

John

Hi John,

There are no such issues... it's just you not doing it right.

If you search VS help, keywords "cryptostream memorystream", will find several samples using MemoryStreams.

I should warn you for a different problem you will face after getting your buffer full of ciphertext.

It's a common (newbie) misunderstanding, about storing ciphertext as strings. The point is you shouldn't. Converting a buffer into a string and vice-versa is called Text encoding and decoding. There are several encoding schemes (Ascii, Unicode, UTF8, etc). Each ciphertext byte can range from 0 to 255. Encoding schemes do not preserve all those unique values. And a single wrong byte is enough to make decryption impossible.

You should always store and handle ciphertext as a buffer (array of bytes). If you really need to store it as a string then you need to get a Base encoding representation of the buffer (hexadecimal, or Base64/Mime). You can create your own Base encoding functions or use framework System.Convert.ToBase64(buffer).

Regards,
Mario
 
Mario,
Thank you so much for explaining all of this. You've been very helpful.
John

Hi John,



Glad to know it's working now. Yes, FlushFinalBlock() is needed for padding. Symmetric algorithms are block ciphers, and many times plaintext length don't fill the last block completely (for instance, a 25 bytes buffer don't fill the second block completely; because Rijndael block size is 16 bytes, 7 bytes are missing to fill the last block). So we need to pad it with zeros or other byte value. Zero padding cannot be removed unambiguously because last plaintext bytes could be zeros as well. That's why we use a padding scheme called PKCS7 that has rules to remove padded bytes and get the same exact text. Framework default padding mode is PKCS7. As long as you don't change the default padding mode, you don't need to worry about padding at all. It's all done on your back by framework classes. I'm just saying this for curiosity.



Yes, correct, your Base64 encoding/decoding code is safe.



You can store safe Base64 encoded strings in database, no problem at all.

Base64 encoded strings take a little bit more space in the database, but are easier to use.



Most people today use Base64, taking advantage of that Framework Convert class.

In the old days, Hexadecimal format was popular, because Hex encoding is simpler than Base64; and as there was none, people implemented Hex instead of Base64.

I recommend you use Base64 strings.



You made a good choice picking Rijndael. Rijndael is AES, the new NIST standard that replaces DES.

Rijndael can work with several key sizes (128 bits, 192, 256); larger keys are safer, and take the same CPU time to encrypt/decrypt. On the other side, 128 bits (16 bytes) are safe enough. So, if you don't mind handling 32 bytes keys instead of 16 bytes keys, you can choose 256 bits keysize.



Best Regards,

Mario


"Stingray" <nospam> wrote in message Thanks Mario,

The search on my VS help didn't return anything useful but I searched on MSDN and found a good example. I was missing a FlushFinalBlock() on the CryptoStream (evidently a Flush() isn't enough). Anyway it works now!

Regarding conversion to/from string, is the following code "safe"?

byte[] buffer = ... // buffer contains encrypted byte array
// convert byte array to string
string encryptedString = Convert.ToBase64String (buffer, 0, buffer.Length);
// store encrypted string in an nvarchar in database
...
// read from database
...
// convert back to byte array
byte[] buffer = Convert.FromBase64String (encryptedString);

This seems to work perfectly, but let me know if there's something that will bite me. I can store it in the database as varbinary directly from the byte array or as an nvarchar (converted as above) and they both seem to work. I know you recommend keeping it in a byte array, but text is easier to work with, especially when it needs to be exported to xml and back. But I can deal with this if it is necessary. Is there a "standard" way that most people store encrypted values?

Also, do you have a recommendation on which algrorithm is best (Rijndael, DES, etc)? I'm using Rijndael right now. Thanks again.

John

Hi John,

There are no such issues... it's just you not doing it right.

If you search VS help, keywords "cryptostream memorystream", will find several samples using MemoryStreams.

I should warn you for a different problem you will face after getting your buffer full of ciphertext.

It's a common (newbie) misunderstanding, about storing ciphertext as strings. The point is you shouldn't. Converting a buffer into a string and vice-versa is called Text encoding and decoding. There are several encoding schemes (Ascii, Unicode, UTF8, etc). Each ciphertext byte can range from 0 to 255. Encoding schemes do not preserve all those unique values. And a single wrong byte is enough to make decryption impossible.

You should always store and handle ciphertext as a buffer (array of bytes). If you really need to store it as a string then you need to get a Base encoding representation of the buffer (hexadecimal, or Base64/Mime). You can create your own Base encoding functions or use framework System.Convert.ToBase64(buffer).

Regards,
Mario
 
One other thing to look out for - Rijndael streams can be self correcting. If you change a bit in the encrypted data, part of the plaintext will come out okay, and part of it will be corrupted. Rijndael won't tell you if there is an error.

To correct this, run your plaintext through an SHA1 hash and encrypt it along with your data. I find it is easy to put the Base64 version of the hash, along with the plaintext in a CDATA section, into an XML document and encrypt the whole thing. When you decrypt, you can tell if the data has been corrupted in any way by comparing hash values.

Oh, and if your plaintext is an XML document, you can use Zero Padding just fine.
"Stingray" <nospam> wrote in message Mario,
Thank you so much for explaining all of this. You've been very helpful.
John

Hi John,



Glad to know it's working now. Yes, FlushFinalBlock() is needed for padding. Symmetric algorithms are block ciphers, and many times plaintext length don't fill the last block completely (for instance, a 25 bytes buffer don't fill the second block completely; because Rijndael block size is 16 bytes, 7 bytes are missing to fill the last block). So we need to pad it with zeros or other byte value. Zero padding cannot be removed unambiguously because last plaintext bytes could be zeros as well. That's why we use a padding scheme called PKCS7 that has rules to remove padded bytes and get the same exact text. Framework default padding mode is PKCS7. As long as you don't change the default padding mode, you don't need to worry about padding at all. It's all done on your back by framework classes. I'm just saying this for curiosity.



Yes, correct, your Base64 encoding/decoding code is safe.



You can store safe Base64 encoded strings in database, no problem at all.

Base64 encoded strings take a little bit more space in the database, but are easier to use.



Most people today use Base64, taking advantage of that Framework Convert class.

In the old days, Hexadecimal format was popular, because Hex encoding is simpler than Base64; and as there was none, people implemented Hex instead of Base64.

I recommend you use Base64 strings.



You made a good choice picking Rijndael. Rijndael is AES, the new NIST standard that replaces DES.

Rijndael can work with several key sizes (128 bits, 192, 256); larger keys are safer, and take the same CPU time to encrypt/decrypt. On the other side, 128 bits (16 bytes) are safe enough. So, if you don't mind handling 32 bytes keys instead of 16 bytes keys, you can choose 256 bits keysize.



Best Regards,

Mario


"Stingray" <nospam> wrote in message Thanks Mario,

The search on my VS help didn't return anything useful but I searched on MSDN and found a good example. I was missing a FlushFinalBlock() on the CryptoStream (evidently a Flush() isn't enough). Anyway it works now!

Regarding conversion to/from string, is the following code "safe"?

byte[] buffer = ... // buffer contains encrypted byte array
// convert byte array to string
string encryptedString = Convert.ToBase64String (buffer, 0, buffer.Length);
// store encrypted string in an nvarchar in database
...
// read from database
...
// convert back to byte array
byte[] buffer = Convert.FromBase64String (encryptedString);

This seems to work perfectly, but let me know if there's something that will bite me. I can store it in the database as varbinary directly from the byte array or as an nvarchar (converted as above) and they both seem to work. I know you recommend keeping it in a byte array, but text is easier to work with, especially when it needs to be exported to xml and back. But I can deal with this if it is necessary. Is there a "standard" way that most people store encrypted values?

Also, do you have a recommendation on which algrorithm is best (Rijndael, DES, etc)? I'm using Rijndael right now. Thanks again.

John

Hi John,

There are no such issues... it's just you not doing it right.

If you search VS help, keywords "cryptostream memorystream", will find several samples using MemoryStreams.

I should warn you for a different problem you will face after getting your buffer full of ciphertext.

It's a common (newbie) misunderstanding, about storing ciphertext as strings. The point is you shouldn't. Converting a buffer into a string and vice-versa is called Text encoding and decoding. There are several encoding schemes (Ascii, Unicode, UTF8, etc). Each ciphertext byte can range from 0 to 255. Encoding schemes do not preserve all those unique values. And a single wrong byte is enough to make decryption impossible.

You should always store and handle ciphertext as a buffer (array of bytes). If you really need to store it as a string then you need to get a Base encoding representation of the buffer (hexadecimal, or Base64/Mime). You can create your own Base encoding functions or use framework System.Convert.ToBase64(buffer).

Regards,
Mario
 
Thanks Jason. I'll keep this in mind also.

I do have one further question though for anyone: Where is a good place to store keys? Apps need to be able to get to them easily but most people should not.

John
One other thing to look out for - Rijndael streams can be self correcting. If you change a bit in the encrypted data, part of the plaintext will come out okay, and part of it will be corrupted. Rijndael won't tell you if there is an error.

To correct this, run your plaintext through an SHA1 hash and encrypt it along with your data. I find it is easy to put the Base64 version of the hash, along with the plaintext in a CDATA section, into an XML document and encrypt the whole thing. When you decrypt, you can tell if the data has been corrupted in any way by comparing hash values.

Oh, and if your plaintext is an XML document, you can use Zero Padding just fine.
"Stingray" <nospam> wrote in message Mario,
Thank you so much for explaining all of this. You've been very helpful.
John

Hi John,



Glad to know it's working now. Yes, FlushFinalBlock() is needed for padding. Symmetric algorithms are block ciphers, and many times plaintext length don't fill the last block completely (for instance, a 25 bytes buffer don't fill the second block completely; because Rijndael block size is 16 bytes, 7 bytes are missing to fill the last block). So we need to pad it with zeros or other byte value. Zero padding cannot be removed unambiguously because last plaintext bytes could be zeros as well. That's why we use a padding scheme called PKCS7 that has rules to remove padded bytes and get the same exact text. Framework default padding mode is PKCS7. As long as you don't change the default padding mode, you don't need to worry about padding at all. It's all done on your back by framework classes. I'm just saying this for curiosity.



Yes, correct, your Base64 encoding/decoding code is safe.



You can store safe Base64 encoded strings in database, no problem at all.

Base64 encoded strings take a little bit more space in the database, but are easier to use.



Most people today use Base64, taking advantage of that Framework Convert class.

In the old days, Hexadecimal format was popular, because Hex encoding is simpler than Base64; and as there was none, people implemented Hex instead of Base64.

I recommend you use Base64 strings.



You made a good choice picking Rijndael. Rijndael is AES, the new NIST standard that replaces DES.

Rijndael can work with several key sizes (128 bits, 192, 256); larger keys are safer, and take the same CPU time to encrypt/decrypt. On the other side, 128 bits (16 bytes) are safe enough. So, if you don't mind handling 32 bytes keys instead of 16 bytes keys, you can choose 256 bits keysize.



Best Regards,

Mario


"Stingray" <nospam> wrote in message Thanks Mario,

The search on my VS help didn't return anything useful but I searched on MSDN and found a good example. I was missing a FlushFinalBlock() on the CryptoStream (evidently a Flush() isn't enough). Anyway it works now!

Regarding conversion to/from string, is the following code "safe"?

byte[] buffer = ... // buffer contains encrypted byte array
// convert byte array to string
string encryptedString = Convert.ToBase64String (buffer, 0, buffer.Length);
// store encrypted string in an nvarchar in database
...
// read from database
...
// convert back to byte array
byte[] buffer = Convert.FromBase64String (encryptedString);

This seems to work perfectly, but let me know if there's something that will bite me. I can store it in the database as varbinary directly from the byte array or as an nvarchar (converted as above) and they both seem to work. I know you recommend keeping it in a byte array, but text is easier to work with, especially when it needs to be exported to xml and back. But I can deal with this if it is necessary. Is there a "standard" way that most people store encrypted values?

Also, do you have a recommendation on which algrorithm is best (Rijndael, DES, etc)? I'm using Rijndael right now. Thanks again.

John

Hi John,

There are no such issues... it's just you not doing it right.

If you search VS help, keywords "cryptostream memorystream", will find several samples using MemoryStreams.

I should warn you for a different problem you will face after getting your buffer full of ciphertext.

It's a common (newbie) misunderstanding, about storing ciphertext as strings. The point is you shouldn't. Converting a buffer into a string and vice-versa is called Text encoding and decoding. There are several encoding schemes (Ascii, Unicode, UTF8, etc). Each ciphertext byte can range from 0 to 255. Encoding schemes do not preserve all those unique values. And a single wrong byte is enough to make decryption impossible.

You should always store and handle ciphertext as a buffer (array of bytes). If you really need to store it as a string then you need to get a Base encoding representation of the buffer (hexadecimal, or Base64/Mime). You can create your own Base encoding functions or use framework System.Convert.ToBase64(buffer).

Regards,
Mario
 
check out:
http://msdn.microsoft.com/msdnmag/issues/03/11/ProtectYourData/default.aspx



Thanks Jason. I'll keep this in mind also.

I do have one further question though for anyone: Where is a good place to store keys? Apps need to be able to get to them easily but most people should not.

John
One other thing to look out for - Rijndael streams can be self correcting. If you change a bit in the encrypted data, part of the plaintext will come out okay, and part of it will be corrupted. Rijndael won't tell you if there is an error.

To correct this, run your plaintext through an SHA1 hash and encrypt it along with your data. I find it is easy to put the Base64 version of the hash, along with the plaintext in a CDATA section, into an XML document and encrypt the whole thing. When you decrypt, you can tell if the data has been corrupted in any way by comparing hash values.

Oh, and if your plaintext is an XML document, you can use Zero Padding just fine.
"Stingray" <nospam> wrote in message Mario,
Thank you so much for explaining all of this. You've been very helpful.
John

Hi John,



Glad to know it's working now. Yes, FlushFinalBlock() is needed for padding. Symmetric algorithms are block ciphers, and many times plaintext length don't fill the last block completely (for instance, a 25 bytes buffer don't fill the second block completely; because Rijndael block size is 16 bytes, 7 bytes are missing to fill the last block). So we need to pad it with zeros or other byte value. Zero padding cannot be removed unambiguously because last plaintext bytes could be zeros as well. That's why we use a padding scheme called PKCS7 that has rules to remove padded bytes and get the same exact text. Framework default padding mode is PKCS7. As long as you don't change the default padding mode, you don't need to worry about padding at all. It's all done on your back by framework classes. I'm just saying this for curiosity.



Yes, correct, your Base64 encoding/decoding code is safe.



You can store safe Base64 encoded strings in database, no problem at all.

Base64 encoded strings take a little bit more space in the database, but are easier to use.



Most people today use Base64, taking advantage of that Framework Convert class.

In the old days, Hexadecimal format was popular, because Hex encoding is simpler than Base64; and as there was none, people implemented Hex instead of Base64.

I recommend you use Base64 strings.



You made a good choice picking Rijndael. Rijndael is AES, the new NIST standard that replaces DES.

Rijndael can work with several key sizes (128 bits, 192, 256); larger keys are safer, and take the same CPU time to encrypt/decrypt. On the other side, 128 bits (16 bytes) are safe enough. So, if you don't mind handling 32 bytes keys instead of 16 bytes keys, you can choose 256 bits keysize.



Best Regards,

Mario


"Stingray" <nospam> wrote in message Thanks Mario,

The search on my VS help didn't return anything useful but I searched on MSDN and found a good example. I was missing a FlushFinalBlock() on the CryptoStream (evidently a Flush() isn't enough). Anyway it works now!

Regarding conversion to/from string, is the following code "safe"?

byte[] buffer = ... // buffer contains encrypted byte array
// convert byte array to string
string encryptedString = Convert.ToBase64String (buffer, 0, buffer.Length);
// store encrypted string in an nvarchar in database
...
// read from database
...
// convert back to byte array
byte[] buffer = Convert.FromBase64String (encryptedString);

This seems to work perfectly, but let me know if there's something that will bite me. I can store it in the database as varbinary directly from the byte array or as an nvarchar (converted as above) and they both seem to work. I know you recommend keeping it in a byte array, but text is easier to work with, especially when it needs to be exported to xml and back. But I can deal with this if it is necessary. Is there a "standard" way that most people store encrypted values?

Also, do you have a recommendation on which algrorithm is best (Rijndael, DES, etc)? I'm using Rijndael right now. Thanks again.

John

Hi John,

There are no such issues... it's just you not doing it right.

If you search VS help, keywords "cryptostream memorystream", will find several samples using MemoryStreams.

I should warn you for a different problem you will face after getting your buffer full of ciphertext.

It's a common (newbie) misunderstanding, about storing ciphertext as strings. The point is you shouldn't. Converting a buffer into a string and vice-versa is called Text encoding and decoding. There are several encoding schemes (Ascii, Unicode, UTF8, etc). Each ciphertext byte can range from 0 to 255. Encoding schemes do not preserve all those unique values. And a single wrong byte is enough to make decryption impossible.

You should always store and handle ciphertext as a buffer (array of bytes). If you really need to store it as a string then you need to get a Base encoding representation of the buffer (hexadecimal, or Base64/Mime). You can create your own Base encoding functions or use framework System.Convert.ToBase64(buffer).

Regards,
Mario


Stingray said:
Are there any know problems with using a MemoryStream as a backing store for
a CryptoStream? I'm trying to simply encrypt and decrypt text in memory.
I'd like to create some simple methods to encrypt text before writing to a
database and decrypt it when reading it back. I figured I'd use a
MemoryStream that I can either store as a blob or convert to a string and
store as a varchar. Anyway, I can't get the CryptoStream to write to the
MemoryStream. The buffer remains empty. If I use a FileStream it works
great. But I don't want to write to a file.

Is there something I need to do differently when using a MemoryStream? Are
there any know issues/incompatiblities with CryptoStream and MemoryStream?
Is there an easier way for me to do what I want to do?

TIA,
John
 
Back
Top