StreamReader and NULL characters

  • Thread starter Thread starter brian.gabriel
  • Start date Start date
B

brian.gabriel

I am trying to read a file that was produced on a mainframe, it is a
text output of a mainframe report. I am able to read in part of the
file, the problem is that between each "page" of the report there is a
NULL character. The streamreader will read up until it sees this
character then it stops, the end result being that most of the file
does not get read.

Here is the byte pattern that is at the end of each page: 0D 0A 00 0C
0D 0A

Is there an easy way to read in this file without having to put it
into a byte array and futzing with all of that?

I have tried the following:

StreamReader re = File.OpenText(SourceFile);
string sTemp = re.ReadToEnd();

And:

StreamReader re = File.OpenText(SourceFile);
while((sTemp = re.ReadLine()) != null)
{
sLine = sTemp;
//etc...
}

Thanks,

Brian
(e-mail address removed)
 
NULL chars create a problem. One way to tackle this is to work with command
line tools to change the NULL char into another char, or even separate into
pages. This works well if the application in question is loading the data;
it is not a good solution for an application that reads the file(s) every
time someone uses a specific function, however.

You might be able to use a FileStream and pull the data in as binary. If
nothing else, you can look for the byte pattern 13-10-0-12-13-10 and replace
it with something like 2 CRLFs (13-10). The cleaned file can be read through
the StreamReader. You can also keep everything binary, if you feel
comfortable in that world. :-)

--
Gregory A. Beamer
MVP, MCP: +I, SE, SD, DBA

Subscribe to my blog
http://gregorybeamer.spaces.live.com/lists/feed.rss

or just read it:
http://gregorybeamer.spaces.live.com/

*************************************************
| Think outside the box!
|
*************************************************
 
From my mainframe days I can see that what you have there is Ebcdic, 0D =
CR, 0A = LF, 0C = form feed. I'm wondering if this means that you could use
the StreamReader ctor that takes an Encoding for an 8 bit Ebcdic and it
would just work. Anyway, knowing that this is Ebcdic CR LF FF might help.
 
Those are ASCII control codes, copied into ANSI and ISO code pages.

In EBCDIC, 0x0D = CR but no one used it, 0x25 = LF but no one used it, 0x15
= NL (newline) and it was occasionally used, and 0x0C = FF and I don't know
if anyone used it.

So the original poster receives data that have already been converted from
EBCDIC to ASCII, but that's not the problem. The problem is that
StreamReader chokes when it hits a 0x00 (NUL in both ASCII and EBCDIC).


Phil Wilson said:
From my mainframe days I can see that what you have there is Ebcdic, 0D =
CR, 0A = LF, 0C = form feed. I'm wondering if this means that you could
use the StreamReader ctor that takes an Encoding for an 8 bit Ebcdic and
it would just work. Anyway, knowing that this is Ebcdic CR LF FF might
help.
--
Phil Wilson
[MVP Windows Installer]

I am trying to read a file that was produced on a mainframe, it is a
text output of a mainframe report. I am able to read in part of the
file, the problem is that between each "page" of the report there is a
NULL character. The streamreader will read up until it sees this
character then it stops, the end result being that most of the file
does not get read.

Here is the byte pattern that is at the end of each page: 0D 0A 00 0C
0D 0A

Is there an easy way to read in this file without having to put it
into a byte array and futzing with all of that?

I have tried the following:

StreamReader re = File.OpenText(SourceFile);
string sTemp = re.ReadToEnd();

And:

StreamReader re = File.OpenText(SourceFile);
while((sTemp = re.ReadLine()) != null)
{
sLine = sTemp;
//etc...
}

Thanks,

Brian
(e-mail address removed)
Click to expand...
Click to expand...
 
Norman Diamond said:
Those are ASCII control codes, copied into ANSI and ISO code pages.

In EBCDIC, 0x0D = CR but no one used it, 0x25 = LF but no one used it, 0x15
= NL (newline) and it was occasionally used, and 0x0C = FF and I don't know
if anyone used it.

So the original poster receives data that have already been converted from
EBCDIC to ASCII, but that's not the problem. The problem is that
StreamReader chokes when it hits a 0x00 (NUL in both ASCII and EBCDIC).
Click to expand...

StreamReader doesn't choke on null characters. Here's an example:

using System;
using System.IO;
using System.Text;

class Test
{
static void Main(string[] args)
{
// A NUL B NUL C
byte[] data = { 65, 0, 66, 0, 67 };

using (MemoryStream stream = new MemoryStream(data))
using (StreamReader reader = new StreamReader
(stream, Encoding.ASCII))
{
string line = reader.ReadLine();
for (int i=0; i < line.Length; i++)
{
Console.WriteLine("{0}: {1}", i,
line=='\0'
? "NUL"
: line.ToString());
}
}
}
}
 
He was choking on NULL char due to the way he was looping:

while (!null == (line = reader.ReadLine()))
{
}

As soon as he hits a null char, he gets and end of read. I use this type of
loop, as well, as it is rather simple, but it will choke on (char) 0.

A method I employed, at one time, for EBCDIC, is running everything binary
until it needed to be text. Much more efficient to stay in the binary world,
from a perf perspective, but also harder to program, as we do not think in
0s and 1s. :-)

--
Gregory A. Beamer
MVP, MCP: +I, SE, SD, DBA

Subscribe to my blog
http://gregorybeamer.spaces.live.com/lists/feed.rss

or just read it:
http://gregorybeamer.spaces.live.com/

*************************************************
| Think outside the box!
|
*************************************************
Jon Skeet said:
Norman Diamond said:
Those are ASCII control codes, copied into ANSI and ISO code pages.

In EBCDIC, 0x0D = CR but no one used it, 0x25 = LF but no one used it,
0x15
= NL (newline) and it was occasionally used, and 0x0C = FF and I don't
know
if anyone used it.

So the original poster receives data that have already been converted
from
EBCDIC to ASCII, but that's not the problem. The problem is that
StreamReader chokes when it hits a 0x00 (NUL in both ASCII and EBCDIC).
Click to expand...

StreamReader doesn't choke on null characters. Here's an example:

using System;
using System.IO;
using System.Text;

class Test
{
static void Main(string[] args)
{
// A NUL B NUL C
byte[] data = { 65, 0, 66, 0, 67 };

using (MemoryStream stream = new MemoryStream(data))
using (StreamReader reader = new StreamReader
(stream, Encoding.ASCII))
{
string line = reader.ReadLine();
for (int i=0; i < line.Length; i++)
{
Console.WriteLine("{0}: {1}", i,
line=='\0'
? "NUL"
: line.ToString());
}
}
}
}
Click to expand...
 
Cowboy (Gregory A. Beamer) said:
He was choking on NULL char due to the way he was looping:

while (!null == (line = reader.ReadLine()))
{
}

As soon as he hits a null char, he gets and end of read.
Click to expand...

He shouldn't do - my example shows two null characters been embedded in
a line read with a single call of ReadLine.
I use this type of
loop, as well, as it is rather simple, but it will choke on (char) 0.
Click to expand...

Could you give an example of this, bearing mind my earlier example?
Here's another example using the OP's sample data - again, it doesn't
show StreamReader failing with null characters:

using System;
using System.IO;
using System.Text;

class Test
{
static void Main(string[] args)
{
// A B
byte[] data = { 65, 66, 0x0d, 0x0a, 0,
0x0c, 0x0d, 0x0a, 67, 68};

using (MemoryStream stream = new MemoryStream(data))
using (StreamReader reader = new StreamReader(stream))
{
string line;
while ((line=reader.ReadLine()) != null)
{
Console.WriteLine ("Next line:");
for (int i=0; i < line.Length; i++)
{
Console.WriteLine("{0}: {1}", i,
line=='\0'
? "NUL"
: line.ToString());
}
}
}
}
}
A method I employed, at one time, for EBCDIC, is running everything binary
until it needed to be text. Much more efficient to stay in the binary world,
from a perf perspective, but also harder to program, as we do not think in
0s and 1s. :-)
Click to expand...

It also means that *you* need to worry about all the nasty issues of
text handling rather than getting the framework to do it.
 
Thanks Jon!

After seeing your example I created a much small text file with the
offending characters and was able to better debug the issue. The
issue is not with the StreamReader, but with displaying the results in
a text box. A simple Replace removed the offending characters and
everything is displaying properly.

Again thanks for the help.

Brian
(e-mail address removed)

Could you give an example of this, bearing mind my earlier example?
Here's another example using the OP's sample data - again, it doesn't
show StreamReader failing with null characters:

using System;
using System.IO;
using System.Text;

class Test
{
    static void Main(string[] args)
    {
        // A B
        byte[] data = { 65, 66, 0x0d, 0x0a, 0,
                0x0c, 0x0d, 0x0a, 67, 68};

        using (MemoryStream stream = new MemoryStream(data))
        using (StreamReader reader = new StreamReader(stream))
        {
            string line;
            while ((line=reader.ReadLine()) != null)
            {
                Console.WriteLine ("Next line:");
                for (int i=0; i < line.Length; i++)
                {
                    Console.WriteLine("{0}: {1}", i,
                                      line=='\0'
                                      ? "NUL"
                                      : line.ToString());
                }
            }
        }
    }

}
A method I employed, at one time, for EBCDIC, is running everything binary
until it needed to be text. Much more efficient to stay in the binary world,
from a perf perspective, but also harder to program, as we do not think in
0s and 1s. :-)
Click to expand...

It also means that *you* need to worry about all the nasty issues of
text handling rather than getting the framework to do it.
Click to expand...
 
What do you mean by "nobody used it"? I used to see it all the time on
printers, some TTY green screens etc. It was (is?) very common in the 8-bit
EBCDIC mainframe world.
--
Phil Wilson
[MVP Windows Installer]

Norman Diamond said:
Those are ASCII control codes, copied into ANSI and ISO code pages.

In EBCDIC, 0x0D = CR but no one used it, 0x25 = LF but no one used it,
0x15 = NL (newline) and it was occasionally used, and 0x0C = FF and I
don't know if anyone used it.

So the original poster receives data that have already been converted from
EBCDIC to ASCII, but that's not the problem. The problem is that
StreamReader chokes when it hits a 0x00 (NUL in both ASCII and EBCDIC).


Phil Wilson said:
From my mainframe days I can see that what you have there is Ebcdic, 0D =
CR, 0A = LF, 0C = form feed. I'm wondering if this means that you could
use the StreamReader ctor that takes an Encoding for an 8 bit Ebcdic and
it would just work. Anyway, knowing that this is Ebcdic CR LF FF might
help.
--
Phil Wilson
[MVP Windows Installer]

I am trying to read a file that was produced on a mainframe, it is a
text output of a mainframe report. I am able to read in part of the
file, the problem is that between each "page" of the report there is a
NULL character. The streamreader will read up until it sees this
character then it stops, the end result being that most of the file
does not get read.

Here is the byte pattern that is at the end of each page: 0D 0A 00 0C
0D 0A

Is there an easy way to read in this file without having to put it
into a byte array and futzing with all of that?

I have tried the following:

StreamReader re = File.OpenText(SourceFile);
string sTemp = re.ReadToEnd();

And:

StreamReader re = File.OpenText(SourceFile);
while((sTemp = re.ReadLine()) != null)
{
sLine = sTemp;
//etc...
}

Thanks,

Brian
(e-mail address removed)
Click to expand...
Click to expand...
Click to expand...
 
OK, I never saw the 0x0D 0x25 sequence used in EBCDIC in the TTY green
screen world, I saw the 0x15 single byte newline character used in EBCDIC in
the TTY green screen world. If the TTY firmware (or hardware) used ASCII
then the driver would have to translate from EBCDIC to ASCII, but I didn't
see anyone do that translation for TTYs at the application level.


Phil Wilson said:
What do you mean by "nobody used it"? I used to see it all the time on
printers, some TTY green screens etc. It was (is?) very common in the
8-bit EBCDIC mainframe world.
--
Phil Wilson
[MVP Windows Installer]

Norman Diamond said:
Those are ASCII control codes, copied into ANSI and ISO code pages.

In EBCDIC, 0x0D = CR but no one used it, 0x25 = LF but no one used it,
0x15 = NL (newline) and it was occasionally used, and 0x0C = FF and I
don't know if anyone used it.

So the original poster receives data that have already been converted
from EBCDIC to ASCII, but that's not the problem. The problem is that
StreamReader chokes when it hits a 0x00 (NUL in both ASCII and EBCDIC).


Phil Wilson said:
From my mainframe days I can see that what you have there is Ebcdic, 0D
= CR, 0A = LF, 0C = form feed. I'm wondering if this means that you
could use the StreamReader ctor that takes an Encoding for an 8 bit
Ebcdic and it would just work. Anyway, knowing that this is Ebcdic CR
LF FF might help.
--
Phil Wilson
[MVP Windows Installer]

I am trying to read a file that was produced on a mainframe, it is a
text output of a mainframe report. I am able to read in part of the
file, the problem is that between each "page" of the report there is a
NULL character. The streamreader will read up until it sees this
character then it stops, the end result being that most of the file
does not get read.

Here is the byte pattern that is at the end of each page: 0D 0A 00 0C
0D 0A

Is there an easy way to read in this file without having to put it
into a byte array and futzing with all of that?

I have tried the following:

StreamReader re = File.OpenText(SourceFile);
string sTemp = re.ReadToEnd();

And:

StreamReader re = File.OpenText(SourceFile);
while((sTemp = re.ReadLine()) != null)
{
sLine = sTemp;
//etc...
}

Thanks,

Brian
(e-mail address removed)
Click to expand...
Click to expand...
Click to expand...
 
Duh!

I should have thought about that. The first NULL char will destroy display
at that point.

--
Gregory A. Beamer
MVP, MCP: +I, SE, SD, DBA

Subscribe to my blog
http://gregorybeamer.spaces.live.com/lists/feed.rss

or just read it:
http://gregorybeamer.spaces.live.com/

*************************************************
| Think outside the box!
|
*************************************************
Thanks Jon!

After seeing your example I created a much small text file with the
offending characters and was able to better debug the issue. The
issue is not with the StreamReader, but with displaying the results in
a text box. A simple Replace removed the offending characters and
everything is displaying properly.

Again thanks for the help.

Brian
(e-mail address removed)

Could you give an example of this, bearing mind my earlier example?
Here's another example using the OP's sample data - again, it doesn't
show StreamReader failing with null characters:

using System;
using System.IO;
using System.Text;

class Test
{
static void Main(string[] args)
{
// A B
byte[] data = { 65, 66, 0x0d, 0x0a, 0,
0x0c, 0x0d, 0x0a, 67, 68};

using (MemoryStream stream = new MemoryStream(data))
using (StreamReader reader = new StreamReader(stream))
{
string line;
while ((line=reader.ReadLine()) != null)
{
Console.WriteLine ("Next line:");
for (int i=0; i < line.Length; i++)
{
Console.WriteLine("{0}: {1}", i,
line=='\0'
? "NUL"
: line.ToString());
}
}
}
}

}
A method I employed, at one time, for EBCDIC, is running everything
binary
until it needed to be text. Much more efficient to stay in the binary
world,
from a perf perspective, but also harder to program, as we do not think
in
0s and 1s. :-)
Click to expand...

It also means that *you* need to worry about all the nasty issues of
text handling rather than getting the framework to do it.
Click to expand...
 
Back
Top