file I/O

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

Inorder to handle large files over 100 MB, is there a way in dotnet to open
only certain bytes of the file by buffer ?

It is shaky if the entire file is opened and splitted. Instead is there a
way or any alternatives on opening small chunk of the file.
 
I don't think there is any different between opening a 1M file or 100M file
if you use a stream. Streams are designed with buffers and will only read
in as much information as you want. So, you could open your file using
FileStream and seek to a specified location, then read as many bytes as you
want.

Now, if you need more complex processing like getting the 200th row of some
file that are delimited by newlines, then that is something that you might
have to process yourself or rely on something like a Stream reader.
 
Thank you for the feedback Sir. Well, i am just trying to print a 30MB file
and its shaky. It gets hogged sometimes amd prints sometimes as if it works
when we have luck. Unless or otherwise something is wrong with the following
code. There is lot more transactions to do than just printing, but it is
skaky right here when printing.

Private Sub Button1_Click(ByVal sender As Object, ByVal e As
System.EventArgs) Handles Button1.Click
'Open a file for reading
Dim FILENAME As String
Dim FileContents As String
Dim iCount As Integer
iCount = 1

FILENAME = "c:\\Filereader\\File1.txt"

Try

'Get a StreamReader class that can be used to read the file
Dim objStreamReader As StreamReader
objStreamReader = File.OpenText(FILENAME)

'Read Line By Line
While True
FileContents = CType(objStreamReader.ReadLine, String)
If Not FileContents = Nothing Then
Response.Write(iCount & "Begin Line1" & "<br>")
Response.Write(FileContents)
Response.Write("<br>" & iCount & "End Line1" & "<br>")
iCount = iCount + 1
'lblRawOutput.Text &= FileContents & vbCrLf
Else : Exit While
End If
End While

'Set the text of the file to a Web control
'lblRawOutput.Text = contents

objStreamReader.Close()

Catch ex As Exception
Response.Write(ex.Message)
End Try

End Sub
 
Thank you for the feedback Sir. Well, i am just trying to print a 30MB file
and its shaky. It gets hogged sometimes amd prints sometimes as if it works
when we have luck. Unless or otherwise something is wrong with the following
code. There is lot more transactions to do than just printing, but it is
skaky right here when printing.

Code: 
Private Sub Button1_Click(ByVal sender As Object, ByVal e As
System.EventArgs) Handles Button1.Click
'Open a file for reading
Dim FILENAME As String
Dim FileContents As String
Dim iCount As Integer
iCount = 1

FILENAME = "c:\\Filereader\\File1.txt"

Try

'Get a StreamReader class that can be used to read the file
Dim objStreamReader As StreamReader
objStreamReader = File.OpenText(FILENAME)

'Read Line By Line
While True
FileContents = CType(objStreamReader.ReadLine, String)
If Not FileContents = Nothing Then
Response.Write(iCount & "Begin Line1" & "<br>")
Response.Write(FileContents)
Response.Write("<br>" & iCount & "End Line1" & "<br>")
iCount = iCount + 1
'lblRawOutput.Text &= FileContents & vbCrLf
Else : Exit While
End If
End While

'Set the text of the file to a Web control
'lblRawOutput.Text = contents

objStreamReader.Close()

Catch ex As Exception
Response.Write(ex.Message)
End Try

End Sub
 
Thank you for the feedback Sir. Well, i am just trying to print a 30MB file
and its shaky. It gets hogged sometimes amd prints sometimes as if it works
when we have luck. Unless or otherwise something is wrong with the following
code. There is lot more transactions to do than just printing, but it is
skaky right here when printing.

Code: 
Private Sub Button1_Click(ByVal sender As Object, ByVal e As
System.EventArgs) Handles Button1.Click
'Open a file for reading
Dim FILENAME As String
Dim FileContents As String
Dim iCount As Integer
iCount = 1

FILENAME = "c:\\Filereader\\File1.txt"

Try

'Get a StreamReader class that can be used to read the file
Dim objStreamReader As StreamReader
objStreamReader = File.OpenText(FILENAME)

'Read Line By Line
While True
FileContents = CType(objStreamReader.ReadLine, String)
If Not FileContents = Nothing Then
Response.Write(iCount & "Begin Line1" & "<br>")
Response.Write(FileContents)
Response.Write("<br>" & iCount & "End Line1" & "<br>")
iCount = iCount + 1
'lblRawOutput.Text &= FileContents & vbCrLf
Else : Exit While
End If
End While

'Set the text of the file to a Web control
'lblRawOutput.Text = contents

objStreamReader.Close()

Catch ex As Exception
Response.Write(ex.Message)
End Try

End Sub
 
Shiva,

Tell us first what type of file it is, text, a kind of own database, are
there char in it etc. or maybe it are pictures. Now it is just shooting in
the dark.


Cor
 
shiva said:
Thank you for the feedback Sir. Well, i am just trying to print
a 30MB file
and its shaky. It gets hogged sometimes amd prints sometimes as
if it works
when we have luck. Unless or otherwise something is wrong with
the following
code. There is lot more transactions to do than just printing,
but it is
skaky right here when printing.

Private Sub Button1_Click(ByVal sender As Object, ByVal e As
System.EventArgs) Handles Button1.Click
'Open a file for reading
Dim FILENAME As String
Dim FileContents As String
Dim iCount As Integer
iCount = 1

FILENAME = "c:\\Filereader\\File1.txt"

Try

'Get a StreamReader class that can be used to read
the file
Dim objStreamReader As StreamReader
objStreamReader = File.OpenText(FILENAME)

'Read Line By Line
While True
FileContents = CType(objStreamReader.ReadLine,
String)
Click to expand...

ReadLine returns a string already, so there is no reason to
re-caste it.
If Not FileContents = Nothing Then
Response.Write(iCount & "Begin Line1" &
"<br>")
Response.Write(FileContents)
Response.Write("<br>" & iCount & "End Line1"
& "<br>")
iCount = iCount + 1
'lblRawOutput.Text &= FileContents & vbCrLf
Else : Exit While
End If
End While
Click to expand...

It doesn't look like your speed issues are due to the way you are
working with the files, but rather with what you are doing with
all the data.

What is this Response object you are working with? I'm assuming
it's something like a web response. If this is the case then it
can take quite a while to send the 30 MB of data via the network.
Even on a local network it can take a while depending on network
configuration and current traffic loads.

Also you have
'lblRawOutput.Text &= FileContents & vbCrLf
commented out. However, if your tests were done with this
uncommented it could have a drastic performance hit for large
files, as the larger a string gets, the longer it takes to
concatenate more data on to the end of it.

Andrew Faust
 
shiva said:
Dear Cor,

For example if the file is a 30MB file, i am looking for
opening for 1MB
at a time.
Click to expand...

You can't open only 1 MB at a time. You either have a file open
or you don't. Size is irrelevant. However, you can read in to
memory only a small portion at a time.

Andrew Faust
 
Shiva,

When you process it line by line why it can than not be in a kind of pseudo
Private rowcounter = 0

Process 10000rows(not endoffile)

Procedure Process10000rows
do while not endoffile
For rowcounter to 10000
read line into whatever
next

When this can fit and you cannot make this in real code tell than?

Cor
 
Back
Top