How best to view huge amounts of text?

  • Thread starter Thread starter Chris Mullins
  • Start date Start date
C

Chris Mullins

I'm building a GUI that needs to be able to view a large amount of text
arranged in rows. Large being anywhere from a few hundred lines through a
few hundred thousand. I need a way to "cap" the max number of rows, so that
old rows are discarded in favor of new rows if the limit is reached.

My use case is very similar to that of the SQL Profiler GUI - I'm going to
be receiving a large amount of data from a server that I want to display.
I'm fine defining a "window" on the data, or a max number of rows on the
control (say 200K) and throw out old data as the new data comes in.

I started out with a List View control, and appended list view items to the
beginning, and deleted items from the end. This worked fairly well, but
would have very strange behavior on occasion. It would, on a not infrequent
basis, hang the entire app-domain, including the VS.NET debugger. Very, very
frustrating.

I then moved on to a DataGrid bound to a typed dataset. I manage the rows in
the dataset and discard old rows, and only display "new" rows. This works
alright, but has serious performance issues. I haven't tried using the data
grid in "unbound" mode yet.

I am receiving data from the server on a background thread - so this gives
me some leeway to massage data prior to marshalling over to the GUI thread
and updating the control. This allows me to queue updates so that I only
have to paint the GUI every few seconds, rather than each time a new data
chunk arrives.

I'm pretty much at a loss to how to do this with reasonable performance. I
am willing to spend money for a 3rd party control, so long as it's royalty
free.
 
I would say first write the incoming data to a file. Using datagrid just
keepdisplaying the window into the file that the user is interested. If
fresh data is the priority keep the datagrid topped.
 
There's no good way to keep a datagrid "topped" that I know of. All the ways
that I've tried have resulted in re-rendering the entire datagrid, which
with a large number of records in there, is not a good solution. When I
benchmarked this using the DevPartner Profiler for .NET is was waaaaaaaay
slow.

Plus this approach has the drawback of needing quite a bit in the way of
additional resources - I've got to write the data to a file, read it back,
index into it properly, etc. I don't think this approach will meet the
performance requirements that I'm looking to meet.

Possabilities such as a pure memory mapped file are there, but they seem
like hacks.

The list view is a great answer, except for the fact that it crashes. A Text
Box would be fine, if it had an "AppendLine".

--
Chris

Rajesh.V said:
I would say first write the incoming data to a file. Using datagrid just
keepdisplaying the window into the file that the user is interested. If
fresh data is the priority keep the datagrid topped.

Chris Mullins said:
I'm building a GUI that needs to be able to view a large amount of text
arranged in rows. Large being anywhere from a few hundred lines through a
few hundred thousand. I need a way to "cap" the max number of rows, so that
old rows are discarded in favor of new rows if the limit is reached.

My use case is very similar to that of the SQL Profiler GUI - I'm going to
be receiving a large amount of data from a server that I want to display.
I'm fine defining a "window" on the data, or a max number of rows on the
control (say 200K) and throw out old data as the new data comes in.

I started out with a List View control, and appended list view items to the
beginning, and deleted items from the end. This worked fairly well, but
would have very strange behavior on occasion. It would, on a not infrequent
basis, hang the entire app-domain, including the VS.NET debugger. Very, very
frustrating.

I then moved on to a DataGrid bound to a typed dataset. I manage the
Click to expand...
rows
Click to expand...
 
Hi Chris,
From your description, I think ListView is a good answer too. However, I
can't figure out what strange behavior ListView had. Can you repro this
behavior? If yes, please send me your repro project so that I can do
something for you!
BTW: Can you try to use MS FlexGrid control for your huge amounts of text?
In Toolbox Right click General->Add/Remove Item -> COM component ->Add
Microsoft FlexGrid Control v6.0 into toolbox and use it.

Rhett Gong [MS]
Microsoft Online Partner Support

This posting is provided "AS IS" with no warranties, and confers no rights.
Please reply to newsgroups only. Thanks.
 
Hi Rhett,

I saw two different classes of problems:

The first problem was the generation of SEH exceptions from the control.
Although VS.NET would still repsond to user input it made running and
debugging the application impossible. I searched as best I could for more
information on the problem, and was unable to find anything. I do have
"Break on SEH Exceptions" turned off in the debugger. When This started
happening, the only option was to close down and restart VS.NET - stopping
and restarting the app would result in the same behavior almost immediatly.

The 2nd problem is that calling control.invoke on the list view (from a
background thread) would sometimes cause the entire application (including
the VS.NET debugger) to hang. Determining this as the cause of the lockup
took quite a bit of time! I don't want to use Control.BeginInvoke, as I'm
already running out of thread pool threads in my AppDomain.

I tried updating the list view every way I could think of - 1 row at a time,
bulk updates (using BeginUpdate/EndUpdate), etc.

The problem is difficult to replicate but generall happens when, as a user,
I interact with the form - minimizing it, resizing it, clickingn on a column
header in the list view to re-sort things, clicking on a line to update the
detail view at the bottom of the form. If I just let the form sit there, it
generally works ok.

I haven't tried the flex grid yet. On a seperate topic, why doesn't MS
release primary interop assemblies for these controls (especially IE!)
Sheesh.
 
Hi Chris,
Can you tell me how you updated your dataset to the listview? It seems
that there are some problems in thread interaction. Please make sure that
you don't call another thread to operate GUI or Dataset simultaneously. If
possible, send me your thread code which is used to update the listview.
Thanks!

Rhett Gong [MS]
Microsoft Online Partner Support

This posting is provided "AS IS" with no warranties, and confers no rights.
Please reply to newsgroups only. Thanks.
 
Hi Rhett,

Unfortunatly sending the code would be difficult, as it's awfully tied in
with quite a bit of other stuff. I'm going to try to find time in the next
few days to break it out into something more testable. I'll post that when I
get a chance.

I'm not doing updates from background threads. My basic algorithm is this:
On a background thread, I get a notifaction that there's new data to put
into the list view.
Still on the background thread, I decide if there's either a) enough data to
warrent the update, or b) a minimum time interval has elapsed since my last
update.
Still on the background thread, I grab a Monitor on a shared datastructure
(an arraylist) and stick all the new data that I'm going to want updated
into there.
On the background thread, I release the Monitor lock, and Control.Invoke
over to a new function.
On the GUI thread (via the Control.Invoke) I grab the lock on the array list
(via Monitor.Enter), set the ListView into BeginUpdate mode, iterate through
the array list creating ListViewItems and adding them into the ListView.
When this is done, I issue the EndUpdate to the Listview, clear out the
arraylist, and exit the lock on the Monitor.

At this point everything should work properly.
 
Hi Chris,
In my opinion, the symptom you have come across is a typical thread
synchronizing issue. From your description, I find that you use many
threads to achieve your goal. I couldn't get any information from the
algorithm you described. It's a good algorithm if you can control it
perfectly. However, due to the complicated user input logic, I think it is
hard to cover every side with so many threads in your application.
I have tried inserting huge amounts of text into a ListView control
within a single thread application and it works fine.
I suggest you to trace your code step by step to see where this problem
happens.


Rhett Gong [MS]
Microsoft Online Partner Support

This posting is provided "AS IS" with no warranties, and confers no rights.
Please reply to newsgroups only. Thanks.
 
Back
Top