Text editor to open massive files needed

  • Thread starter Thread starter skyman302
  • Start date Start date
S

skyman302

I need a program to open an 8gb text file and edit it. Is there
anything out there that can handle it?
 
I need a program to open an 8gb text file and edit it. Is there
anything out there that can handle it?
Click to expand...

Such a large file. Obviously, you don't want to view 8gb of actual
text. And obviously, you don't intend to manually make the changes to
the file.

Assuming you plan something simple, like search and replace, try
breaking up the file into manageable pieces. Perform your editing on
each and reassemble.

Then, break the file up at a different point for each piece, and
repeat.

Beats hiring 8g monkeys and buying 8g typewriters.
 
jo said:
LOL.
Click to expand...

I was thinking more WTF? 8gb has got to be either a typo, or he's got the
entire contents of his local library backed up on his hard drive.

DG.

--
Excerpt from an as yet unaired episode of Who Wants To Be A Millionaire? :-

Chris Tarrant "So Dan, you want to phone a friend?"
Dan Glybitz "Yes please Chris, I'll call Skyman302."
CT "Does he know a lot then, this Skyman?"
DG "He's got an 8gb text file, Chris. He knows EVERYTHING!"
 
I need a program to open an 8gb text file and edit it. Is there
anything out there that can handle it?
Click to expand...

Way back at the begining of time, when Cray computers were still in
use, I received several large text data files (~ 1GB). When I
complained to the computer time vendor ( back in the dark ages, we
bought time on a computer) that the supplied text editor would not
open the file he said he would not even attempt to get someone to
adress the problem, that it was just silly, and that I should go back
and learn how to handle data
 
Actually it's a set of geographic points and elevations. It is in fact
8 "GB". What I need to do is delete a few collums of data. Any
suggestions?
 
I need a program to open an 8gb text file and edit it. Is there
anything out there that can handle it?
Click to expand...

You could try The Gun (Grown Up Notepad), which is written in assembly:

http://www.movsd.com/thegun.htm

Version 3.0f has had minor internal changes and has taken advantage of
the technology of a new linker written by Pelle Orinius to bring it down
to its preferred size of 6k (6144 bytes). TheGun from its origin was
designed to be useful within the boundaries of minimum size, high speed
performance and no useless features. Its original purpose was to dump
results from a compiler or an assembler build into for display which
demanded near instant load.

TheGun does not use any additional runtime DLLs and is coded in Microsoft
Assembler (MASM) using the Windows API functions.. It does not use or
write to the registry at all and will run on Windows versions from Win95b
upwards. In common with the last version, it uses a very high speed
dispatcher internally for system message processing, it now tests for
read only files and handles the XP style of file dialog correctly. It
uses an extended version of the system "MessageBox" to display various
forms of information.

For 6k in file size, it is fully drag and drop enabled, it supports
wordwrap and it has a seperate accessory to change some of the startup
settings. The accessory is SETGUN.EXE and it patches the settings
directly into the disk file so there is no need for an INI file or a
registry setting. It has a system defined text search capacity where you
can search for text with the options of case sensitive, insensitive or
search for whole words only. There is "right click" support for the menu
options built into TheGun. You can quickly exit TheGun by pressing the
ESC key.

TheGun does not have an effective file size limit and the maximum size
that can be loaded into it is determined by available memory and loading
speed of the file. It can typically load files in excess of 10 megabytes
with no problems. It has been speed optimised for both file load and
save.

Printing is a low priority issue with a text editor of this type and the
capacity it has is farmed out to the system file Wordpad.Exe. The margin
settings are those that are set in Wordpad.

TheGun still maintains its size and speed advantage over the system
Notepad.Exe so if these are issues that, Number One Son will do the job
for you "just fine".
Click to expand...
 
Actually it's a set of geographic points and elevations. It is in fact
8 "GB". What I need to do is delete a few collums of data. Any
suggestions?
Click to expand...

What format? Is it *.e00, *.dbf? Please describe your problem as detailed
as possible. Even if you want to delete last rows or some out of the
middle of the file will matter.

Generally, you don't open such large files the 'normal way' even if they
are pure text. You need a program which loads part of the file from disk
and does its works section-wise. You'll often find hex editors better
prepared for this kind of job than text editors. Unfortunately, most
free hex editors stop working on the 4 GByte boundary, at the latest.

The only free one I remember opening larger files is JHEditor:

http://home.arcor.de/mschierlm/sm-soft/jheditor.htm

Unfortunately, it is one of these hex editors which can't be switched
to a text-only view. And it requires a Java runtime...

BeAr
 
Actually it's a set of geographic points and elevations. It is in fact
8 "GB". What I need to do is delete a few collums of data. Any
suggestions?
Click to expand...

If you know someone who can write a small program. Just read in a
"line" of data and write the line, minus the data you want to delete,
out to a new file. Do that until you reach the end of the input file.

Memory-wise you wouldn't have to deal with more than is required to
process the single "line".

In C it would be about a 50-line program. In Fortran probably half
that much. Other text processing applicatons could also do the job
easily.

How long are the "lines"?
 
Actually it's a set of geographic points and elevations. It is in fact
8 "GB". What I need to do is delete a few collums of data. Any
suggestions?
Click to expand...

Ultraedit will handle files > 8GB. Although [OT] payware... they offer a free
download that may be able to do the task at hand (one time) if none of the
freeware ones work.
http://www.ultraedit.com/

ozzy
 
"Rodders" <[email protected]> wrote:
Click to expand...
I recommend ultraedit, it also has a coloumn select feature.
Click to expand...

That looks to be shareware. What is the largest file you've ever
opened and edited with it?

An 8 gig file is going to be a nightmare. The program will have to
load the file into memory and then into swap space. All caches will
have to be written and updated as any action is taken and the whole
thing will have to be written back from swap to file on save or exit.

On one hand, it will be interesting to see if editing and saving an 8
gig file is possible.

On the other I don't think it is, for all practical purposes.

If the OP is still reading, what is this file? I think I can write a
program that will read the original, write temp 100 meg files to be
edited, and then put it back together.
 
MLC said:
(e-mail address removed)_, giovedì 06/gen/2005:




"EditPad does not impose a limit on the size of files you can open and edit
with it."
http://www.editpadpro.com/editpadlite.html
Click to expand...
But unless it has been written using 64-bit file handling, it won't do
more than 2 or 4 gigabytes.

A stream editor, such as "sed", or a text munger such as "perl" or "awk"
is usually what is used for such tasks.

Cheers,
Gary B-)
 
REM wrote:
[SNIP]
If the OP is still reading, what is this file? I think I can write a
program that will read the original, write temp 100 meg files to be
edited, and then put it back together.
Click to expand...
If it's nice and regular, an "awk" script would be even smaller.

If the OP would post a few lines, it would be easy to have a go.

Cheers,
Gary B-)
 
REM said:
That looks to be shareware. What is the largest file you've ever
opened and edited with it?

An 8 gig file is going to be a nightmare. The program will have to
load the file into memory and then into swap space. All caches will
have to be written and updated as any action is taken and the whole
thing will have to be written back from swap to file on save or exit.

On one hand, it will be interesting to see if editing and saving an 8
gig file is possible.

On the other I don't think it is, for all practical purposes.
Click to expand...

The Vedit - Buyware - site is well worth a visit to get info about some
of this stuff.
Their basic editor is 80 dollars and bottles out at 2GB, though it can
split and recombine larger files.
For a mere 150 dollars more you get an editor that they claim can
effortlessly handle files of over 100GB.

http://www.vedit.com/huge_files.htm
 
Rodders said:
That looks to be shareware.
That's because it is.

What is the largest file you've ever opened and edited with it?
Click to expand...
1 GB, It was slow but accurate. Afterwards I had the Edited File plus
the orginal (.bak).
An 8 gig file is going to be a nightmare. The program will have to
load the file into memory and then into swap space. All caches will
have to be written and updated as any action is taken and the whole
thing will have to be written back from swap to file on save or exit.
Said like a Real programmer.

On one hand, it will be interesting to see if editing and saving an 8
gig file is possible.
Click to expand...
Are you accepting the challenge?
On the other I don't think it is, for all practical purposes.
Click to expand...
It's only natural to have doubts.
If the OP is still reading, what is this file? I think I can write a
program that will read the original, write temp 100 meg files to be
edited, and then put it back together.
Click to expand...
The only logical solution for a file this size would be a conversion
program which would read the input file and output a new file in the
format desired. The OP sounded like the file was some sort of csv?

Mel

PS: UltraEdit is well worth the price.
 
Back
Top