Help choosing a fast computer

  • Thread starter Thread starter Michael 182
  • Start date Start date
M

Michael 182

Hi - any help appreciated..,

I do a lot of data crunching, imaging hard disks, doing sequential and grep
searches, and full text indexing the images. The last index process took 28
hours to index approximately 1,000,000 items in a 50 gig datastore using a
Pentium 4 2.08 gh processor with 1 gb RAM and standard USB 2.0 external hard
disk. I'm not looking for better software tools - given what I do, these are
the tools I need. But I am looking for better hardware performance.

I am considering getting a AMD Atlon 64 3000+ CPU on the MSI K8N Neo2
Platinum 939 NForce3 motherboard with 3Gb of memory (I've heard that 4Gb
seems to cause some problems - be interested if anyone else thinks that...)

I'd appreciate any feedback - will I see significant speed improvements? Is
there a better solution? I am not a hardware guy, so please excuse any
errors of omission or commission...

Thanks,

Michael
 
Hi - any help appreciated..,

I do a lot of data crunching, imaging hard disks, doing sequential and grep
searches, and full text indexing the images. The last index process took 28
hours to index approximately 1,000,000 items in a 50 gig datastore using a
Pentium 4 2.08 gh processor with 1 gb RAM and standard USB 2.0 external hard
disk. I'm not looking for better software tools - given what I do, these are
the tools I need. But I am looking for better hardware performance.
Click to expand...

Then don't use USB external drives.

I am considering getting a AMD Atlon 64 3000+ CPU on the MSI K8N Neo2
Platinum 939 NForce3 motherboard with 3Gb of memory (I've heard that 4Gb
seems to cause some problems - be interested if anyone else thinks that...)

I'd appreciate any feedback - will I see significant speed improvements? Is
there a better solution? I am not a hardware guy, so please excuse any
errors of omission or commission...
Click to expand...


Better to the extent that it's faster, yes. Probably not
anywhere near as much an improvement as switching from
external to internal drives, which are the largest
bottleneck with either. It's even possible the new system
won't do the job ANY faster.
 
kony said:
Then don't use USB external drives.
Click to expand...

I have to switch out hard drives continuously. My current inventory is about
200 drives. Externals are very simple to do this with. What is the speed
difference between internal and external drives? Is there another solution -
maybe hanging and power connections outside the computer? are there
"extension cords" for these?

Better to the extent that it's faster, yes. Probably not
anywhere near as much an improvement as switching from
external to internal drives, which are the largest
bottleneck with either. It's even possible the new system
won't do the job ANY faster.
Click to expand...

I am pegged at 100% CPU time (according to task manager) for a fair amount
of the process - I assume I'll get some improvement on that part, right?

Thanks,

Michael
 
Michael 182 said:
I have to switch out hard drives continuously. My current inventory is
about 200 drives. Externals are very simple to do this with. What is the
speed difference between internal and external drives? Is there another
solution - maybe hanging IDE and power connections outside the computer?
are there "extension cords" for these?
Click to expand...
 
i use removable drive kits
however , if you are going to be swapping out a lot of drives...
the 80 wire ide cable will not last long.

i have one (older) enclosure setup with a 40 wire ribbon...
which has been holding up well for some time now.
since the ribbon is very short (1") there does not seem to be a major
performance hit...
but how it compares to USB2...I don't know
 
I have to switch out hard drives continuously. My current inventory is about
200 drives. Externals are very simple to do this with. What is the speed
difference between internal and external drives?
Click to expand...

It can be over twice as fast. Nothing doing exactly this
task I can't give you an exact figure.
Is there another solution -
maybe hanging and power connections outside the computer? are there
"extension cords" for these?
Click to expand...

SATA would be the better alternative for that type of
changes. A power cord extension (of reasonable quality)
should be fine but parallel ATA cables of excessive length
(perhaps around 24", especially with the additional
connector inline) may be a problem.

I am pegged at 100% CPU time (according to task manager) for a fair amount
of the process - I assume I'll get some improvement on that part, right?
Click to expand...

That may simply be the USB effect.
Indexing files should not require much CPU, memory, or (much
of anything except the drive and drive-interface speed).

You might try benchmarking the USB ports to those particular
enclosures to see what the current rate is. For example,
since you have so many small files, try taking 2GB worth and
time how long it takes to read that many from the drive,
simply time them with a clock while copying to an internal
drive known to have sufficient free space and low to no
fragmentation to be sure the bottleneck is practically the
USB alone.
 
kony said:
That may simply be the USB effect.
Indexing files should not require much CPU, memory, or (much
of anything except the drive and drive-interface speed).

You might try benchmarking the USB ports to those particular
enclosures to see what the current rate is. For example,
since you have so many small files, try taking 2GB worth and
time how long it takes to read that many from the drive,
simply time them with a clock while copying to an internal
drive known to have sufficient free space and low to no
fragmentation to be sure the bottleneck is practically the
USB alone.
Click to expand...

I'll try that - as opposed to copying internally from and to the same drive?
Or would having a separate source and target drive internally make a
substantial difference?

Thanks for your help.

Michael
 
I'll try that - as opposed to copying internally from and to the same drive?
Or would having a separate source and target drive internally make a
substantial difference?

Thanks for your help.

Michael
Click to expand...
Keep in mind that indexing is very seek intensive, and any slowness in
the USB drive will only be aggravated by seeking.
Don't know about your software, but in heavy duty databases indexes
are keep on a separate drive to minimize head movement, and you may be
able to tweak that configuration into your work. There may be other
indexing performance tweaks your software allows, so dig in to the
docs.
Another option may be to do your indexing on a fast internal drive(s),
then transfer to the external when done. As kony said, benchmark
different methods.
I've seen badly designed or kludged indexing/sorting jobs that ran for
a 36 hours reduced to 30 minutes when properly configured.
Any index job running 28 hours, even 50gig with USB, probably
needs better or better-configured software.

--Vic
 
Victor Smith said:
Keep in mind that indexing is very seek intensive, and any slowness in
the USB drive will only be aggravated by seeking.
Don't know about your software, but in heavy duty databases indexes
are keep on a separate drive to minimize head movement, and you may be
able to tweak that configuration into your work. There may be other
indexing performance tweaks your software allows, so dig in to the
docs.
Another option may be to do your indexing on a fast internal drive(s),
then transfer to the external when done. As kony said, benchmark
different methods.
I've seen badly designed or kludged indexing/sorting jobs that ran for
a 36 hours reduced to 30 minutes when properly configured.
Any index job running 28 hours, even 50gig with USB, probably
needs better or better-configured software.

--Vic
Click to expand...

No doubt you are correct on this, but there is little software tweaking
available. I can decide what files to include in the indexing, such as
excluding exe files. Beyond that the indexing is blackbox, and, in fact, is
not written by the software I use, but licensed. I'm using FTK (Forensic
Toolkit). The indexing is done by dtSearch.

Part of the problem is I need to use a write blocker when I take an image,
as part of standard forensic methodology. Pretty much all the write blockers
are firewire devices. How about Firewire 800 - is it living up to it's name?

Thanks,

Michael
 
No doubt you are correct on this, but there is little software tweaking
available. I can decide what files to include in the indexing, such as
excluding exe files. Beyond that the indexing is blackbox, and, in fact, is
not written by the software I use, but licensed. I'm using FTK (Forensic
Toolkit). The indexing is done by dtSearch.
Click to expand...
Sorry, I know nothing about the software or processes you mention.
Can only suggest you find user groups that share information on this
on usenet or the web. Most computer application areas have a presence
there.
Part of the problem is I need to use a write blocker when I take an image,
as part of standard forensic methodology. Pretty much all the write blockers
are firewire devices. How about Firewire 800 - is it living up to it's name?
Click to expand...
Sorry again, don't know anything about firewire.
Good luck.

--Vic
 
Back
Top