O.T. copying 1TB from drive to drive in XP

  • Thread starter Thread starter cj
  • Start date Start date
C

cj

I ask a lot of VB questions here but this isn't related to VB. I'm not
sure where to get help with this question but perhaps someone here would
know and not mind giving me some ideas.

We've got a Linux samba server currently storing very close to a
terabyte of 70kb to 956kb files. They are stored in a directory
structure by day. For instance files from August 1st 2007 are in a
directory s:\2007\08\01.

I need to copy all these file off this system and onto a 2TB windows
drive. The Linux server needs to be left in production during this
time. I'd like this to occur as fast as possible. I've tried
everything and nothing moves anything very fast. I've tried TeraCopy,
RoboCopy, SyncToy, and each was a real disappointment. I'm currently
running 6 command windows each running XCopy on a different set of
directories. For instance one command window runs this batch file:

xcopy r:\2007\01\*.* s:\2007\01 /D /E
xcopy r:\2007\02\*.* s:\2007\02 /D /E
xcopy r:\2007\03\*.* s:\2007\03 /D /E

another picks up with months 4, 5 and 6 etc.

It's important to remember that this process could get interrupted and
it would be terrible to have to overwrite files that were already copied
in order to make sure all of them got copied so I want something that'll
copy only files that don't already exist on the target drive -- hence
the /D on the xcopy.

According to Task Manager my pc is not taxed at all. Network trafficd
is at 2.5 % MAX -- ususally less than 1%. The CPU Usage is 0 to 2%.
This machine has 2GB ram. It's on a 100mb ethernet network with limited
traffic.

Any ideas would be appreciated.
 
Hi Cj,

If the size of files are really large and you're also worried about the
potential interruption, do you think it possible to setup a FTP on the
server so that your windows can connect as a FTP client to download those
files?

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead


Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
--------------------
 
@TK2MSFTNGP03.phx.gbl:

I need to copy all these file off this system and onto a 2TB windows
drive. The Linux server needs to be left in production during this
time. I'd like this to occur as fast as possible. I've tried
everything and nothing moves anything very fast. I've tried TeraCopy,
RoboCopy, SyncToy, and each was a real disappointment. I'm currently
running 6 command windows each running XCopy on a different set of
directories. For instance one command window runs this batch file:
Click to expand...


The disk could be extremely fragmented. Try defragging first?

You could ZIP the files first, then transfer the archive. That should
increase your transfer rates since the disk has to do less seeking when
reading one large file.

Also where are you running your copy utility? Directly from the windows
server or from a 3rd machine (i.e. your desktop)?

You could also look into using rsync or a similar linux friendly sync
protocol to do the file transfer/
 
Given the source is Linux and must remain in production I really don't
think I'll be able to defrag it. The source is almost full so there is
no space to zip the files to before copying them. We are using a 2TB
RAID NAS device as the destination for these files. Given we didn't
have a DHCP Server at this site the NAS is actually attached to a pc
running the command windows via USB. While USB is showing 425mbps
connection speed vs the ethernet link to the linux source at 100mbps so
I wouldn't think that would be a problem.
 
Given the source is Linux and must remain in production I really don't
think I'll be able to defrag it.  The source is almost full so there is
no space to zip the files to before copying them.  We are using a 2TB
RAID NAS device as the destination for these files.  Given we didn't
have a DHCP Server at this site the NAS is actually attached to a pc
running the command windows via USB.  While USB is showing 425mbps
connection speed vs the ethernet link to the linux source at 100mbps so
I wouldn't think that would be a problem.









- Show quoted text -
Click to expand...

If you experience slowness with transferring large files over the LAN,
have you tried using gigabit (1000mbps) switch with at least Cat6
cable?
 
cj said:
Given the source is Linux and must remain in production I really don't
think I'll be able to defrag it. The source is almost full so there is
no space to zip the files to before copying them. We are using a 2TB
RAID NAS device as the destination for these files. Given we didn't
have a DHCP Server at this site the NAS is actually attached to a pc
running the command windows via USB. While USB is showing 425mbps
connection speed vs the ethernet link to the linux source at 100mbps so
I wouldn't think that would be a problem.
Click to expand...

Have you tried copying other large files from your PC to the NAS drive?

What sort of transfer rates do you get?

Most low-to-medium NASes can't sustain more than 5 - 10MB per second.
 
kimiraikkonen said:
If you experience slowness with transferring large files over the LAN,
have you tried using gigabit (1000mbps) switch with at least Cat6
cable?
Click to expand...

Cat5e cable works for gigabit Ethernet.

I agree about the speed, seeing as 100Mbps is only about 10MB/s. 1TB at that
speed will take at least 28 hours, and the Linux box is probably busy doing
other things at the same time.

Andrew
 
I've done some testing today. I changed the NAS over to an ethernet
connection vs the usb connection. It does seem faster however it
doesn't come close to copying to c: I copied 15,129 files (3.66GB) to
the NAS in 1 hour and 45 minutes. At this speed that would mean it
would take approximately 9 days to copy a years worth of files and I
have more than that to copy. I then tried copying the same files to C:
and it took only 15 minutes. So I've got some options to look at. I'm
thinking use a desktop with a 1 or 2 Terabytes of internal HD space or
perhaps I can get a external SATA drive attached to this laptop. I
don't know. Thanks to everyone for the help on this off topic
discussion at least I know where to go from here.
 
Thanks for posting that. I was thinking of buying a network drive for my
iTunes directory (80GB), but will stick to USB instead .

RobinS.
 
I still haven't gotten all the speed issues figured out. I'm not able
to spend the time I need to do so. I have done testing with several USB
drives on my desktop pc and found the transfer speeds to the USB drives
to be exactly the same as to C: and the bottle neck seems to be the lan
speed--100MB or 1GB. Another fellow tested the same USB drives with a
laptop pc and found the copy to c: was faster than I got using the
desktop but copying to any of the USB drives plugged into it were MUCH
slower. Both PCs are USB 2.0.

laptop to c: 377MB/Min with 100MB lan
laptop to USB 188MB/Min with 100MB lan
desktop to c: 282MB/Min with 100MB lan
desktop to USB 282MB/Min with 100MB lan
desktop to c: 678MB/Min with 1GB lan
desktop to USB 678MB/Min with 1GB lan

The tests were were using 1,593 files totaling 565MB The files are a
tiny subset of what we want to move. We have to work with a 100MB lan
where the actual work needs to be done. I'm thinking we need to take a
desktop pc there since the laptop seems to have issues with external
devices and I don't know of a laptop with a 1TB c:. Another mystery is
why the laptop will not transfer from the network to the 2TB NAS drive
faster. I'd like to test if a desktop would write to it faster. I know
neither is actually attached to a NAS that's the mystery.

Oh well we'll see where folks here want to go with this.
 
I still haven't gotten all the speed issues figured out.  I'm not able
to spend the time I need to do so.  I have done testing with several USB
drives on my desktop pc and found the transfer speeds to the USB drives
to be exactly the same as to C: and the bottle neck seems to be the lan
speed--100MB or 1GB.  Another fellow tested the same USB drives with a
laptop pc and found the copy to c: was faster than I got using the
desktop but copying to any of the USB drives plugged into it were MUCH
slower.  Both PCs are USB 2.0.

laptop to c: 377MB/Min with 100MB lan
laptop to USB 188MB/Min with 100MB lan
desktop to c: 282MB/Min with 100MB lan
desktop to USB 282MB/Min with 100MB lan
desktop to c: 678MB/Min with 1GB lan
desktop to USB 678MB/Min with 1GB lan

The tests were were using 1,593 files totaling 565MB  The files are a
tiny subset of what we want to move.  We have to work with a 100MB lan
where the actual work needs to be done.  I'm thinking we need to take a
desktop pc there since the laptop seems to have issues with external
devices and I don't know of a laptop with a 1TB c:.  Another mystery is
  why the laptop will not transfer from the network to the 2TB NAS drive
faster.  I'd like to test if a desktop would write to it faster.  I know
neither is actually attached to a NAS that's the mystery.

Oh well we'll see where folks here want to go with this.




- Show quoted text -
Click to expand...

cj,
Always remember that, though you may have configured every networking-
related stuff correct, harddrives are so responsible for slowdown
during data transfer. Though you have a SATA 3G or hi-speed SCSI
drive, if the drive's internal transfer speed is slow than networking
broadband, NAS or cabling have nothing to do in this case.

Also, defragment requirement and filesystem configuration, file
locations are big factors.

A basic example, when you transfer a single and big file to and from
your flash drive across your computer, the speed is at flash drive's
max and may work flawlessly, but if you have a lot of folder(s), sub
folder structures with small-sized files, then the transfer will
extemely be slower than normal speed due to microcontroller on the
drive.

Thanks
 
Last follow up.

The end of last week I was able to do more testing in a controlled
environment myself using a USB drive on the laptop and desktop and have
thankfully concluded both systems can copy to the USB drive and their
respective C: as fast as even a 1GB lan can get the data to them. I
think the fellow who ran the laptop speed testing must not have timed it
too good when we were testing the first time. As for the NAS I can't
even get connected to it any more, and I had had enough of my own
testing on it to say it just isn't fast enough. I think Spam Catcher is
right on these NASes. This one was supposed to be a good one but it sucks.

That's all folks!
 
Back
Top