Send a file over a TCP network connection

  • Thread starter Thread starter David Glover
  • Start date Start date
D

David Glover

Hi,

I posted a question a while back relating to sending data over a
network, and was told to go and look into how networks send data.

I know that TCP can not guarentee the order in which data is delivered
to a device. In light of this, I have a problem which I can not find a
way out of:

I need to send a file of around 4-5Mb over a network. To prevent
communicaiton thread being locked, I was attempting to segment the
data into 8k segments of data to be sent. In the header of each
segment, I was attaching a 4byte integer representing the segment
number, which will allow the reciever to place the data in the
relevant position in the output file. Occasionally the client decodes
the segment identifier into a bizzare value (last time I tried this, I
was expecting to see segment number 234, and recieved -432352 or
similar).

However, this is not working,and the data being written out to a file
at the reciever is not the same as the input data at the server. Why
would this be?

Please can anyone shed any light on this matter, or show me an example
of how I should send the data/ensure that the data at the client end
is pieced together correctly?

Many thanks for any help - I am very confused by this matter.

David
 
Well,

No, TCP *does* guarantee delivery order. UDP does not. If you establish a
socket connection between two points, then the order in which data is sent
from one end *will be* the order in which it is received at the other end.

In your particular case, just send a header saying "Here comes a file that's
xxxx bytes long", then send the entire content of the file (being sure to
send the right number of bytes). On the other end, receive the header,
then, using the indicated number of bytes, start receiving and writing to
the file. Breaking up the file into pieces and sending them in separate
sends *is* a good idea, though. You don't have to put a header with every
piece, though.

Paul T.
 
Hi,

Paul's answer is correct. TCP "assures" (assurance is just high
probability, but it should be good enough) that packets will be delivered,
and that delivery will be in the order of transmission. There may be
reasons to packetize data, and to provide extra error-detection and recovery
mechanisms (such as might be desired if using a wireless connection, for
example). However, in general, it is sufficient to do as Paul suggests --
though I sometimes include a CRC calculation in addition to the file size.

Dick

--
Richard Grier (Microsoft Visual Basic MVP)

See www.hardandsoftware.net for contact information.

Author of Visual Basic Programmer's Guide to Serial Communications, 3rd
Edition ISBN 1-890422-27-4 (391 pages) published February 2002.
 
Basically the file needs to be segmented to allow a situation where by
the client (which is on a wireless link) uploads a file, and at the same
time, the user presses a button such as "get lastest server
information", which needs to send a message to the server, and then
recieve a message back. Other wise the user will have to wait for the
compelete transmission of the file in question before they can initiate
another operation.

Considering the applicaiton being developed is a distributed network of
files (using MP3 files in this case), the server get a request from a
client for a file off another client. The 2nd client then starts to
upload the file. During this up load the user of the second client
wishes to see what tracks are availible on the system. The user would be
blocked by an operation they did not initiate, and this would be poor
from a usability perspective.

Many thnaks for your input however, I am reluctant to give up this idea
of segmenting the file considering the operating envrionment. This is
being a harder nut to crack that 1st thought.
 
Many thanks for the reply. I fear my initial response to your post may
have been lost. In case it has - here is another response:

I have gone back to basics and written a basic full .NET applicaiton
which will segment the file and send it to a client. The output file and
the input files are different when compared in a hex editor (I am using
HexCmp).

The test app below does not implement segment ID fields, however the
final app will. Can anyone see/suggest what may be going wrong here:

***SENDER APPLICAITON***
...
private Socket clientSocket
...

private void button1_Click(object sender, System.EventArgs e)
{
Socket serverSocket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream,
ProtocolType.Tcp) ;

IPEndPoint localEndPoint = new IPEndPoint(IPAddress.Any, 676) ;
serverSocket.Bind(localEndPoint) ;
serverSocket.Listen(10) ;
clientSocket = serverSocket.Accept() ;
}

private void button2_Click(object sender, System.EventArgs e)
{
bool f = clientSocket.Blocking ;

//NetworkStream ns = new NetworkStream(clientSocket,
FileAccess.Write, true) ;


FileStream fs = new FileStream(@"e:\dump\out.mp3",FileMode.Open,
FileAccess.Read, FileShare.Read) ;
BinaryReader b = new BinaryReader(fs, System.Text.Encoding.ASCII) ;
long dataLength = fs.Length ;

//calculate the number of segments that are required to send the file
int fileSizei = (int)dataLength ;
double fileSized = fileSizei ;
double segmentsd = fileSized / 8192 ;

int segments = Convert.ToInt32(segmentsd) ;
MessageBox.Show("File size : " + fileSizei.ToString()) ;
MessageBox.Show("Segments : " + segments.ToString()) ;


int totalBytesWritten = 0;
for(int i = 0 ; i < segments ; i++)
{
//read the data from the buffer
if(i == (segments-1))
{
//last segment
//calculate the number of bytes left
int bytesLeft = fileSizei - totalBytesWritten ;

byte[] send = b.ReadBytes(bytesLeft) ;

fs.Read(send, 0, bytesLeft) ;

int bytesSent = clientSocket.Send(send, 0, bytesLeft,
SocketFlags.None) ;
Console.WriteLine(bytesSent) ;
if(bytesSent != bytesLeft)
{
MessageBox.Show("ERROR WRITING BYTES") ;
}
}
else
{
//byte[] send = new byte[8192] ;
//fs.Read(send, 0,8192) ;
byte[] send = b.ReadBytes(8192) ;
int sentBytes = clientSocket.Send(send,0,8192,SocketFlags.None) ;
Console.WriteLine(sentBytes) ;
if(sentBytes != 8192)
{
MessageBox.Show("ERROR WRITING 8192 bytes") ;
}
totalBytesWritten += 8192 ;
}
//wait for the "OK"
byte[]ok = new byte[16] ;
clientSocket.Receive(ok, 0, 16, SocketFlags.None) ;


}
//fs.Close() ;
b.Close() ;
MessageBox.Show("SEND COMPLETE") ;
}

***RECIEVER***
...

private Socket sock ;

...
private void button1_Click(object sender, System.EventArgs e)
{
sock = new Socket(AddressFamily.InterNetwork, SocketType.Stream,
ProtocolType.Tcp) ;

IPAddress sip = IPAddress.Parse("192.168.10.2") ;
IPEndPoint remote = new IPEndPoint(sip,676) ;
sock.Connect(remote) ;

}

private void button2_Click(object sender, System.EventArgs e)
{
//make the user enter hte file size and number of segments
//these aer messagebox'd to the screen in the sender
//adn this is only a test harness when all said and done :)
int fileSize = Int32.Parse(textBox1.Text) ;
int segments = Int32.Parse(textBox2.Text) ;
bool block = sock.Blocking ;
FileStream f = new FileStream(@"c:\recieved.mp3",
FileMode.OpenOrCreate, FileAccess.ReadWrite) ;
BinaryWriter b = new BinaryWriter(f, System.Text.Encoding.ASCII) ;

int totalrecieved = 0;
for(int i = 0 ; i < segments ; i++)
{
if(i == (segments -1))
{
//last segment
int remaining = fileSize - totalrecieved ;
byte[] read = new byte[remaining] ;
sock.Receive(read, 0, remaining, SocketFlags.None) ;
b.Write(read) ;
}
else
{
byte[] read = new byte[8192] ;
sock.Receive(read, 0, 8192, SocketFlags.None) ;

b.Write(read) ;
totalrecieved += 8192;
}

string OK = "OK" ;
byte[] okb = System.Text.Encoding.ASCII.GetBytes(OK) ;
Console.WriteLine(okb.Length) ;
sock.Send(okb, 0, okb.Length, SocketFlags.None) ;
}

b.Close() ;
MessageBox.Show("DONE") ;
//f.Close() ;

}


I have put the 2 files used at:

http://www.lancs.ac.uk/ug/gloverd/out.mp3 - the file used by the SENDER
http://www.lancs.ac.uk/ug/gloverd/recieved.mp3 - the output file from
the reciever. Using a hex comparer, it can be seen that the errors start
on byte 19519.


Many thanks for any help anyone can provide, as I have spent quite a
while getting nowhere on this problem.

Regards and thanks,

David
 
At a quick guess, the encoding might be responsible. It might do things
like convert CR/LF pairs to CR or something along those lines. Maybe the
results of the comparison would tell you that. There might also be problems
with null characters, depending on what the content of the file looks like.

Paul T.

David Glover said:
Many thanks for the reply. I fear my initial response to your post may
have been lost. In case it has - here is another response:

I have gone back to basics and written a basic full .NET applicaiton
which will segment the file and send it to a client. The output file and
the input files are different when compared in a hex editor (I am using
HexCmp).

The test app below does not implement segment ID fields, however the
final app will. Can anyone see/suggest what may be going wrong here:

***SENDER APPLICAITON***
..
private Socket clientSocket
..

private void button1_Click(object sender, System.EventArgs e)
{
Socket serverSocket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream,
ProtocolType.Tcp) ;

IPEndPoint localEndPoint = new IPEndPoint(IPAddress.Any, 676) ;
serverSocket.Bind(localEndPoint) ;
serverSocket.Listen(10) ;
clientSocket = serverSocket.Accept() ;
}

private void button2_Click(object sender, System.EventArgs e)
{
bool f = clientSocket.Blocking ;

//NetworkStream ns = new NetworkStream(clientSocket,
FileAccess.Write, true) ;


FileStream fs = new FileStream(@"e:\dump\out.mp3",FileMode.Open,
FileAccess.Read, FileShare.Read) ;
BinaryReader b = new BinaryReader(fs, System.Text.Encoding.ASCII) ;
long dataLength = fs.Length ;

//calculate the number of segments that are required to send the file
int fileSizei = (int)dataLength ;
double fileSized = fileSizei ;
double segmentsd = fileSized / 8192 ;

int segments = Convert.ToInt32(segmentsd) ;
MessageBox.Show("File size : " + fileSizei.ToString()) ;
MessageBox.Show("Segments : " + segments.ToString()) ;


int totalBytesWritten = 0;
for(int i = 0 ; i < segments ; i++)
{
//read the data from the buffer
if(i == (segments-1))
{
//last segment
//calculate the number of bytes left
int bytesLeft = fileSizei - totalBytesWritten ;

byte[] send = b.ReadBytes(bytesLeft) ;

fs.Read(send, 0, bytesLeft) ;

int bytesSent = clientSocket.Send(send, 0, bytesLeft,
SocketFlags.None) ;
Console.WriteLine(bytesSent) ;
if(bytesSent != bytesLeft)
{
MessageBox.Show("ERROR WRITING BYTES") ;
}
}
else
{
//byte[] send = new byte[8192] ;
//fs.Read(send, 0,8192) ;
byte[] send = b.ReadBytes(8192) ;
int sentBytes = clientSocket.Send(send,0,8192,SocketFlags.None) ;
Console.WriteLine(sentBytes) ;
if(sentBytes != 8192)
{
MessageBox.Show("ERROR WRITING 8192 bytes") ;
}
totalBytesWritten += 8192 ;
}
//wait for the "OK"
byte[]ok = new byte[16] ;
clientSocket.Receive(ok, 0, 16, SocketFlags.None) ;


}
//fs.Close() ;
b.Close() ;
MessageBox.Show("SEND COMPLETE") ;
}

***RECIEVER***
..

private Socket sock ;

..
private void button1_Click(object sender, System.EventArgs e)
{
sock = new Socket(AddressFamily.InterNetwork, SocketType.Stream,
ProtocolType.Tcp) ;

IPAddress sip = IPAddress.Parse("192.168.10.2") ;
IPEndPoint remote = new IPEndPoint(sip,676) ;
sock.Connect(remote) ;

}

private void button2_Click(object sender, System.EventArgs e)
{
//make the user enter hte file size and number of segments
//these aer messagebox'd to the screen in the sender
//adn this is only a test harness when all said and done :)
int fileSize = Int32.Parse(textBox1.Text) ;
int segments = Int32.Parse(textBox2.Text) ;
bool block = sock.Blocking ;
FileStream f = new FileStream(@"c:\recieved.mp3",
FileMode.OpenOrCreate, FileAccess.ReadWrite) ;
BinaryWriter b = new BinaryWriter(f, System.Text.Encoding.ASCII) ;

int totalrecieved = 0;
for(int i = 0 ; i < segments ; i++)
{
if(i == (segments -1))
{
//last segment
int remaining = fileSize - totalrecieved ;
byte[] read = new byte[remaining] ;
sock.Receive(read, 0, remaining, SocketFlags.None) ;
b.Write(read) ;
}
else
{
byte[] read = new byte[8192] ;
sock.Receive(read, 0, 8192, SocketFlags.None) ;

b.Write(read) ;
totalrecieved += 8192;
}

string OK = "OK" ;
byte[] okb = System.Text.Encoding.ASCII.GetBytes(OK) ;
Console.WriteLine(okb.Length) ;
sock.Send(okb, 0, okb.Length, SocketFlags.None) ;
}

b.Close() ;
MessageBox.Show("DONE") ;
//f.Close() ;

}


I have put the 2 files used at:

http://www.lancs.ac.uk/ug/gloverd/out.mp3 - the file used by the SENDER
http://www.lancs.ac.uk/ug/gloverd/recieved.mp3 - the output file from
the reciever. Using a hex comparer, it can be seen that the errors start
on byte 19519.


Many thanks for any help anyone can provide, as I have spent quite a
while getting nowhere on this problem.

Regards and thanks,

David
Click to expand...
 
If you're going to handle multiple clients, maybe a multi-threaded approach
would be the best way to go. If a single client wants to grab status
information in the middle, you still have to decide whether to send that
request on the established connection or start a new one just for the status
information, but at least separate clients won't bump into each other.

Paul T.
 
I see no reason it must be packetized. Just use a worker thread to transfer
the data. You're not limited to a single socket connection (I think it's
128?).
 
I think you may be right!! Looking at the file, everytime the hex values
0D 0A appears, I only get one of them out at the other end! and 0D and
0A refer to carrige return and line feed in ASCII.

I am however still having the problem with the above values when I use
FileStream.write, and again if I use a BinaryWriter with Encoding.UTF-8
set. What could be causing this?? Is there any way to combat this -
other than to process each individual byte at a time in the buffer?

Thanks for your help in tracking down this problem thus far :) Hugley
appriciated.

David
 
I haven't done any file transfer with .NET CF sockets, but what you really
want is *no* encoding or binary encoding. I'd look around for a way to make
*that* happen.

Paul T.
 
Just a follow up post to thank you all for your help in this matter. I
have changed the way the code work to remove the encoding problem. I
also discovered why in some cases I was 'loosing' data. I was forgetting
to check the number of bytes *actually* read by the socket.recieve
method. To combat this I have implemented a loop where by if the ammout
of data read does not equal the fixed size of my data units, read the
availible data from the socket until such time as I have read all the
reqired data.

Many thanks to all of you for your help. Especially the suggestion
regarding the encoding issue - I would not have thought of that one for
quite a while :)

Thanks again,

Regards,

David - I owe you guys a beer or two :)
 
Back
Top