Web Service To No Where...

  • Thread starter Thread starter jabailo
  • Start date Start date
J

jabailo

I wrote a program that loops through a file of records.

It parses each line in the file and sends them to a web service that
inserts them into an AS400DB2 database using Asynch calls.

This is the wierd part. Say their are 500 records in the file.

If I run the process once, maybe 250 will appear.

If I run it a second time, maybe 400 or all the records will appear.

Usually the third time I run it, all 500 appear.

From the consuming program, that reads the records, there is never any
error message thrown. There are always 500 calls to the web service.

I can see from my web server logs that the service is just not called
when the records are not written, so I don't think its on the database side.

So my question is: If they don't error out, where do all these web
services calls go?? I ASP.NET webservices just that flawed and
inconsistent that it cannot be depended on for this type of operation?

Would a more robust web server such as Apache 2.0 help?
 
It could be that error messages are generated, but not received. This is the
case with one-way Web services where the Web service is defined to NOT return
any value. In that case, the proxy sends the SOAP message and closes the
connection right away without waiting to see if SOAP Faults are returned. Try
to force your Web service to return some value (or change the proxy generated
file) and try again. You might start receiving a lot of errors.

Waleed
 
I would buy that explaination, except the web service server logs ( iis
) don't show those missing calls.

So, where are those calls?
 
PS -- I also have tracing and logging on the web service -- those calls
are never recorded there.
 
There is a client-side limit to how many simultaneous calls you can make
from one AppDomain to one web service (see the
ServicePointManager.DefaultCon­nectionLimit property). Make sure that the
calls are not dropped in the client side. You should see an exception in the
callback of the async call if the call is not completed.

Also, you could try debugging the problem by using your own threads to issue
the calls instead of using the built-in BeginXXX methods. This way you could
log every outgoing call and determine if and where the calls are dropped.

Regards,
Sami
 
Sami said:
There is a client-side limit to how many simultaneous calls you can make
from one AppDomain to one web service (see the
ServicePointManager.DefaultConnectionLimit property). Make sure that the
calls are not dropped in the client side. You should see an exception in
the callback of the async call if the call is not completed.
Click to expand...

I wrap the Begin_ method in a try catch block. Is that enough? Or are you
saying there is a property of the IAsyncResult that I can access that traps
errors?
Also, you could try debugging the problem by using your own threads to
issue the calls instead of using the built-in BeginXXX methods. This way
you could log every outgoing call and determine if and where the calls are
dropped.
Click to expand...

Wow! Ok, so I could launch the calls using the synchronized method, but,
say put them all into the ThreadPool?
 
John Bailo said:
I wrap the Begin_ method in a try catch block. Is that enough? Or are you
saying there is a property of the IAsyncResult that I can access that
traps
errors?
Click to expand...

It is not enough if you just wrap the Begin_ call in a try/catch block. You
also need to call the End_ method in the callback inside a try/catch block -
that's where you should be getting the exceptions, if any.

Regards,
Sami
 
Thanks, I do call the End_method but not in a try/catch!

Someone said that if the return type of the method is /void/ that there
would be no error there.

Is that true?
 
Sami said:
There is a client-side limit to how many simultaneous calls you can make
Click to expand...


Actually, it looks like the default is unlimited:

http://msdn.microsoft.com/library/d...stemNetServicePointManagerPropertiesTopic.asp


"Property Value

The maximum number of ServicePoint instances to maintain. The default
value is 0, which means there is no limit to the number of ServicePoint
instances."


I did take your advice below on putting the try/catch around the End__
method, and will test with that.
 
Ok, we're getting closer.

Using Sami's advice, I nested the End_ method in a try/catch block.

I immediately saw an error, but I'm not sure what to do about it.

So, I called the web service 380 times, once for each record in a file,
in quick succession. For 29 of those calls, the method threw this error:

"The request failed with HTTP status 403: Access Forbidden."

Can anyone suggest why the web service would throw this error 29 times
out of 380 when called asynchronously?
 
This might be an answer:

http://www.dotnet247.com/247reference/msgs/43/219122.aspx

"I eventually found a couple KB article explaining this and the solution
(262635 - Error Message: HTTP 403.9 - Access Forbidden: Too Many Users Are
Connected, 308186 - HOW TO: Optimize Web Server Performance in Windows
2000). The keep-alives can be disabled through IIS admin interface. This
resolves my problem."
 
Well, maybe not.

I set the keep alives off and when I ran the program, it loaded 20
records and then locked up in studio without ever getting to a
breakpoint or error.
 
Let me though in some more pointers to check:

1. Are you using SSL? If so, try without it and see if there is a
difference. SSL connections are typically left open longer to conserve
resources. If your web service can't establish an SSL connection, a Web
service request won't show in your server logs.

2. Could it be that the the processing on the server is slow, and your web
server is overwhelmved? If the IIS pipe is full, I suspect it will reject new
requests without passing them to .NET to be logged and processed.

3. When you say you're looking at the server logs, are you refering to the
..NET Web services logging feature or the IIS logs? If you're not checking the
IIS logs, check them to see if the number of requests matches the number of
requests in the .NET WS logs.

good luck,
Waleed
 
More info. See article below.

So, what I think is:

My dev workstation is an XP Pro machine. I'm assuming that it has the
10 user limit ( which is great from a marketing perspective that M$ can
limit people from running an XP server, but sucks eggs as far as me, the
developer, wanting to benchtest a production web service locally!) The
article says that even if I disable the keep-lives, I still am up
against the 10 person limit....in fact, by disabling the keep alives, my
web service calls probably happen so fast they overwhelm the II5 server
on XP and thus the lock up decribed below ( possibly ).

So, my guess is the disabling of the Keep-Alives on my production web
server ( which is running 2k server ) may have the intended effect of
lessening the number of calls to the web service that return the 403.9
error.

We'll see!

http://support.microsoft.com/default.aspx?scid=kb;en-us;262635

Error Message: HTTP 403.9 - Access Forbidden: Too many users are connected

RESOLUTION
If more than ten connections are needed, use the version of IIS 5.0 that
is included in Windows 2000 Server.

If you use IIS 5.0 on Windows 2000 Professional or IIS 5.1 on Microsoft
Windows XP Professional, disable HTTP keep-alives in the properties of
the Web site. When you do this, a limit of 10 concurrent connections
still exists, but IIS does not maintain connections for inactive users.
 
Waleed said:
Let me though in some more pointers to check:

1. Are you using SSL? If so, try without it and see if there is a
difference. SSL connections are typically left open longer to conserve
resources. If your web service can't establish an SSL connection, a Web
service request won't show in your server logs.
Click to expand...

No SSL.
2. Could it be that the the processing on the server is slow, and your web
server is overwhelmved? If the IIS pipe is full, I suspect it will reject new
requests without passing them to .NET to be logged and processed.
Click to expand...

Quite possibly. I thought it was a dual proc, but now looking via the
MMC is is single proc!

With the help of this group, at least I am now able to trap the error --
403 -- whereas before it just looked like the call was goings to "no
where" as the title of this thread says....
3. When you say you're looking at the server logs, are you refering to the
.NET Web services logging feature or the IIS logs? If you're not checking the
Click to expand...

I am refering to the IIS logs.
IIS logs, check them to see if the number of requests matches the number of
requests in the .NET WS logs.
Click to expand...

I wasn't aware of those! Where are they located?
 
2. Could it be that the the processing on the server is slow, and your web
Quite possibly. I thought it was a dual proc, but now looking via the
MMC is is single proc!
Click to expand...

This theory should be easy to test. Put a delay or something to slow down
your client and see if there is a difference.

Also, you mentioned that in the third attemp all goes fine. How long do you
need to leave a period of inactivity before the Web service starts acting up
again?

I wasn't aware of those! Where are they located?
Click to expand...

I'm refering to the tracing logs. These can be activated if you're using WSE
2.0. They keep a log of the complete message in a file for debugging.
However, if the request is not showing on your IIS logs then it's not going
to show on your .NET trace anyway.
 
Indeed, it is likely that you are running into the limit of 10 users on
Windows XP. But even if you upgraded to Windows 20003 Server which does not
have this limit, it is still possible to overflow the IIS request queue. You
can increase the request queue size through IIS configuration, but if you
push many requests you can eventually overflow the queue no matter how large
you make it.

The real solution IMO is to handle the error in the client, wait for a while
and then retry the requests that failed.

Regards.
Sami
 
Sami said:
The real solution IMO is to handle the error in the client, wait for a
while and then retry the requests that failed.
Click to expand...

I absolutely agree...that programmatic solution will work no matter which
class of web server that it's consuming the web service from.
 
John said:
Sami Vaaraniemi wrote:




I absolutely agree...that programmatic solution will work no matter which
class of web server that it's consuming the web service from.
Click to expand...

After implementing a new try catch block around the End_method I now see
the cause of the "web service to no where" error, a "503: Service
Unavailable".

There are several articles from MS Support regarding the error (
although they seem to insist that it's unique to II6.0 and I'm
definitely using II5.0/2k ). Anyway, the basics are that the server is
getting overwhelmed by the rapidity of the client requests.

http://www.microsoft.com/technet/pr...IIS/0c92f7cf-8003-42da-8e06-431187ad946e.mspx

Here's the thing though: It seems to occur when I run the requests
initially after the service has been restarted...or if I've made code
changes. My guess is there's a 'wakeup' stage where the dll is loaded
into memory. If I wait 5 minutes and run it again ( which is what my
client code does ) then all the requests process lightening fast(!).

I know I can implement some retry procedures in my async calls to handle
the 503 error; however, I am wondering, does anyone have any advice as
to IIS5.0/2k performance, optimization or configuration tuning changes
that can allow my IIS server to handle the requests?

I am generating about 30 calls per second to a database insert process
on a DB2 database.

Is there any 'queue' or 'pipe' configuration setting that can be
increased to allow the calls to be queued up for later processing if the
server is *busy* ?








In my production logs, many of the client calls to the web service report:

7/10/2005 8:11:21 PM:Callback System.Net.WebException: The request
failed with HTTP status 503: Service Unavailable.
at
System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage
message, WebResponse response, Stream responseStream, Boolean asyncCall)
at
System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult
asyncResult)
at PolarFW.as400data.Service1.EndinsertPolarData(IAsyncResult
asyncResult)
at PolarFW.PolarFW.AsyncCB(IAsyncResult ar)
 
Back
Top