Geeks With Blogs

News View Michael Stephenson's profile on BizTalk Blog Doc View Michael Stephenson's profile on LinkedIn
Michael Stephenson keeping your feet on premise while your heads in the cloud

I have just encountered a very bizarre and annoying problem today and thought Id do a quick post to vent my frustration. 

Problem

Ok so in one of the testing environments we have a BizTalk application running which had been fine.  There are a combination of recieve locations from some orchestrations published as web services.  2 ports are one way recieve and the rest are 2 way recieve.

As I say all had been working fine, then today the one way recieve ports started to have problems.  The symptoms are listed below:

  • In the IIS logs I could see that the client had called the service and recieved a 202 response
  • The client responded that all was fine and continued processing thinking BizTalk was processing the message fine.
  • There was absolutely nothing in the event log
  • I could see nothing in HAT related to either of the one way ports, but there was all of the expected data from the 2 way ports so I could assume that HAT was functioning properly
  • I used Debug view and could see nothing unusual
  • There was no instances to been seen through the Group Hub
  • The BizTalk performance counters did not indicate any new messages being recieved by the host

Based on the above it seemed that IIS was getting the request but before anything to do with BizTalk could get hold of it the message was vanishing into thin air, and a successful response was returned.  Even when the message is in the IIS process you would expect to see in HAT the instance of the recieve pipeline which in this case would be the standard XmlReceive pipline.

I have never in my work with BizTalk seen this behaviour before.  It was a standard published SOAP web service, and on any other environment we had seen nothing like this.

Cause

I did all of the normal BizTalk troubleshooting tricks and a few fly by the seat of your pants ideas to try and workout what was going on.  It was a situation like on a lot of integration projects where because BizTalk sits in the middle connecting all of the systems together the fact that it wasnt working was BizTalks fault until proven otherwise.  Im sure most readers will be familiar with that.

Anyway I started to think what the other possibilities could be and eventually by chance a collegue and myself noticed that a 3rd party tool had been setup on this environment to profile and trace .net code.  This tool (Name omitted)  had been setup to profile the w3wp process on the BizTalk servers and it seemed to be doing its job ok on the services where a response was returned, but on the one way ports where the web service returns void it seemed to be breaking things.

We were able to prove this because we started to use a test client to call the services on the individual boxes rather than via the load balanced address.  Then on one box we disabled this tracing and restarted IIS, suddenly the one way port started working where as on the other box it still displayed the above symptoms.

Completely removing the tracing tool from both boxes fixed the problem.

Takeaways

There are a couple of lessons to be learnt from this:

  1. If you do not manage the test environment it is worth ensuring a log of changes to the environment is kept so you can correlate issues to changes.
  2. When considering profiling and monitoring tools such as in this case they should really be tested on a low priority environment to see what their impact on the solution is before they are used an an environment where if they break the solution it will be a big deal.  Unfortunately with this being a big project our development team werent even aware of this tool being used until it was already pretty much deployed.
  3. It it looks like sh!t and smells like sh!t then it probably is, but if it doesnt then it could be anything!.  Basically if you eliminate all of the obvious things that could cause the problem be openminded it could be anything.  To be honest we got lucky in solving this in a reasonable timeframe.  We very nearly didnt even know that this tracing tool was being used at all, and if this was the case it could have taken a long time to workout what the cause of this issue was.

 

 

Disclaimer

I have noticed a few sites that seem to copy the content of blog articles and display them in their own site.  It is a bit annoying that they do not clearly reference or acknowledge the author so I have decided to put this note on the bottom of all of my posts from now so it is clear who wrote it.

This article was written by: Michael Stephenson

The source of this article is: http://www.geekswithblogs.net/michaelstephenson

Posted on Wednesday, November 28, 2007 11:17 PM BizTalk | Back to top


Comments on this post: Expect the unexpected

# re: Expect the unexpected
Requesting Gravatar...
nice post admin thanks.
mobdro for Windows
xender download app
ASDLKSNA
SLASKD
Left by mobdro tv app for pc on May 12, 2017 7:43 PM

Your comment:
 (will show your gravatar)


Copyright © Michael Stephenson | Powered by: GeeksWithBlogs.net