@Dewayne
Good thought on the caching hypothesis. I just got a response from the BCL team and they also said that the loop probably wouldn't be a good test because of caching.

I just sent them a follow-up email asking if anything had changed between 1.1 and 2.0 that might have affected the sequence order. I'll publish the answer as soon as I get it.

Since I'm developing on a laptop that is already on the slow side, I fear that I might be restricted to notepad if I took any RAM out...:-)

BTW, Nice move spelunking that method with Reflector. That is why you'll always out-geek me...:-)

I got a response within a few hours from Matt Greig on the BCL test team. Here's what he had to say:

My guess would be that your first method of testing (a hundred thousand loop iteration) is not actually testing the order the properties will be returned more than once. There is quite a lot of cacheing in the System.Reflection code looping is probably just getting the same cached list of properties each time - so really only testing the order once on the first time through the loop when the list is cached (didn't verify this but seems most likely given your results). I would trust the separate processes more.

From looking at this it seems this is a bug in your code (the order dependency), that you have already identified and have a fix for. As to how critical this issue is, or even if it is actually an issue - I can't tell without a lot more information and if the fix is warranted given the risk really is up to you and your management to make the call on.

I would also try to verify that it is indeed the order that is causing the failures (you didn't seem certain of this). Maybe something as simple as printing the expected and actual list of properties when the failure occurs to see if it is just the order could verify that.

Hope some of this helps,

Matt Greig
BCL test team

Here's my follow-up question to them:

Thank you very much for the prompt reply and the insight into how caching works in loops.

I verified that the reflection sequence was actually the problem in our code by following your suggestion and adding debug code to verify how the sequence was being returned. I was able to do this because the properties all have custom attributes with a sequence property on them. (this was how I eventually fixed the dependency problem in our current code branch). The output of the intermittent test failures verifies that the properties were returned in a sequence that the code didn’t originally expect (see code excerpts at the bottom).

The one other interesting discovery we made is that we don’t have this issue if we force mbunit to use the 1.1 framework instead of the 2.0 framework(it’s a config option). Would changes in the 2.0 framework have affected how the order was returned in this method?

At the moment I am assuming that this is only a problem when our code is called from 2.0 code. Would you agree that this is a reasonable assumption to make? If so, I think I have enough evidence to wait until 1Q next year to push our fix out to production since this code isn’t currently being called by any 2.0 code in production.

Thanks again for your help,
Russ

Here's the final response I got back from MS. I was impressed by the level of effort and timeliness that went into helping me. Thanks Matt and kudos MS!

Russell,
I wasn’t around when the changes from 1.1 to 2.0 were made so aren’t that familiar with exactly what changed. But with the documentation saying the order is not fixed it does mean that it is something that we would feel very comfortable changing between versions for any number of reasons – the most likely one seems to be performance.

If you have code that can fairly reliably reproduce the failure on 2.0 as your test seems to be able to and it isn’t able to reproduce it on 1.1, then it seems pretty safe that a change to 2.0 made this more of an issue. I hesitate to say that it is a problem only when calling from 2.0 code. It seems most likely that it was a change only present in 2.0 but it may be some other change just makes it more likely on 2.0. As an example, it could be that the Garbage Collector moving the object in memory may cause the properties to be returned in the different order and some changes to the Garbage Collector made it more likely to move the object in 2.0 vs 1.1. In that case the problem would still be present in 1.1 just less visible and thus harder to reproduce. Without knowing what causes the properties to come back in the specific order it is hard to make any firm claims about when it will happen and when it won’t. I guess what I’m saying is that your assumption that it is a 2.0 issue seems reasonable but I am not able to guarantee it in anyway.

Basically, I think you have it right. You have found the bug, and if it is an issue (depends on what happens when the list is not in the expected order) it is mostly (if not only) when running on 2.0. To me it sounds safe to wait, but if the results from hitting the bug are severe enough (i.e. leaking someone’s personal private data) it may justify more investigation to try and see if it can be reproduced under 1.1. At that point you really have two options that I could see. The first would be doing heavy debugging on the 2.0 version to try and determine what is causing the order to be different and then reproducing those conditions in 1.1 and verify it doesn’t affect the order. This approach would be the most complete but the debugging of the 2.0 version could be very intensive and may not justify the cost. The second approach would be more a blind brute force approach. This approach would be testing without knowing what is causing the properties to be returned in a different order in 2.0 and thus much harder to complete verify. Basically at that point you are needing to test Type.GetProperties() to see if an assumption about the order being constant is the case or not under 1.1, which would lead back to your question of what conditions that may cause this order to be different – and honestly this could be almost anything most likely though I would think would be what you originally mentioned threading or memory/cpu pressure. The trick would then be to do enough testing that you can justify an argument that it is constant. The advantage of this approach is that you only need to do as much testing as is warranted and it gradually increase confidence (rather than all or nothing like the previous approach), however the drawback is that it will never give you complete confidence that you have reproduced what caused the 2.0 order to change.

Hope this gives you what you need to know to move forward,
Matt