“There are consistencies in what we see as inconsistent”.
I said this out loud today. We have an automated UI test build where all passed 2 times in a row, now without any code changes 2 have failed, with 1 failing test (a different test each time). We’ve seen this with our VM environment and Selenium tests where an element couldn’t be found or a button doesn’t click (we run the same test on small and large and the small one may fail where the large one passes => it isn’t the code 98% of the time in our case). We haven’t been able to avoid these, even though we’ve gone through some refactors and a fair amount of time analyzing things.
So I asked others around me, why is it just one test and not 1 – 10 tests each time if it is a “random” Selenium/environment problem. I’m going to dig into some logs and look for timing issues. Maybe it’s the first or last test to run on the VM before the end of the tests? Who knows…
This all reminded me of the Pragmatic Programmer book and how the bug is probably in my code, not a bug in the OS. In this case, maybe it’s an environment setup issue…