Jeanine Matthews - Divergent Hunter

[totten]

"Divergents threaten that system. It won't be safe until they're removed."
  -- Jeanine Matthews

I've been working recently on some projects that involve running the test-suite in different ways. One frustrating thing is that there's often some number of tests which produce different results depending on how they're run. For example, suppose we run api_v3_AllTests in these four configurations:

• Call the tests with Civi's bundled copy of phpunit -- with one big test run
• Call the tests with Civi's bundled copy of phpunit -- with a bunch of individual test runs
• Call the tests with a recent, standalone version of phpunit -- with one big test run
• Call the tests with a recent, standalone version of phpunit -- with a bunch of individual test runs

We would get different results in each case, and we haven't had a great tool to suss out those inconsistencies. So I've started some scripting to compare/sort the variations. For api_v3_AllTests (using code circa yesterday with MAMP 3.x, PHP 5.3, on my local machine, with buildkit's "drupal-clean" configuration), the comparison is:

https://docs.google.com/spreadsheets/d/1LOH_dfWyVWVcvJu_QWCGsD0KtZz6Dsc7DUFCHUokm5Y/edit?usp=sharing

About 70/2400 tests (~3%) produce inconsistent results. (We can expect the % to be higher in the WebTest suite -- but that suite is particularly slow, and I'm trying to get the process/scripts straightened with a faster suite.)

The theory behind some of the inconsistency is that there are hidden interactions between some tests. I've got WIP code to identify test-pairings which produce those inconsistencies and will post when that comes to fruition.

[totten]

One item produced from this effort (which was recently merged) is called "EnvTests". Basically, it allows you run any mix of tests -- which helps to hunt interactions among tests.

For example: CRM_Core_BAO_SettingTest::testConvertConfigToSettingNoPrefetch has been failing when run as part of the CRM suite -- but it passes when run individually. This suggests that there's some kind of antagonist -- somewhere in the suite is a second test that conflicts with this test. But where? There's ~150 test-classes with ~650 test-functions in the CRM suite.

The backtrace mentions ManagedEntities, so my first theory was an interaction with the preceding ManagedEntitiesTest. We can try running the tests together:

Code: [Select]

civicrm$ cd tools
tools$ env PHPUNIT_TESTS="CRM_Core_ManagedEntitiesTest CRM_Core_BAO_SettingTest" ./scripts/phpunit EnvTests
 ==> Test result: PASS

All the tests pass here, so the antagonist is elsewhere. Next theory -- in 4.5, we've started to use ManagedEntities as part of CiviCase. Perhaps one of the CiviCase tests manipulated ManagedEntities -- and then the ManagedEntities is out-of-whack for SettingTest?

Code: [Select]

tools$ env PHPUNIT_TESTS="CRM_Case_BAO_CaseTest CRM_Case_BAO_CaseTypeTest CRM_Core_BAO_SettingTest" ./scripts/phpunit EnvTests
 ==> Test result: FAIL

Sure enough -- either CaseTest or CaseTypeTest is the culprit. So we repeat with smaller chunks:

Code: [Select]

tools$ env PHPUNIT_TESTS="CRM_Case_BAO_CaseTypeTest CRM_Core_BAO_SettingTest" ./scripts/phpunit EnvTests
 ==> Test result: PASS
 ==> Conclusion: CaseTypeTest is not antagonistic

tools$ env PHPUNIT_TESTS="CRM_Case_BAO_CaseTest CRM_Core_BAO_SettingTest" ./scripts/phpunit EnvTests
 ==> Test result: FAIL
 ==> Conclusion:  CaseTest is antagonistic. (But specifically... which test function?)

tools$ env PHPUNIT_TESTS="CRM_Case_BAO_CaseTest::testAddCaseToContact CRM_Core_BAO_SettingTest" ./scripts/phpunit EnvTests
 ==> Test result: PASS
 ==> Conclusion: CaseTest::testAddCaseToContact is not antagonistic

tools$ env PHPUNIT_TESTS="CRM_Case_BAO_CaseTest::testGetCaseType CRM_Core_BAO_SettingTest" ./scripts/phpunit EnvTests
 ==> Test result: FAIL
 ==> Conclusion:  CaseTest::testGetCaseType is antagonistic

At this point, EnvTests has helped us about as much as it can -- we'll have to read the particular tests to understand the details of what's going on. But at least now we know positively which tests to examine.