CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site
This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Developer Discussion »
  • Unit Testing (Moderator: Michał Mach) »
  • Jeanine Matthews - Divergent Hunter
Pages: [1]

Author Topic: Jeanine Matthews - Divergent Hunter  (Read 827 times)

totten

  • Administrator
  • Ask me questions
  • *****
  • Posts: 695
  • Karma: 64
Jeanine Matthews - Divergent Hunter
August 08, 2014, 12:20:03 pm
Quote
"Divergents threaten that system. It won't be safe until they're removed."
  -- Jeanine Matthews

I've been working recently on some projects that involve running the test-suite in different ways. One frustrating thing is that there's often some number of tests which produce different results depending on how they're run. For example, suppose we run api_v3_AllTests in these four configurations:

  • Call the tests with Civi's bundled copy of phpunit -- with one big test run
  • Call the tests with Civi's bundled copy of phpunit -- with a bunch of individual test runs
  • Call the tests with a recent, standalone version of phpunit -- with one big test run
  • Call the tests with a recent, standalone version of phpunit -- with a bunch of individual test runs

We would get different results in each case, and we haven't had a great tool to suss out those inconsistencies. So I've started some scripting to compare/sort the variations. For api_v3_AllTests (using code circa yesterday with MAMP 3.x, PHP 5.3, on my local machine, with buildkit's "drupal-clean" configuration), the comparison is:

https://docs.google.com/spreadsheets/d/1LOH_dfWyVWVcvJu_QWCGsD0KtZz6Dsc7DUFCHUokm5Y/edit?usp=sharing

About 70/2400 tests (~3%) produce inconsistent results. (We can expect the % to be higher in the WebTest suite -- but that suite is particularly slow, and I'm trying to get the process/scripts straightened with a faster suite.)

The theory behind some of the inconsistency is that there are hidden interactions between some tests. I've got WIP code to identify test-pairings which produce those inconsistencies and will post when that comes to fruition.

totten

  • Administrator
  • Ask me questions
  • *****
  • Posts: 695
  • Karma: 64
Re: Jeanine Matthews - Divergent Hunter
August 09, 2014, 06:35:40 pm
One item produced from this effort (which was recently merged) is called "EnvTests". Basically, it allows you run any mix of tests -- which helps to hunt interactions among tests.

For example: CRM_Core_BAO_SettingTest::testConvertConfigToSettingNoPrefetch has been failing when run as part of the CRM suite -- but it passes when run individually. This suggests that there's some kind of antagonist -- somewhere in the suite is a second test that conflicts with this test. But where? There's ~150 test-classes with ~650 test-functions in the CRM suite.

The backtrace mentions ManagedEntities, so my first theory was an interaction with the preceding ManagedEntitiesTest. We can try running the tests together:

Code: [Select]
civicrm$ cd tools
tools$ env PHPUNIT_TESTS="CRM_Core_ManagedEntitiesTest CRM_Core_BAO_SettingTest" ./scripts/phpunit EnvTests
 ==> Test result: PASS

All the tests pass here, so the antagonist is elsewhere. Next theory -- in 4.5, we've started to use ManagedEntities as part of CiviCase. Perhaps one of the CiviCase tests manipulated ManagedEntities -- and then the ManagedEntities is out-of-whack for SettingTest?

Code: [Select]
tools$ env PHPUNIT_TESTS="CRM_Case_BAO_CaseTest CRM_Case_BAO_CaseTypeTest CRM_Core_BAO_SettingTest" ./scripts/phpunit EnvTests
 ==> Test result: FAIL

Sure enough -- either CaseTest or CaseTypeTest is the culprit. So we repeat with smaller chunks:

Code: [Select]
tools$ env PHPUNIT_TESTS="CRM_Case_BAO_CaseTypeTest CRM_Core_BAO_SettingTest" ./scripts/phpunit EnvTests
 ==> Test result: PASS
 ==> Conclusion: CaseTypeTest is not antagonistic

tools$ env PHPUNIT_TESTS="CRM_Case_BAO_CaseTest CRM_Core_BAO_SettingTest" ./scripts/phpunit EnvTests
 ==> Test result: FAIL
 ==> Conclusion:  CaseTest is antagonistic. (But specifically... which test function?)

tools$ env PHPUNIT_TESTS="CRM_Case_BAO_CaseTest::testAddCaseToContact CRM_Core_BAO_SettingTest" ./scripts/phpunit EnvTests
 ==> Test result: PASS
 ==> Conclusion: CaseTest::testAddCaseToContact is not antagonistic

tools$ env PHPUNIT_TESTS="CRM_Case_BAO_CaseTest::testGetCaseType CRM_Core_BAO_SettingTest" ./scripts/phpunit EnvTests
 ==> Test result: FAIL
 ==> Conclusion:  CaseTest::testGetCaseType is antagonistic

At this point, EnvTests has helped us about as much as it can -- we'll have to read the particular tests to understand the details of what's going on. But at least now we know positively which tests to examine.
Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Developer Discussion »
  • Unit Testing (Moderator: Michał Mach) »
  • Jeanine Matthews - Divergent Hunter

This forum was archived on 2017-11-26.