Author Topic: Identifying inconsistent tests (Read 1788 times)

totten · December 21, 2015, 06:32:53 pm

There are a handful of unit tests which produce inconsistent/sporadic results. I just wanted to make a list to keep track of them.

api_v3_JobTest.testCallSendReminderLimitTo
api_v3_RelationshipTest.testRelationshipCreateDuplicateWithCustomFields2
CRM_Core_BAO_ActionScheduleTest.testContactModifiedAnniversary
CRM_Core_Payment_AuthorizeNetTest.testCreateSingleNowDated
CRM_Core_Payment_AuthorizeNetTest.testCreateSinglePostDated

These most likely have different variables at play. Eileen has suggested that some tests depend on mix of timezone/time-of-day. But it might help to have some data.

For a point of comparison, I've tried running each of these suites 6 times locally on 21-Dec-15 circa 6:00pm PST. (MAMP, PHP 5.4; configured for PST). Since it's local, the machine may have more lax capacity (CPU/RAM) than the normal worker nodes. Results:

api_v3_RelationshipTest: Passed 2/10 times
CRM_Core_Payment_AuthorizeNetTest: Passed 8/8 times
api_v3_JobTest: Passed 8/8 times
CRM_Core_BAO_ActionScheduleTest: Passed 8/8 times

totten · July 07, 2016, 12:00:54 pm

The test api_v3_SyntaxConformanceTest::testCreateSingleValueAlter is an interesting one. If you see it failing, don't ignore -- it should be a solvable problem. However, it does have an element of randomness which merits explanation.

In testCreateSingleValueAlter, it iterates through every API entity and makes an example record. Then, it alters every field in the record (one field at a time). For some fields, this value is chosen randomly. Suppose, for example, the "Foo" widget has field "bar" which accepts values "A", "B", or "C" -- then the test will choose "A" or "B" or "C" randomly. However, if "Foo" has an edgy bug (e.g. it crashes whenever "bar==C"), then there is a random probability (1/3) that the test will fail.

When reproducing a failure in testCreateSingleValueAlter, note the failing entity (e.g. "Foo") and try running the test multiple times. In this case, we might run 5 times:

Code: [Select]

cd civicrm
for n in `seq 5`; do env SYNTAX_CONFORMANCE_ENTITIES=Foo ./tools/scripts/phpunit --filter testCreateSingleValue api_v3_SyntaxConformanceTest; done

Depending on the probability of hitting a failure, you may want to adjust the number of trials ("1 2 3 4 5").

CiviCRM Community Forums (archive)

News:

Author Topic: Identifying inconsistent tests (Read 1788 times)

totten

Identifying inconsistent tests

totten

Re: Identifying inconsistent tests