Use vanilla phpunit to execute the unit-tests

[Roman Zimmermann]

Currently I try to use the current vanilla phpunit to execute the tests, instead of the phpunit in tools/scripts/.

For me it seems that everything just works, but I was not able to execute all unit tests up to now.

To setup phpunit from scratch we need two files bootstrap.php and phpunit.xml:

tests/phpunit/bootstrap.php

Code: [Select]

<?php
$GLOBALS['mysql_user'] = 'root';
$GLOBALS['mysql_pass'] = 'somepass';
$GLOBALS['mysql_host'] = 'localhost';

include 'CiviTest/civicrm.settings.php';

tests/phpunit/phpunit.xml

Code: [Select]

<phpunit bootstrap="bootstrap.php"
        convertErrorsToExceptions="true"
        convertNoticesToExceptions="true"
        convertWarningsToExceptions="true"
        stopOnFailure="false">
</phpunit>

Now I am able to execute phpunit (as installed by my distribution's package manager) simply with:

Code: [Select]

phpunit AllTests

Note that this configuration is much stricter than the old one, so some tests may fail simply because they produce notices or warnings.

This is how far I got this night, until PHP reached it's memory limit of 128MB:

Code: [Select]

CRM_Core_BAO_CustomValue
 [x] Type check with valid input
 [x] Type check with invalid input
 [x] Type check with wrong input
 [x] Type to field with valid input
 [x] Type to field with wrong input
 [x] Fix field value of type memo
 [x] Fix field value of type memo with empty params

CRM_Core_BAO_Email
 [x] Add
 [x] Hold email
 [x] All emails

CRM_Core_BAO_CustomValueTableSetGet
 [x] Set get values date
 [x] Set get values yes no radio

CRM_Core_BAO_CustomValueTable
 [x] Store country
 [x] Store state province
 [x] Store date
 [x] Store rich text editor
 [x] get entity values
 [x] Custom group multiple

CRM_Core_BAO_CustomGroup
 [x] Get tree
 [x] Retrieve empty params
 [x] Retrieve invalid params
 [x] Retrieve
 [x] Set is active
 [x] Get group detail empty params
 [x] Get group detail invalid params
 [x] Get group detail
 [x] Get title with invalid params
 [x] Get title
 [x] Delete group
 [x] Create table
 [x] Check custom field
 [x] Get active groups with invalid params
 [x] Get active groups
 [x] Create
 [x] Is group empty
 [x] get group titles with invalid params
 [x] get group titles

CRM_Core_BAO_Address
 [x] Create
 [x] Add
 [x] all address
 [x] nullall address
 [x] Get values
 [x] Parse street address

CRM_Core_BAO_IM
 [x] Add
 [x] All i ms

CRM_Core_BAO_Phone
 [x] Add
 [x] All phones

CRM_Core_BAO_Preferences
 [x] Value options
 [x] Set value options

CRM_Core_BAO_CustomValueTableMultiple
 [x] Custom group multiple single
 [x] Custom group multiple double
 [x] Custom group multiple update
 [x] Custom group multiple old formate

CRM_Core_BAO_OpenID
 [x] Add
 [x] If allowed to login
 [x] All open i ds

CRM_Core_BAO_FinancialTrxn
 [x] Create

CRM_Core_BAO_Location
 [x] Create with missing params
 [x] Create without loc block
 [x] Create with loc block
 [x] Delete loc block
 [x] Loc blockget values

CRM_Core_BAO_CustomField
 [x] Create customfield
 [x] Get fields
 [x] Get displayed values
 [x] Delete customfield

CRM_Event_BAO_ParticipantStatus
 [x] Create and delete
 [x] Add status type
 [x] Retrieve status type
 [x] Set is active status type

CRM_Event_BAO_Participant
 [x] Add
 [x] get values with valid params
 [x] get values without valid params
 [x] event full
 [x] importable fields
 [x] participant details
 [x] delete participant
 [x] check duplicate
 [x] Create
 [x] exportable fields
 [x] fix event level

CRM_Utils_String
 [x] Strip path chars

CRM_Utils_Mail
 [x] Format r f c 822

CRM_Member_BAO_Membership
 [x] Create
 [x] Get values
 [x] Retrieve
 [x] Active members
 [x] Delete membership
 [x] Get contact membership
 [x] get contribution page id
 [x] get membership starts
 [x] Get membership count
 [x] status avilability
 [x] sort name
 [x] delete related memberships
 [ ] Renew membership
 [ ] Stale membership

This really needs more testing! So take a look and tell me about everything that doesn't work as expected!

Making the tests execute faster would help a lot. So I'm going to take a look at the ongoing work in that direction.

[xavier]

Hi,

Why don't you increase the memory for php-cli ? you can either put it in your bootstrap.php or in the php cli settings.

Not sure I understand why you are saying it's stricter, but if it's about the error reporting level, E_NOTICE and co, i'd suggest you change as well the level in bootstrap so it's the same as the test in drupal.

Let us know how it progress, I like the idea of using the phpunit from the distributions (I have like 30 versions of php unit installed by civicrm spread all around my hard disk

As for the speed (I *very much* like the idea of speeding it up), the key factor is if you re-init the database or not. Some tests leave the db in a clean state (ie. like it was before starting), and can explicitly mention they behave, so it doesn't need to drop/create/re-populate the db -that takes forever-

I thought about using MEMORY as the engine, but haven't tried yet see if it makes a difference, or if it behave like the real engines.

So tests/phpunit/api/v3/SyntaxConformanceAllEntitiesTest.php runs about 500 tests in a handful of seconds, thanks to 
    public $DBResetRequired = false;


To be able to put it in more test cases, you need to do the needed in setUp and TearDown to be sure it doesn't leave in the db the records.

Not sure what's the best practice on how to be sure we delete what has been created, but collecting in an attribute of the class the ids that have been created and use the api to delete them in the TearDown should work ok enough.

X+

X+



X+

[Roman Zimmermann]

Why don't you increase the memory for php-cli ? you can either put it in your bootstrap.php or in the php cli settings.

I have no problem with increasing the memory - except when I'm sleeping. There was no time to rerun them with an increased memory limit. BTW: civicrm.settings.php enforces a memory limit of at least 64M so I thought 128M where enough.

Not sure I understand why you are saying it's stricter, but if it's about the error reporting level, E_NOTICE and co, i'd suggest you change as well the level in bootstrap so it's the same as the test in drupal.

It's because of the notices. Most of them are really easy to fix and I think it's not worth it to leave them around much longer.

Let us know how it progress, I like the idea of using the phpunit from the distributions (I have like 30 versions of php unit installed by civicrm spread all around my hard disk

I'd really like to get some other people try this, because I never got the CiviCRM-flavoured phpunit setup to work. Thus, I have no comparison.

As for the speed (I *very much* like the idea of speeding it up), the key factor is if you re-init the database or not. Some tests leave the db in a clean state (ie. like it was before starting), and can explicitly mention they behave, so it doesn't need to drop/create/re-populate the db -that takes forever-

I thought about using MEMORY as the engine, but haven't tried yet see if it makes a difference, or if it behave like the real engines.

So tests/phpunit/api/v3/SyntaxConformanceAllEntitiesTest.php runs about 500 tests in a handful of seconds, thanks to 
    public $DBResetRequired = false;

To be able to put it in more test cases, you need to do the needed in setUp and TearDown to be sure it doesn't leave in the db the records.

Not sure what's the best practice on how to be sure we delete what has been created, but collecting in an attribute of the class the ids that have been created and use the api to delete them in the TearDown should work ok enough.

That sounds promising! Have you looked into using transactions and savepoints to restore a clean state?

I think best practice is everything that makes writing unit tests easy and guarantees that they execute in reasonable time. After all there whole reason of existence is to quickly give feedback.

[Roman Zimmermann]

I don't think that changing the DB-Engine is in general the right thing to do. I think it's better to keep the test environment as close to the productive environment and as simple as possible.

Minimizing the DB-resets sounds much more promising. I'd suggest using transactions for that (pseudo code):

Code: [Select]

START TRANSACTION;
testSomething();
ROLLBACK;

With MySQL things are a bit more complicated, as it can't handle nested transactions. So as soon as testSomething() uses transactions the ROLLBACK will fail.

So for each test we could ask the following questions:

• Does the test possibly leave the database in an unclean state? -> If no we shouldn't do a reset at all
• Does the test use transactions? -> If no we can use transactions to reset the DB.
• Is it possible to rewrite the test in a way that we can answer question 1 or 2 with no? -> If yes we should do so.

Only if all 3 questions fail to handle the test-case we have to reset the database.

But I think this is going a bit OT now as the original topic was: Using vanilla phpunit.

[xavier]

Yeap, moved the bla about MEMORY in another topic.

http://forum.civicrm.org/index.php/topic,19812.0.html

Sure, unlikely to work 100% the same than InnoDB, but asked a mysql dev and seems to be fairly similar (until you need concurrency but not an issue for tests).

As for the transaction, pretty sure they are enough commit here and there so you won't be able to rollback.

X+

P.S. Assuming we can have more 99% of the tests that run the same on MEM and Inno in one 10th of the time, worthwhile the risk of incompatibilities, that will be caught by the test suites that run non stop on the server anyway.

[Donald Lobo]

1. we have a branch that we are working on:

http://svn.civicrm.org/civicrm/branches/v3.4.tests/

which is a lot faster than our current version. It truncates the DB's once per test file for pretty good performance. Startup time can be pretty good and u can use your existing DB for single test stuff

2. Dont know anything about memory to know what/why u r getting the error

3. we should revert phpunit to the original. There was another hack we did for logging purposes (I found that the hard way when i got an install in supercell).

4. we should create an installer etc and make it easy for a developer to get a test install going

lobo

[Roman Zimmermann]

3. we should revert phpunit to the original. There was another hack we did for logging purposes (I found that the hard way when i got an install in supercell).

Before we can revert to the original phpunit we need to know where the difference is. Executing the tests seems to be no problem.

So what kind of logging is done by the current setup? What is logged in which format? Where are the logs stored? Where is the code that sets up the logging?

[Donald Lobo]

michal will be able to let us know what the differences are. i'll ask him to respond here. the one change that tripped me up when replicating the test suite infrastructure was in phing:

where we modified the file:

./tasks/ext/phpunit/PHPUnitReportTask.php

not sure of the reason, since i wanted to get things to work

lobo

[Donald Lobo]

we use generic phpunit (junit style logging). we run all the tests with this command

tools/scripts/runTests.sh.txt

which calls:

scripts/phpunit --tap --log-junit tests/reports/logfile.$testName.xml -u USERNAME -pPASSWORD -h HOST $testName_AllTests

lobo

[Michał Mach]

3. we should revert phpunit to the original. There was another hack we did for logging purposes (I found that the hard way when i got an install in supercell).

Before we can revert to the original phpunit we need to know where the difference is. Executing the tests seems to be no problem.

Ok - seems like first steps to come back to vanilla PHPUnit are not hard, so I'll give it a try on the testing speed up branch that I'm currently working on (all api_v2 tests in 2 minutes :-) ) and see what it does to my sandbox.

All the changes to vanilla PHPunit can be obtained by comparing files from tools/packages/PHPUnit with files from tools/packages.orig/PHPUnit - however, this might not be reliable source of information, since people sometimes put in hacks without storing original file.

Stay tuned, let me try out the trick with bootstrap.

Thx,
m

[Michał Mach]

Just quick heads up - I'll be a bit delayed with checking out the above, need to wrap up suite speed up first - tried it initially and came to the conclusion that, (doh) introducing a few large changes in the same time seems like good recipe for failure. Suite speed up is almost there, than I'll introduce Xavier's patch to have configurable db (should be a quickie), and with clean slate around those two things, I'll get on getting vanilla PHPUnit compatible. So will start hacking around it sometimes next week.

Thx,
m

[xavier]

Have we moved vanilla?

Just quick heads up - I'll be a bit delayed with checking out the above, need to wrap up suite speed up first - tried it initially and came to the conclusion that, (doh) introducing a few large changes in the same time seems like good recipe for failure. Suite speed up is almost there, than I'll introduce Xavier's patch to have configurable db (should be a quickie), and with clean slate around those two things, I'll get on getting vanilla PHPUnit compatible. So will start hacking around it sometimes next week.

[xavier]

Ping