PHPUnit and Packaging

[totten]

To add support for parallel test execution ( https://issues.civicrm.org/jira/browse/INFRA-124 ), we need change some dependencies (e.g. add https://github.com/brianium/paratest and switch to vanilla phpunit ). I've found two arrangements so far which I believe work:

 a. Remove packages/PHPUnit. Add phpunit, paratest, etc. to packages/composer.json.
 b. Remove packages/PHPUnit. Add phpunit, paratest, etc. to buildkit (via its composer.json).
 c. (In theory, it may be possible to manually copy paratest into packages, but my initial experiments left me wary of this -- IIRC, it requires changing the autoloader.)

There are some trade-offs/considerations:

 1. IDE support -- With packages/composer.json, the entire phpunit source tree is inside "packages", so IDEs should automatically support code-browsing/auto-completion of PHPUnit base-classes. (This is handy when writing a test-case.) However, with buildkit's composer.json, phpunit lives in  a separate tree -- so there's an extra config step in the IDE. (You must open two source-trees.)

 2. Maintainability -- With packages/composer.json, each version/copy of Civi has a different version/copy of PHPUnit; but with buildkit's composer.json, there's only one version of PHPUnit. Allowing different PHPUnit versions make it easier to rework the latest Civi (without fear of breaking old versions of Civi), but it also makes it harder to apply changes to multiple versions. (IMHO, if 4.4 is LTS, then we should aim to support 4.4+.)

 3. Efficiency -- With packages/composer.json, each copy of Civi has a copy of PHPUnit+Paratest (~8mb * $x). With buildkit's, there's only one copy (8mb).

 4. Inertia -- Any change requires communicating with our dev community. We can do a blog post and patch "tools/scripts/phpunit", but there's some kind of change.

[totten]

Eileen followed up on IRC to suggest that another criterion be support for running tests in the IDE. When I get a chance to upgrade PHPStorm (or when I get back to my desktop with a more recent version), I'll take a stab at it to see if the recent or proposed changes have made this any easier.

[totten]

OK, I was able to run Civi's tests inside PHPstorm 7.1 on my desktop (which is currently setup per option #b -- buildkit's composer.json). The key points:

 * Don't use PHPUnit 4.x. PHPStorm 7.x is incompatible. (This took a while to figure out...) The Jetbrains blog says that the EAP/8.x-pre-release is compatible.

 * Set a few options in the *project* configuration. (For OS X, go to "PHPStorm => Preferences" or hit "Cmd-,". Not sure where it is on other systems) The settings are:
    + PHP => PHPUnit
         - (x) Use custom loader: BUILDKIT/vendor/autoload.php
         - (x) Default configuration: BUILDKIT/build/BUILDNAME/sites/all/modules/civicrm/phpunit.xml.dist
         - (Adjust BUILDKIT and BUILDNAME as appropriate)
    + Directories
         - Add Content Root: BUILDKIT/vendor

 * To run a test, open the test's .php file and right-click on the class name or function name (depending on which you wish to run) and choose "Run". You may be prompted to create a "Run Configuration" (e.g. "running" by opening a web-browser or "running" by opening phpunit). You have to look closely at the icons to discern which one is phpunit. If you get it wrong, go to the menu "Run => Edit Configurations" and delete the new configuration. Then try again.

I think that either option #a or #b can work with PHPStorm.

---

There are a couple more trade-offs/considerations that didn't make it into the list before:

5. IDE Interaction -- If a developer has other tools (eg an IDE) that constrains the version#, then it may be best to allow the developer to choose what version of phpunit to use. If we follow our current pattern (for packages/PHPUnit or packages/composer.json), then we're copying all the phpunit code into Civi's source tree -- which will produce conflicts if you try to run a third-party build of phpunit.

6. Installation process - Comparison:
  - With option A (packages/composer.json) -- You must use the PHPUnit that's included with civicrm-packages.git. (Anything else will conflict) You never need to separately download phpunit.
  - With option B (buildkit/composer.json) -- If you want all the same tools+versions as civicrm.org (e.g. versions of phpunit/drush/wp-cli/etc), then install buildkit. If you want to pick/choose your own versions, then don't.

[Coleman Watts]

FYI PhpStorm 8.0 has been released.

[Eileen]

OK - so it seems that the desire to use phpstorm to run unit tests puts me in the B camp.

FYI if you run unit tests with coverage in phpstorm it increases the coverage cumulatively (ie. run one test & it calc some percentages, run another & it adds to them). Even cooler it will annotate your code with which lines are tested

[Eileen]

and yes - I am running version 8 - I've been running the EAPs for however long - not because of any deliberate choice I remember making

[totten]

Yeah, I'm in the B camp, too. I'll aim to merge/deploy the changes before CiviCon UK so that it's operational during the sprints -- but will be juggling that with the other 4.5.1 stuff.