Rant: don't give us excuses not to run the tests

[xavier]

Hi,

Small issues, but that are honestly a roadblock on my tests

say that I modify api/v3/Whatever.php and the corresponding tests/phpunit/api/v3/Whatever.php

I'd be on the root of my civicrm install and alternate between
me@computer:/var/www/civicrm$vi  api/v3/Whatever.php
me@computer:/var/www/civicrm$vi  tests/phpunit/api/v3/WhateverTest.php

(or whatever editor you want that you prefer joe is quite irrelevant here michal

However, to run the tests I have to:
cd tools
me@computer:/var/www/civicrm/tools$php scripts/phpunit -udb_username -pdb_password -hdb_host api_v3_WhateverTest


This command line is genuinely blocking me and I never remember it (I tend to do the tests when I have no other distraction, ie when I can't google to find that wicked syntax because I don't have internet access).

My issues:
1) I expect the test script to be under tests, or bin, or tool/bin but NOT under scripts
2) I hate when a script can't run from the root of civi (where I am to edit)
3) I never remember the passwords I put for the db. Why on earth do you have the password as a mandatory param whilst you forced me to hardcode the name of the db?
I expect to have a config file (under sites/test or wherever you want) where I store whatever user/password/dbname and what else is needed and never having to type them
4) test_class_name. I want to be able to put the file name, not the class name. Having to use the previous line and forced to replace "/" by "_" is strangely blocking me

Moreover, I'd want to have an easy way to skip the db reset stage (that is taking forever)

So to wrap up to make me (us?) an happy tester,

me@computer:/var/www/civicrm$vi  tests/phpunit/api/v3/WhateverTest.php

me@computer:/var/www/civicrm$php tools/bin/phpunit.php  tests/phpunit/api/v3/WhateverTest.php

or optionally php tools/bin/phpunit.php  tests/phpunit/api/v3/WhateverTest.php --no-db-reset


It would have been proven convenient to have an option to choose different dbs (eg to run the test on a real db or one that has custom fields or whatever), but can leave without

I haven't followed the magic done on cli.php at the latest sprint, could it be put in action here too?

http://wiki.civicrm.org/confluence/display/CRM/Setting+up+your+personal+testing+sandbox+HOWTO

[Donald Lobo]

At the various code sprints we fixed the tests to do the foll:

1. Take your user name / password from ~/.my.cnf  if availaible. This avoids you needing to send in username/password/host

2. Allow a different db name to be sent instead of civicrm_tests_dev

3. Recent tests do not recreate the DB, do they? I've not checked what the current status of this is, but tests are a lot faster these days. If they do, adding a new flag to it should not be too hard

4. You can write a small wrapper shell script to do a:

% pushd tools
% scripts/phpunit WHATEVER_ARGS_YOU SENT TO THIS SCRIPT
% popd

For extra credit u can also convert filename to classname

lobo

[xavier]

At the various code sprints we fixed the tests to do the foll:

1. Take your user name / password from ~/.my.cnf  if availaible. This avoids you needing to send in username/password/host

Ok, will check thx.
I'd prefer that it's a different user than mine, cause I'm likely to have more access rights than just the civicrm test db (and I don't think that the test will go full rampage on other db, but I'd rather have it block at the db access level if they do)

2. Allow a different db name to be sent instead of civicrm_tests_dev

How? Did I miss a doc somewhere?

3. Recent tests do not recreate the DB, do they? I've not checked what the current status of this is, but tests are a lot faster these days. If they do, adding a new flag to it should not be too hard

I'm not sure, it seems to be "thinking" a lot in the init phase before working on the tests.

4. You can write a small wrapper shell script to do a:

% pushd tools
% scripts/phpunit WHATEVER_ARGS_YOU SENT TO THIS SCRIPT
% popd

For extra credit u can also convert filename to classname

Ok, will hack something.

[Eileen]

Good to hear there have been some improvements. I went to write a test night but when I did an svn up nothing seemed to be working - I couldn't check it against the tests on the server as the results aren't showing up there.  I didn't know if the problem was at my end or the server so I just handled the bug in my own code & didn't write the test

I think not having visibility on the test pass rate for trunk (plus active branches) is a blocker on writing tests. Ideally we would be able to see not just the current day but the trend (e.g. to see when they all started failing).

[Donald Lobo]

yes agreed. We do need to have better test infrastructure and coverage on more branches, including trunk.

however, we are also quite maxed out in time and resources (and i know you are too!). so we need to find someone who can help with test infrastructure and rig up various processes Know anyone suitable? Lets see if we can manage to find someone to do this in the community

lobo