CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site

This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Developer Discussion »
  • APIs and Hooks (Moderator: Donald Lobo) »
  • getCount fix?
Pages: [1]

Author Topic: getCount fix?  (Read 2092 times)

adixon

  • I post frequently
  • ***
  • Posts: 314
  • Karma: 19
    • Blackfly Solutions
getCount fix?
May 10, 2012, 08:21:00 am
There's another topic about the getCount behaviour with the required rowCount parameter to have it make sense, but the problem is really deeper than that - the getCount implementation is useless for large implementations because it loads them all instead of just counting them (so runs out of memory).

Is it because no one else has cared, or is there a reason why there's not a real implementation (i.e., one that doesn't load them all, but uses a SQL count to count them)?

Happy to look deeper and provide a patch, but just looking for hints about what seems obvious.

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: getCount fix?
May 10, 2012, 02:23:46 pm
Not that we don't care about it, but it hasn't make it near the top enough to get it implemented properly.

I'm not sure there is a simple and easy way to make it generic, but if you could find a way of having a _civicrm_api3_basic_getcount like there is a  _civicrm_api3_basic_get

then having a generic getCount isn't far away (under api/v3/Generic)

Haven't looked at what DAO offers, but definitely would be a better implementation. Would be great if you could have a look

X+



-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

Eileen

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4195
  • Karma: 218
    • Fuzion
Re: getCount fix?
May 10, 2012, 02:53:14 pm
GetCount is simply a wrapper for the get function - but we implemented it as a separate function because we knew sooner or later we would want to refine it.

It should be fairly easy to do a generic getcount for the DAO based functions - but the ones like contact, contribute, participant which rely on the query object will need a little more thought.

Note that if any individual API has a getcount function it will be used - otherwise the one in the generic file will be used.
Make today the day you step up to support CiviCRM and all the amazing organisations that are using it to improve our world - http://civicrm.org/contribute

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: getCount fix?
May 10, 2012, 11:11:35 pm
The API is somewhat different from the rest of the code as we don't control the consumer side and it might be better to let them use the right getcount api that is badly implemented than having them calling get and do the count, as when we fix it, it's automatically fixed everywhere.

For the contact.getcount, this code (from api v2, should/might do the trick?)

Code: [Select]
   
function civicrm_api3_contact_getcount($param){
    require_once 'CRM/Contact/Form/Search.php';
    $newP = CRM_Contact_BAO_Query::convertFormValues( $params );
    $query = new CRM_Contact_BAO_Query( $newP );
    return array ('count'=>$query->searchQuery( 0, 0, null, true ));
//here, probably one of the few cases where we don't use civicrm_api3_create_success as we don't want a values array, just count, to be checked

}
-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

adixon

  • I post frequently
  • ***
  • Posts: 314
  • Karma: 19
    • Blackfly Solutions
Re: getCount fix?
May 11, 2012, 07:01:58 am
Thanks for the background and hints, and I'll take this opportunity to start understanding the api a bit more.

Eileen

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4195
  • Karma: 218
    • Fuzion
Re: getCount fix?
May 12, 2012, 03:20:26 am
We'd like to welcome you to the API Team - but first you have to pass the stringent entry criteria - you have to tell a bad joke
Make today the day you step up to support CiviCRM and all the amazing organisations that are using it to improve our world - http://civicrm.org/contribute

adixon

  • I post frequently
  • ***
  • Posts: 314
  • Karma: 19
    • Blackfly Solutions
Re: getCount fix?
May 15, 2012, 02:50:44 pm
bad jokes - i'll have to think one up.

I've got what seems to me a reasonable solution, involving a few small changes.

http://issues.civicrm.org/jira/browse/CRM-10229

adixon

  • I post frequently
  • ***
  • Posts: 314
  • Karma: 19
    • Blackfly Solutions
Re: getCount fix?
May 16, 2012, 06:37:24 am
For reference, my patch doesn't fix all getCount implementations. As per xavier's helpful notes, getCount is implemented with a 'generic' function (i.e. it's used as a fallback if an entity doesn't implement it specifically). The current implementation just calls get and then counts the result. What my patch does is add an extra parameter "getCount" = TRUE before doing that call, and then if the result is a scalar instead of an array, assumes the entity is implementing the new getCount parameter and returns that instead of the count.

For the contact implementation, I dug into the dao object and saw that when the sql is generated by the query method, there's an optional first argument 'count' which if set to TRUE will generate the sql I want, so it made sense to use that. I only found one other case of code actually using this count parameter, but I'm crossing my fingers that it should work as advertised and it did in my various tests. So the rest of the patch (CRM/Contact/BAO/Query.php and api/v3/Contact.php) are really just adding the extra parameter and passing it down.

My hope is that any other implementations of getCount would be fairly straightforward and can be done as needed now.

You might have thought that I should follow the pattern set up in the api and implement the getCount for each entity, but that would mean copying a lot of the same stuff in the get implementation, so I opted for a more surgical approach ...

I've added patches to the issue for both 3.4 and trunk/4.2. The trunk/4.2 patch works on 4.1 with minimal cleanup, I'll post a clean 4.1 patch if anyone requests.

adixon

  • I post frequently
  • ***
  • Posts: 314
  • Karma: 19
    • Blackfly Solutions
Re: getCount fix?
May 22, 2012, 08:31:08 am
Anyone? Don't want to loose this ...

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: getCount fix?
May 22, 2012, 03:21:34 pm
Hi,

The 'normal' way to have a specific behaviour for an entity that is different than the generic is to simply add it (in api/v3/Contact.php)

function civicrm_api3_contact_getCount($params) {
   return civicrm_api3_contact_get($params,true);
}

by doing it that way it makes it clearer what entities have been fixed already.

and instead of adding a getCount in the param, I'd change the signature
function civicrm_api3_contact_get($params,$countOnly=false) {
}

Otherwise on api v2, search_count was using
    return $query->searchQuery( 0, 0, null, true );

but I think what you did by modifying the apiQuery is safer (more change to get the same result).

Make sense? Could you alter your patch?
-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

Eileen

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4195
  • Karma: 218
    • Fuzion
Re: getCount fix?
May 28, 2012, 03:58:34 pm
Hi,

I had a go at this but it didn't quite seem to work to me - the problem was with the bit in the apiquery returning the count I think as the SQL seemed to be correct

      if ($count) {
        $values[] = $dao->{'count(*)'};
      }

I have gone with the original plan of having a get & a getcount action - but have moved shared code into separate functions. The getcount function is not yet committed due to above but this is what I was working with. I can't see any reason contribution should not look much the same

Code: [Select]
/**
 * Get the count of objects that meet search params
 *
 * @param array input parameters
 *
 * @return integer count of result
 * @static void
 * @access public
 *
 */
function civicrm_api3_contact_getcount($params) {
  $options = array();
  _civicrm_api3_contact_get_supportanomalies($params, $options);
  $contacts = _civicrm_api3_get_using_query_object($params,$options, 1);
  return $contacts;
}
Make today the day you step up to support CiviCRM and all the amazing organisations that are using it to improve our world - http://civicrm.org/contribute

Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Developer Discussion »
  • APIs and Hooks (Moderator: Donald Lobo) »
  • getCount fix?

This forum was archived on 2017-11-26.