Repeated calls to crm_get_contact() and running out of memory

[davo]

This post is a record of something discovered and solved, rather than a question. The issue was actually solved over on a Drupal forum, but lobo asked me to post the results here in case it's of use to someone down the track.

In summary, I wrote a Drupal module that made many CiviCRM API calls. This all worked fine until I wrote a function that iterated over all users and called crm_get_contact() for each one. Despite calls to unset() for the returned reference, the memory used by the function grew quite quickly and eventually the PHP memory limit was exceeded and I got an out of memory error. The problem was quickly related to the number of users being processed. Clearly there was memory leak of some sort. lobo came to the rescue with

Code: [Select]

CRM_Core_DAO::freeResult()
If I called this function after each call to crm_get_contact() memory usage stayed roughly constant and my function completed normally. Apparently, CiviCRM uses "a pear library (DB_DataObject) which has a nasty habit of storing a lot of stuff in static arrays including memory expensive resources".

Note: freeResult is "pretty destructive, but free's all the mysql resources associated with civicrm." This was ok in my case as I was simply pulling the bits out of a contact record that I needed and moving on.

So there you have it. I hope this is of use to someone one day

[Michał Mach]

So there you have it. I hope this is of use to someone one day

Right on! Thanks a million for reposting. :-)

Thx,
m

[flug]

I was having a similar problem in CiviCRM 4.2.  I wrote a little API extension that runs periodically and updates 200 contacts' custom fields, using API calls.

It was running out of memory somewhere below 200.

The reason is because every single api call to save custom field values, the SQL query info is being cached in $_DB_DATAOBJECT; and (as near as I can tell!) never, ever cleared out!

Now maybe there is a solution to that--maybe there is some setting that could be made in the SQL calls that would allow that memory to be automatically released or something.

But the only good solution I could find is the solution outlined above, to call freeresult() periodically.

However, I didn't want to call freeresult() because all 200 contacts were being pulled in with one SQL query and calling freeresult() would erase the results of that query.

One possible solution would have been to break up the query--say 10 queries with 20 results each rather than one query with 200--and then call freeresult() after each query.  But that seems a bit excessive--if you're pulling down 2000 or 20,000 it seems reasonable to break it up, but can't we process at least a couple hundred records at once?

The solution I came up with was to write the modified freeresult() routine below, which clears out all DB results EXCEPT the one you're still working on (or any other one you specify).

Seems to work fine--maybe this general approach could be part of the solution to some of our out of memory errors on importing, exporting, general API use, and so on?

Maybe it could go in CRM_CORE_DAO alongside freeresult() (or freeresult() could be modified to include this functionality).

Code: [Select]

//Solving a memory leak issue with api use
//Idea is to free all items in the DB_Dataobject EXCEPT the one
//we are still working on.
//Call with something like freeResultExcept(array($dao->_DB_resultid));
function freeResultExcept($except_ids = NULL) {
    global $_DB_DATAOBJECT;

    if (!$_DB_DATAOBJECT ||
        !isset($_DB_DATAOBJECT['RESULTS'])
      ) {
            return;
      }
     
      $existing_ids = array_keys($_DB_DATAOBJECT['RESULTS']);
     
      if (is_array($except_ids)) $remove_ids=array_diff($existing_ids,$except_ids);
      else $remove_ids=$existing_ids;
       
      CRM_Core_DAO::freeResult ($remove_ids);
   
}



[xavier]

It was running out of memory somewhere below 200.
The reason is because every single api call to save custom field values, the SQL query info is being cached in $_DB_DATAOBJECT; and (as near as I can tell!) never, ever cleared out!

Was this something specific to having custom values or is this for other calls as well?
I used the api to write import or sync that work with several 1000th contacts without memory problems (I run for cli so tend to have more memory available there).

Anyway, you seem to be on something, but I think it's worthwhile investigating first why it's cached in the first place and if there is a was to avoid that caching, might help improving more if we can find a generic solution (either in the core or at the api level to avoid caching).

Could you dig a bit more to find if there is a way to avoid the cache instead of having to delete it?

I think the API should consider that the calling party does the caching and we tried to clear as much as we could. The things that make sense caching are the underlying data like pseudo constants, but the data that is queried shouldn't be cached ( nor sql/resources, whatever is cached by the BD_Object). This being said, having an option.cache =boolean might make sense.

X+

[Donald Lobo]

yes, its kinda annoying that $_DB_DATAOBJECT likes to cache things in a static field

in civi core code, we structure queries so we can clean up the static array periodically (typically on every loop of import / send mail etc). makes it a lot easier to manage memory

lobo