CiviCRM Community Forums (archive)

This post is a record of something discovered and solved, rather than a question. The issue was actually solved over on a Drupal forum, but lobo asked me to post the results here in case it's of use to someone down the track.

In summary, I wrote a Drupal module that made many CiviCRM API calls. This all worked fine until I wrote a function that iterated over all users and called crm_get_contact() for each one. Despite calls to unset() for the returned reference, the memory used by the function grew quite quickly and eventually the PHP memory limit was exceeded and I got an out of memory error. The problem was quickly related to the number of users being processed. Clearly there was memory leak of some sort. lobo came to the rescue with CRM_Core_DAO::freeResult()
If I called this function after each call to crm_get_contact() memory usage stayed roughly constant and my function completed normally. Apparently, CiviCRM uses "a pear library (DB_DataObject) which has a nasty habit of storing a lot of stuff in static arrays including memory expensive resources".

Note: freeResult is "pretty destructive, but free's all the mysql resources associated with civicrm." This was ok in my case as I was simply pulling the bits out of a contact record that I needed and moving on.

So there you have it. I hope this is of use to someone one day

I was having a similar problem in CiviCRM 4.2.  I wrote a little API extension that runs periodically and updates 200 contacts' custom fields, using API calls.

It was running out of memory somewhere below 200.

The reason is because every single api call to save custom field values, the SQL query info is being cached in $_DB_DATAOBJECT; and (as near as I can tell!) never, ever cleared out!

Now maybe there is a solution to that--maybe there is some setting that could be made in the SQL calls that would allow that memory to be automatically released or something.

But the only good solution I could find is the solution outlined above, to call freeresult() periodically.

However, I didn't want to call freeresult() because all 200 contacts were being pulled in with one SQL query and calling freeresult() would erase the results of that query.

One possible solution would have been to break up the query--say 10 queries with 20 results each rather than one query with 200--and then call freeresult() after each query.  But that seems a bit excessive--if you're pulling down 2000 or 20,000 it seems reasonable to break it up, but can't we process at least a couple hundred records at once?

The solution I came up with was to write the modified freeresult() routine below, which clears out all DB results EXCEPT the one you're still working on (or any other one you specify).

Seems to work fine--maybe this general approach could be part of the solution to some of our out of memory errors on importing, exporting, general API use, and so on?

Maybe it could go in CRM_CORE_DAO alongside freeresult() (or freeresult() could be modified to include this functionality).

//Solving a memory leak issue with api use
//Idea is to free all items in the DB_Dataobject EXCEPT the one
//we are still working on.
//Call with something like freeResultExcept(array($dao->_DB_resultid));
function freeResultExcept($except_ids = NULL) {
    global $_DB_DATAOBJECT;

    if (!$_DB_DATAOBJECT ||
        !isset($_DB_DATAOBJECT['RESULTS'])
      ) {
            return;
      }
     
      $existing_ids = array_keys($_DB_DATAOBJECT['RESULTS']);
     
      if (is_array($except_ids)) $remove_ids=array_diff($existing_ids,$except_ids);
      else $remove_ids=$existing_ids;
       
      CRM_Core_DAO::freeResult ($remove_ids);
   
}



yes, its kinda annoying that $_DB_DATAOBJECT likes to cache things in a static field

in civi core code, we structure queries so we can clean up the static array periodically (typically on every loop of import / send mail etc). makes it a lot easier to manage memory

lobo