Memory Limit on Relationship Imports

[PrincessAng417]

I have been successfully importing Civi Relationships for several weeks on a D7.28/Civi 4.4.5 installation.

As of last week, I am hitting memory issues when trying to import.

My CSV files have only two columns: the external ID of the first contact, and the Civi ID of the second contact.  I have been successfully importing these relationships, maybe 100-150 records or so, at a time for weeks.  Now, I can't even get a CSV file with two records in.

I am hitting: "Fatal error: Out of memory (allocated 121634816) (tried to allocate 48 bytes) in xxx/sites/all/modules/civicrm/packages/DB/mysql.php on line 383"

Or, sometimes, I just white screen.

My hosting company increased the memory available on our VPS account; no success: at one point, we were getting a 500 Server error instead, but still no success.  They believe the problem is not a resources issues, but instead that the import script is entering a loop.

I cannot figure out why we had been successful (for weeks) performing these imports, and all of a sudden we are at a stop.  The timeout errors occur within the first few seconds.  Is there some sort of cache or reserved memory block or something that is “clogged”?

[JonGold]

I wonder if there's something weird in the data set.  Does it matter which two records you import?

[JonGold]

Also, have you turned on debugging and backtrace in Administer menu > System Settings > Debugging and Error Handling?  If so, does it yield additional information?  If you have a backtrace, please paste it here.

[PrincessAng417]

Nothing odd in the data set... I had even gone in and deleted an existing relationship and tried to re-import what originally went through successfully.  It did not go in the second time.

Having said that...

I have now tried to turn on the Debugging and Error Handling... and the import went through??  I turned off the debug settings, and I got a second file of 25 records to go through.

However, it is taking several minutes, and I am not getting any feedback of how far along the import is-- it just flips to the successful import screen.

To add mystery to the whole thing (but may be a clue in what is going on here), the error file that is being generated each time is kicking back with "Record imported successfully but unable to parse the street address: ..."  The street address of the 2nd contact has a PO Box address which is being referenced in the error file.  The 2nd contact, a servicing organization, is the same contact for each record that is being imported.

Why would it be trying to parse the street address of an already-existing contact, when all I am trying to import is the relationship between the two already-existing contacts, by ID numbers?  Again, the file only has two columns.  I do have the parser set to "update", but there are no address fields in the mapping.

And why on earth did it work after turning on the debugging?

[PrincessAng417]

Well, it's getting stranger.

I can now import 13 (not 12, not 14, but 13) relationships in at a time.  I am assigning the relationship to a Household contact.  Two columns in the CSV... one is the Civi ID of the target contact (Household), and the second column is the Civi ID of the servicing contact (Organization).  The Household, Organization, and all of the Individuals within a Household are all existing contacts.

I am still getting address parsing issues on the error file.

However, one Individual contact in each Household is *being changed into a Household contact*.  For sure, I thought this had to be an error on import... but I checked and we are absolutely selecting "Household" and "Update" on the import settings.

This is, again, something we have been doing for weeks with no issue.  I thought maybe there was a cache issue... cleared all caches.  I went into the database, and sure enough... these contacts that were Individuals, who were not referenced at ALL on the import CSV, are now categorized as Households. 

What is happening??

[PrincessAng417]

And just another piece of information... the Households and their respective Individuals are not even close to consecutive in the database.  Their Civi IDs are thousands of records apart.

[JonGold]

Ah - now I think we're on to something.

The Individual getting changed to a Household on import is a known (fixed) bug.  I can't help but wonder if the rest of what you're describing is related to that bug.  I just looked at your profile, and it says you're using CiviCRM 4.2.0.  Is that correct?  If so, I would definitely upgrade to Civi 4.4.6, which fixes CRM-14681.  If you're still having the problem on upgrade, at least we've eliminated several variables - but I'd be willing to bet this presents itself very differently on a newer version!

[PrincessAng417]

I'm sorry; I'm on D7.28/Civi 4.4.5. 

I see that CRM-14681 did affect 4.4.5, so I will upgrade this weekend and give it another try.  Here's hoping that's it!

[PrincessAng417]

Happy to report that upgrading to 4.4.6 seems to have fixed all of the import issues we were having.

[JonGold]

Glad to hear it, and thanks for reporting back so that other folks in the same situation can benefit!