CiviCRM Community Forums (archive)

Guys, Gals, I need a little bit of help.

I am developing a Civi site on a local VM, staging and production sites are on civihosting servers.

I am ready to put up an alpha so that I can start getting feedback on workflows, but I am having a hell of  a time uploading the 10K contacts via the built in import tool. I thought that if I uploaded a standalone DB containing member info, and pointed the import tool to pull the info via sql query, things would be ok.

They really aren't, I am getting the usual server timeouts (500). Now, this happens when I do the much smaller organizations import via CSV, but I usually get over this by hitting back on the browser and clicking import again.  This does not work with the SQL query, no contacts get created.

As a requirement of this project is that contact data be dumped and updated regularly I do not want to get into the habit of creating csv's from sql queries, splitting them up, and sitting through a time consuming upload process.

Now, though I have never been a developer I have become pretty comfortable with PHP and JS over the past two weeks and I am willing to put in time to get this problem solved. The goal of this post is to ask for some advice and perhaps take advantage of the vast amounts of experience you ladies and gentlemen have at your disposal before I jump in.

My initial coo coo ca choo tells me to write an app to run on a local machine that uses the REST api to manually cycle through my db and create contact records.

Something cycles through every contact record in the DB, builds the contact item, sends it out, waits for reply, then off to the next record.

What I am fuzzy on is:

-I know I cannot the be first person to have this issue: is this the right approach for such a large number of contacts?
-If it is, how could I implement the dedupe/fill functionality on these contacts?
-Any tips or suggestions, maybe some example code?


Thanks!

Hi Hershel,

My VM has a civiCRM dev environment that exists to allow me to develop the civiCRM customizations that are required for workflow functionality. It is not a "mirror" of my hosting environment. The reason I am able to perform the import on my local VM is because I have my server timeouts set to an "import-friendly" duration.

Ongoing updates: hosted site is to  to be dumped of content and updated from the "live source" at a defined, regular interval.  The reason this is being done is because a diff+merge would be a lot more work. 

After getting started, I realized that the import tool did a lot more work than it looks like it does :p

I created a new VM and pulled my DB from hosting, started the upload, as I usually do, but it is taking an extremely extremely long time to import, not only that, but my 35gig virtual drive, which had 30+ GB free, is now filled up, seemingly from this import process.

From looking at my HD usage, I can see that the /tmp directory is getting overrun (25.5 GBs). 

A file named trace.1064904004.xt is now at 26.6GB....Any ideas?!