CiviCRM Community Forums (archive)

Not sure how to fix that, as the table name doesn't contain the date we can't delete it for sure (ie. might be a user doing an import right now).

Would be nice to modify the table name to civicrm_import_{timestamp}_user. As unique, but can be cleaned up.

Was discussing import with changx and CiviCRM-Scalable-Import-Tool and bin/csv/import.php

IMO, the import process should be modified so that the import is split in 3 components


1) the front end that let the user upload+ do the mapping+ verify
(big suggestion, adding a field "source" that you prepopulate with "import {username} {filename}")

2) a new bin/csv/mapimport.php that does the import. basically like bin/csv/import but that does use the map and can define "constant fields" eg contact_type=Individual and group=42

it can be called directly from the cli or from

3) a cron job looks if 1) didn't provide a file to import, imports it and mail the result to the one that submitted it.

I'd love to have an option to import the contacts in error (eg if line 3 has country = UtopiaLand, it would import it without country, tag the contact "incomplete" and add a note "UtopiaLand is not a country"). Much easier to fix when already in civi if that's an odd line, or to advanced search and delete them all the contacts tagged "incomplete" imported by the user today in one go.

Notice how we don't need any temporary tables anymore, and how we could do that by using the existing apis or adding a handful of APIs (eg api.Contact.getDuplicate)

Added benefit, you can run the script from the cron with a low priority, hence not slowing the users because one does an import.

X+

How come is it that whenever I come up with a suggestion, you have already implemented it?  Gets a bit disheartening

yes, would be nice to make import "api-able". We just need someone to write the code

lobo

Think that's a nice starting point : http://civicrm.org/blogs/xavier/api_batch_tools and was discussing with changx today about his tool.
https://github.com/emotive/CiviCRM-Scalable-Import-Tool

To clarify, I don't think that's a good idea to make the import import "api-able", but I want it to be "api-aware". ie. instead of one big monster of a function, split it and have the bolt and nuts accessible throught the api (eg. exposing the field mapping as an entity) or adding a "createMap" action on contact.

When done, that would "only" be a matter of chaining and reusing existing tools, like  bin/csv/import.php, but that the case in general, when everything specific has been developed and added to the common code, that's simply a matter of using the common code

X+

Yes Lobo,

Xavier and I are talking about potential solutions that can be implemented to take advantage of both the user interface that my module offers and the potential power his CSV import tool (using API v3).

I think it can be a win win because my tool currently uses API v2 and API v3 will speed things up quite a bit (especially things with location API). In addition, my module can detect if the host system support shell_exec to use the UI as a wrapper to run from PHP CLI, if not just call the API from The web side of things.


Let me know what you think.