CiviCRM Community Forums (archive)

Hi,

What do you mean by fish-eye display ?

The import already transfer all the lines of the csv into a temporary mysql table. I would make sense indeed to be able to split (or re-run) an import process and be able to see in the import tables which rows have not been imported.

I'm wondering to what extend adding an extra status column on these tables shouldn't allow to do both that and the error records (I would love being able to fix some contacts without having to re-run a complete import)

X+

Thanks for the replies - prompt and intelligent as always!

re xavier's "Fish-eye" question - a bit like http://interface.eyecon.ro/demos/fisheye.html but rather than emulating the Mac OS X application doc, imagine a table resizing height by row by vertical proximity, width by column by horizontal proximity, and font-size by both combined, going from tiny 1px text in 3px x 8px cells which simply commuicate how many rows and columns there are, which cells have data and which are completed (e.g. green=done, pale grey=to do), to ordinary spreadsheet size for what is currently being worked on or focused on (with a 'full cell' button label to show a full cell scrollable popup for cells with long content such as notes).

re Lobo's post, the aim is definitely to offer a user-friendly reliable, interactive 'slow but steady' interface for the Import API which can sit alongside fast, task specific/programmatic import options.   

re both posts' more general points, I think there's another dimension to this. The current import script already has a kind of resume functionality with the 'status' column of the temporary tables - but as I've found in the past only certain types of errors lead to resume attempts and the results are sometimes not what is expected. Also, looking at what records I have left of dead imports of my own, many of them failed during the row-by-row process.

Giving it some more thought, speed is expendable because expert-led mega-import projects will be best done bypassing the import interface entirely, programming into the import API directly - and most orgs with high-performance requirements and a decent database budget already seem to be pointing this direction. The priority for interface import users are that it's failsafe, and that they have tools available to avoid bad data being imported by mistake (which can include up-to-date data being overwritten by perfectly valid data which only a human can judge is less up to date).

So here's what I think would work best: an asynchronous chained process with each execution of the Import API being just one row and initiated by a chain of AJAX functions.

One major benefit is that unlike a  line of cheap Christmas lights, if one bulb fails, the rest of the process doesn't just die - whether it's an error 500, 'MySQL gone away', etc, AJAX can try again, recording each attempt in a data-* attribute in the tr element for that row, and then mark that row as an error and moves on to the next row if it fails more than X (5?) times.

The other major benefit is that it makes all sorts of useful interactivity possible and simple. A few examples:
- Pausing and resuming imports, at any time, for any reason. AJAX simply doesn't tell the next row to go.
- Setting breakpoints, so a cautious user can check how the import's going after, say, 50 rows before allowing the rest to go. 
- Manually fixing invalid data or choosing whether to merge possible duplicates case-by-case while the rest of the import carries on importing.

And a few rather attractive possibilities that would require a little extra work...
- Cell-by-cell change reporting. When an import is about to 'update' rather than 'add' to an entity, we can get the current data with the AJAX API, or after with some kind of pull or FK from the change log, and show what was overwritten as well as what the new records are.
- Cell-by-cell undo. If we have what a record was as well as ID etc, we can 'undo' any change in an import with a simple AJAX button that takes the .text of the 'before' record and writes it using the API.

I'm sure there's more. Essentially this'd give two import options: the experts can set up their own top speed imports which require technical skills and which trust trust them to know precisely what they're doing. For everyone else, there's a slow but steady route, built on the same core script from the API, which gives numerous opportunities to spot problems of all types and correct them immediately.

Ha - good point! Definitely needs more thought...

(p.s. I trust you meant I'm 'on to something' as in there's some substance to the idea, rather than 'on something' as in narcotics...)

X: 

One quick thought to minimise the hit on the server (disclaimer - performance isn't at all my area of expertise so I may well have missed something obvious):

- To keep the data traffic down to something negligible, each AJAX request passes only the temporary table name and row number, to a wrapper around the API call which retrieves the data from the appropriate row in the temporary table and passes it to the import API. In-line cell by cell edits save to the appropriate row in the temporary table and mark it as not done if it's marked as done.
- To keep the volume of requests at any given time low, as well as chaining the requests to only happen either after processing the previous completed request or when it is clear that the chain has broken, wrap each call in a setTimeout or equivalent, spacing each request out by a number of milliseconds.

The setTimeout option also opens up the interesting possibility of an option, maybe presented in an 'Advanced Options' accordion, maybe as a slider next to a 'Go/Pause' button, to allow users to control the speed of their import: speeding it up if their import needs to be done comparatively quickly, and slowing it down if they aren't in a rush and want to minimise the impact on other users or their own CiviCRM activities in other browser tabs, or, if they want to be able to watch and control the import more closely as it happens.

Unless I've missed something, it should be possible to keep the impact on site performance of an import as close to equivalent of one extremely busy user (I'm not sure how this compares with the current process).

It also feels like there could be a lot of uses for the ability to export a sheet, make changes, import it and slow down the import enough to be able to watch each row go in and interrupt where necessary.