Simple "export all"

[djtrouten]

A great feature to add would be a simple "export" all function that would export the entire user database -- all contact fields, custom fields, contributions, event registrations, memberships, etc. To do that now I have to "export" four different times, and for the contact list I have to create a custom mapping that catches all of the fields (and hope that I remember which of the standard fields I'm using). Dumping the entire database to a spreadsheet -- whether for backup or to use the data in another program -- is such a standard thing that it really should be a lot more straightforward than it is.

[Donald Lobo]

any specific reason why using mysqldump which basically does exactly that does not meet your needs?

lobo

[Denver Dave]

Thanks - I have also been thinking that an export all would be handy.  We have quite a few custom fields and we export to .csv files to be used by others as spreadsheets.  I'm sure we don't have all the custom fields included.

A MySQL dump to a .csv file would do each table, but I don't think it would put them together with the fields for each contact.

We also do a phpmyadmin dump of the database for backup purposes.

[Donald Lobo]

hmm, not sure i understand. for backup purposes, you should use mysqldump or a relevant backup utility for mysql

what would the functionality of "export all" be that mysqldump would not cover?

lobo

[cgnicholas]

Let me clarify by way of our event registration.

Once people register for our events, we would like to automatically make them Drupal users, and in our 'welcome' email send them a link to begin using our site.

Therefore, we need to automatically turn them into full-fledged Drupal users, not mysql backups. :-)

thanks!
Chris Nicholas

[Donald Lobo]

and how is your use case below connected with this thread? i think i'm missing something

lobo

[cgnicholas]

Apologies, as I am a newbie and exploring the relationships between what it means to be a CiviCRM 'member', versus part of a group, how these relate to Drupal organic groups, and how to synchronize them bi-directionally. This came from just wanted to export things automagically as some kind of ascii, and running external syncs, so indeed, not relevant here...sorry!

Chris

[djtrouten]

I have some thoughts on the benefits of a simple "export all" function as opposed to a MySQL dump.

First, I think what I really want is a "backup and restore" function, rather than just "export all." Ideally this function would come in two flavors. One would backup your contact records, including their event registrations, constributions, memberships, relationships, etc. -- all of the stuff that's associated with a particular user. This way everything connected with a specific contact would show up in one place. The second "flavor" would backup all of your customizations, including custom fields, membership types, profiles, etc. -- all of the stuff you stick in apart from the contact-specific info. I'd suggest using a stable file fomat for this, such as CSV, so that this info could be backed up in one version of CiviCRM and restored in the current version.

Benefits:

1. This would make it easy and convenient to get the information out of CiviCRM for manipulation in other programs outside of CiviCRM.

2. A lot of the people who use CiviCRM are more comfortable working within the program than hacking around outside of it. If you don't know what you're doing, you can do a lot of harm by rooting around in your MySQL file. This is particularly true with a complex program like CiviCRM.

3. If somebody misses a few upgrades, this could provide a direct path for upgrading to the newest version, rather than having to install and uninstall each upgrade you've missed so you can run the database upgrade utility.

[Donald Lobo]

This is more a DB / schema related issue and hence is probably a bit easier to code for folks familiar with MySQL / PHP. This would make a great community contribution . Would be awesome to backup your thoughts with some code.

It does get a bit hard to seperate "contact data" from "customized data". The custom values for a contact are pretty much useless without the "custom groups / fields" information. Quite a few values in the "contact data" reference identifiers in the "customized data" (option groups, contribution pages/types etc). So while in theory there seems to be a separation, i think reality dictates otherwise

lobo

[djtrouten]

I wish I had the skills to make this happen, but I'm afraid my programming experience is limited to a bit of hacking around with Basic and Fortran decades ago. Still, I figured I'd throw the idea out there in the hope that it would catch somebody's eye. Being able to backup and restore data seems like such a basic function that I'm surprised it's not already a part of the program.

My thinking in separating "contact data" and "customized data" is that the former probably changes a lot more often than the latter. Once I've built the "structure" by creating custom fields, membership categories, profiles, etc., odds are pretty good that I'll mostly leave that alone unless there are structural changes in my organization that need to be reflected in the structure of my database. But I'll probably change individual user records every day. Breaking out the contact-specific information would make it less of a chore to backup my user data, and on those occasions when I've made structural changes I could do a fresh backup of that part of things. I understand that a lot of the contact data is useless without the customized data, but because it changes less often it could probably be backed up less often.

In the event of catastrophic failure of some kind, I could just install the most current version of CiviCRM, then install my most recent "customized data" file, and then dump my user data back in. Having two backup files to restore would still be a lot easier than having to deal with the many interrelated tables in a CiviCRM installation -- especially for a person without much MySQL experience (probably most folks who use the program).

[Donald Lobo]

ok, hopefully someone will coding skills / resources / $$ will come around and help fill this gap

lobo

[djtrouten]

Hopefuly so. In my admittedly uninformed opinion, it probably wouldn't be terribly difficult. It seems like it would largely be a question of being able to grab the right tables in the right order, and then being able to reverse the process, all from within CiviCRM. But there would also be an ongoing task of keeping up with schema changes as the program continues to grow and improve.

[djtrouten]

Let me add one more thought on this. If somebody has the programming skills and is interested in developing this feature, I'd be interested in chipping in some financial support for time to takes to build this feature.