CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site

This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Upgrading CiviCRM (Moderator: Deepak Srivastava) »
  • db Upgrades and civicrm_activity_history
Pages: [1] 2

Author Topic: db Upgrades and civicrm_activity_history  (Read 6261 times)

FatherShawn

  • Ask me questions
  • ****
  • Posts: 372
  • Karma: 25
    • C3 Design
  • CiviCRM version: 4.2.11
  • CMS version: Drupal 7.23
  • MySQL version: 5.5.32
  • PHP version: 5.3.10
db Upgrades and civicrm_activity_history
May 05, 2008, 05:12:12 pm
I tried to upgrade from 1.9.12432 to 2.0.3 today.  Did the dump and schema upgrade and installed in Joomla. In running the multi-step database upgrade after install I received the "3rd party data" warning about activity history. I rolled back to the pre-upgrade backup and consulted the docs.

I read Migrate 3rd Party Activity History Records to 2.0 which states:
Quote
You do NOT need to add activity types for actions which are related to CiviContribute, CiviEvent, CiviMember, CiviMail and Send Email functions.

I don't have any other activity sources.
When I run SELECT DISTINCT(activity_type) FROM civicrm_activity_history; on the 1.9 db I get:
Event Registration
Email Sent
Membership - Good Standing
Membership - Inactive
Membership - New
Phone Call

Why am I getting this error?
Lead Developer, C3 Design.
Twitter: @FatherShawn

Deepak Srivastava

  • Moderator
  • Ask me questions
  • *****
  • Posts: 677
  • Karma: 65
Re: db Upgrades and civicrm_activity_history
May 06, 2008, 02:46:17 am
Quote
You do NOT need to add activity types for actions which are related to CiviContribute, CiviEvent, CiviMember, CiviMail and Send Email functions.

These are the modules in activity_history table for which history records are migrated to 'activity' table provided -
- their activity-types are common such as 'Membership - New' /  'Membership - Current' (for module=CiviMember), 'Event Registration' (for module=CiviEvent) ..etc
OR
- a corresponding entry is found for the activity-type in civicrm_option_value table for the group 'activity type'.

Quote from: FatherShawn on May 05, 2008, 05:12:12 pm
When I run SELECT DISTINCT(activity_type) FROM civicrm_activity_history; on the 1.9 db I get:
Event Registration
Email Sent
Membership - Good Standing
Membership - Inactive
Membership - New
Phone Call

Why am I getting this error?

So there could be cases when all the history records can't be migrated to activity tables for which activity_history table (along with un-migrated records) is retained in your database and hence you receive that warning, pointing to wiki-doc - which tells you 'how you can migrate rest of the history records to activity table'.

In your case the uncommon activity types are - 'Membership - Good Standing' and 'Membership - Inactive'.
« Last Edit: May 06, 2008, 03:10:57 am by Deepak Srivastava »
Found this reply helpful? Contribute NOW and help us improve CiviCRM with the Make it Happen! initiative.

FatherShawn

  • Ask me questions
  • ****
  • Posts: 372
  • Karma: 25
    • C3 Design
  • CiviCRM version: 4.2.11
  • CMS version: Drupal 7.23
  • MySQL version: 5.5.32
  • PHP version: 5.3.10
Re: db Upgrades and civicrm_activity_history
May 06, 2008, 03:03:21 am
Quote
In your case the uncommon activity types are - 'Membership - Good Standing' and 'Membership - Inactive'.

Thanks for pointing me in the right direction.  Since those activity types are created in CiviMember, I would NEVER have suspected them given the documentation.  I'll try treating them as "3rd party" according to the docs and do the data migration again.

I do have one lingering question though.  When I run:
SELECT DISTINCT(activity_type) FROM civicrm_activity_history;
on the post-upgrade 2.0.3 database, I get:
Email Sent
Membership - Good Standing
Membership - Inactive

Now, the activity Email - Email Sent is in the Activity Types list.  Why wasn't it moved and will adding the custom member statuses as Activities really move them too?
« Last Edit: May 06, 2008, 03:25:48 am by FatherShawn »
Lead Developer, C3 Design.
Twitter: @FatherShawn

Deepak Srivastava

  • Moderator
  • Ask me questions
  • *****
  • Posts: 677
  • Karma: 65
Re: db Upgrades and civicrm_activity_history
May 06, 2008, 03:54:47 am
I just checked the query and confirmed that it checks for 'Email' and NOT 'Email Sent', since 'Email' was used as option_value.label for 1.9 and same has been continued for 2.0.

So 'Email Sent' also comes under uncommon activity-types.

(Sorry for not pointing this out in previous post.)
« Last Edit: May 06, 2008, 04:49:14 am by Deepak Srivastava »
Found this reply helpful? Contribute NOW and help us improve CiviCRM with the Make it Happen! initiative.

FatherShawn

  • Ask me questions
  • ****
  • Posts: 372
  • Karma: 25
    • C3 Design
  • CiviCRM version: 4.2.11
  • CMS version: Drupal 7.23
  • MySQL version: 5.5.32
  • PHP version: 5.3.10
Re: db Upgrades and civicrm_activity_history
May 06, 2008, 04:07:58 am
Awesome help!

So the "Email Sent" is probably left over from an earlier version.  There are a lot of those - close to 100.  Is there sql that I can use to find and replace all instances of "Email Sent" with "Email"?
Lead Developer, C3 Design.
Twitter: @FatherShawn

Deepak Srivastava

  • Moderator
  • Ask me questions
  • *****
  • Posts: 677
  • Karma: 65
Re: db Upgrades and civicrm_activity_history
May 06, 2008, 04:47:27 am
OR another option would be to change 'Email Sent' to 'Email' for your history records.

If you want to change it in the file - here it is : CRM/Upgrade/TwoZero/sql/activity.mysql (line 312)
Found this reply helpful? Contribute NOW and help us improve CiviCRM with the Make it Happen! initiative.

FatherShawn

  • Ask me questions
  • ****
  • Posts: 372
  • Karma: 25
    • C3 Design
  • CiviCRM version: 4.2.11
  • CMS version: Drupal 7.23
  • MySQL version: 5.5.32
  • PHP version: 5.3.10
Re: db Upgrades and civicrm_activity_history
May 06, 2008, 06:33:05 am
Well, I must not have understood the suggestion.  I changed line 312 to read:

SELECT @ov_id_mail := value FROM civicrm_option_value WHERE option_group_id = @og_id_at AND label='Email Sent';

I still got the "3rd Party" warning and all the "Email Sent" records remain in activity_history.

Rolling back the install to pre-upgrade to prepare for another run at it...
Lead Developer, C3 Design.
Twitter: @FatherShawn

Deepak Srivastava

  • Moderator
  • Ask me questions
  • *****
  • Posts: 677
  • Karma: 65
Re: db Upgrades and civicrm_activity_history
May 06, 2008, 07:53:40 am
Sorry again. Infact there are two lines which will require the change -
line-312 and line-351

line-312 migrates record to activity table and

line-351 deletes the records from history table.
Code: [Select]
DELETE FROM civicrm_activity_history WHERE module IN ('CiviCRM') AND activity_type IN ('Meeting', 'Phone Call', 'Email', 'SMS', 'Event');

So records must have been migrated but couldn't have been deleted from history table (since line-351 change was not there).
« Last Edit: May 06, 2008, 07:55:40 am by Deepak Srivastava »
Found this reply helpful? Contribute NOW and help us improve CiviCRM with the Make it Happen! initiative.

FatherShawn

  • Ask me questions
  • ****
  • Posts: 372
  • Karma: 25
    • C3 Design
  • CiviCRM version: 4.2.11
  • CMS version: Drupal 7.23
  • MySQL version: 5.5.32
  • PHP version: 5.3.10
Re: db Upgrades and civicrm_activity_history
May 06, 2008, 08:01:18 am
OK - Well I found success by another route!  I executed the following on the table pre-upgrade in mysql:
UPDATE civicrm_activity_history SET activity_type = "Email" WHERE activity_type = "Email Sent"

Then I went through the steps without modifying any of the 2.0.3 code - every step completed and it looks like I have a functional 2.0.3 installation. 

Thanks for pointing me in the right direction!
Lead Developer, C3 Design.
Twitter: @FatherShawn

davej

  • Ask me questions
  • ****
  • Posts: 404
  • Karma: 21
Re: db Upgrades and civicrm_activity_history
May 06, 2008, 09:08:25 am
I made an amendment and a comment to Upgrade Drupal Sites to 2.0 last week re activity history as a result of my experiences:

Amendment (in italics):
Quote
Migrating Activity History Records from 3rd Party Modules or with Types missing from current Activity Types list

If your CiviCRM installation includes integration with other Drupal modules which have inserted Activity History records, or if - perhaps due to activity types having been renamed - there are Activity History records having types not in the current Activity Types list, and you want to retain these records - please refer to this document before continuing the upgrade. You can check for such records with the following  SQL query:

SELECT DISTINCT(activity_type) FROM civicrm_activity_history
  WHERE activity_type NOT IN (SELECT label FROM civicrm_option_value WHERE option_group_id = 1)

Comment:
Quote
    Sometimes I've been warned about 3rd Party Activity records when there were none.

- Same here. I've just encountered this upgrading a db where no 3rd party apps were used. The problem seems to be inconsistency in the activity_type stored in civicrm_activity_history: the db has over 5000 records with type 'Email Sent' and 31 with type 'Email', but civicrm_option_value has only 'Email'. Similarly civicrm_activity_history has records of type 'Event Registration' but civicrm_option_value has only 'Event'.

I'm not sure how this occurred. The db started life in Civi 1.4 . I couldn't rule out that the client has changed activity type labels in Civi admin; certainly they have added new ones.

Here's a query to identify any problematic activity types:

SELECT DISTINCT(activity_type) FROM civicrm_activity_history WHERE activity_type NOT IN (SELECT label FROM civicrm_option_value WHERE option_group_id = 1)

I resolved the problem by making the civicrm_activity_history records consistent with the civicrm_option_value activity types, e.g.:

update civicrm_activity_history set activity_type = 'Email' where activity_type = 'Email Sent'

The upgrade procedure now completes but much of the activity details have been lost: most dates are now NULL and most view links for activities are broken. So maybe don't do what I did. I'm hoping that the fix for CRM-2914 will resolve this.

I've just re-done the upgrade with Civi 2.0.3 and I still have the issue with null dates and broken links on most activity records. The records in question are custom activity types that the client set up through Civi Admin. No 3rd-party apps are involved. AFAICS the approach of the upgrade script activity.sql is to throw away all the date data (
Code: [Select]
DROP `scheduled_date_time`) and then reinstate it for specific known activity types, e.g. phone calls. Would it be possible instead to retain the dates and overwrite them only where there is a known better source, e.g. by renaming the scheduled_date_time column instead of dropping it?

I'm not clear why I now have broken links for almost all the activity records. Here's an example:

Before upgrade - view link from Civi 1.8:
/civicrm/contact/view/activity?activity_id=6&action=view&selectedChild=activity&id=794&cid=2663&history=0&context=Home

After upgrade: - view link from Civi 2.0.3 for same activity:
/civicrm/contact/view/contribution?action=view&reset=1&id=&cid=3&context=home

Note the missing value for id, hence the broken link ("Sorry. A non-recoverable error has occurred. is not of the type Integer"). The "/contribution" in the URL is odd too, as the client does not use CiviContribute. The activity in question is of a custom activity type ('Newsletter Query') set up through Civi Admin.
« Last Edit: May 06, 2008, 09:12:56 am by davej »

Deepak Srivastava

  • Moderator
  • Ask me questions
  • *****
  • Posts: 677
  • Karma: 65
Re: db Upgrades and civicrm_activity_history
May 06, 2008, 10:59:07 pm
Absent Date values is definitely a problem specially for records which were there in activity table before upgrade. Filed an issue here - http://issues.civicrm.org/jira/browse/CRM-3085

Quote
After upgrade: - view link from Civi 2.0.3 for same activity:
/civicrm/contact/view/contribution?action=view&reset=1&id=&cid=3&context=home

I was able to replicate the problem for few activity links / records.
 
The problem seems to be there /w the code which generates the links - CRM/Activity/Selector/Activity.php
which expects the activities to have valid value (NOT NULL) for 'source_record_id' where activity_type_id IN (5,6,7,8), forgetting that these activity_type_ids might change after upgrade script is run.

And since you have some entries /w activity_type_id IN (5,6,7,8) and source_record_id=NULL, you see those broken links.
« Last Edit: May 07, 2008, 01:01:54 am by Deepak Srivastava »
Found this reply helpful? Contribute NOW and help us improve CiviCRM with the Make it Happen! initiative.

FatherShawn

  • Ask me questions
  • ****
  • Posts: 372
  • Karma: 25
    • C3 Design
  • CiviCRM version: 4.2.11
  • CMS version: Drupal 7.23
  • MySQL version: 5.5.32
  • PHP version: 5.3.10
Re: db Upgrades and civicrm_activity_history
May 07, 2008, 03:08:08 am
The good news for us is that we really haven't started using the activity tracking features much yet.  All we're really using is CiviEvent while my administrator edits the 100's of individual contacts to mark and link each one to "use household address"

However, I think that there is data loss with this upgrade.  I used the system to send an email to event attendees.  That activity shows under my activity history.  Shouldn't it also show under the recipient's history?  Phone calls seem to show on both.
Lead Developer, C3 Design.
Twitter: @FatherShawn

davej

  • Ask me questions
  • ****
  • Posts: 404
  • Karma: 21
Re: db Upgrades and civicrm_activity_history
May 07, 2008, 04:13:44 am
Thanks for the response, Deepak. Here are the db records from the 1.9 and 2.0 dbs for the activity that I referred to in my previous post.

1.9:
Code: [Select]
mysql> select * from civicrm19.civicrm_activity where id = 794\G
*************************** 1. row ***************************
                 id: 794
  source_contact_id: 4207
   activity_type_id: 6
target_entity_table: civicrm_contact
   target_entity_id: 2663
            subject: mailout help
scheduled_date_time: 2007-06-12 10:30:00
     duration_hours: 0
   duration_minutes: 0
           location: NULL
            details: Left a message for Robin to call back to confirm if he can come in 28th + 29th June to do the Newsleter mail out
             status: Scheduled
          parent_id: NULL

2.0.3:
Code: [Select]
mysql> select * from civicrm_activity where id = 794\G
*************************** 1. row ***************************
                id: 794
 source_contact_id: 4207
  source_record_id: NULL
  activity_type_id: 6
           subject: mailout help
activity_date_time: NULL
     due_date_time: NULL
          duration: NULL
          location: NULL
          phone_id: NULL
      phone_number: NULL
           details: Left a message for Robin to call back to confirm if he can come in 28th + 29th June to do the Newsleter mail out
         status_id: 1
       priority_id: NULL
         parent_id: NULL
           is_test: 0

davej

  • Ask me questions
  • ****
  • Posts: 404
  • Karma: 21
Re: db Upgrades and civicrm_activity_history
May 07, 2008, 05:07:21 am
There's another issue with activity history. In 1.9, there seem to be 2 civicrm_activity_history records per activity: one with entity_id = source and the other with entity_id = target and no indication in the db AFAICS of which is which, except that the record for the source seems to have been inserted first and so have a lower id. The 1.8/1.9 activity display code is clever enough to work out which is which and show the correct contacts as 'With Contact' and 'Created By'.

The 2.0.3 code doesn't handle this and instead displays two rows for a single activity.

In the target contact's Activities view, both rows have the correct target contact but one says Added By the target contact, the other says Added By the source.

In the source contact's Activities view, both rows have the correct Added By contact but one shows the target contact under 'With', the other shows the source contact.

(Use case: I came across this issue when the client asked me to set up CiviReport, initially for activity history - they need to tell their funding body how many queries of various types they have dealt with. In 1.8 I was getting 2 rows per activity. At that point I read about the new db structure for activities in 2.0 and so tried upgrading a copy of their db on a test server - their live server is still on MySQL 4.1 . I was hoping that the new structure would resolve the duplicates.)

Deepak Srivastava

  • Moderator
  • Ask me questions
  • *****
  • Posts: 677
  • Karma: 65
Re: db Upgrades and civicrm_activity_history
May 07, 2008, 05:37:59 am
Quote
However, I think that there is data loss with this upgrade.  I used the system to send an email to event attendees.  That activity shows under my activity history.  Shouldn't it also show under the recipient's history?  Phone calls seem to show on both.

This is mainly because you changed 'Email Sent' to 'Email' in your db (as you indicated in your earlier post). Had you created an option in civicrm_option_value table you would see it in recipient's history too (this i just confirmed).
Found this reply helpful? Contribute NOW and help us improve CiviCRM with the Make it Happen! initiative.

Pages: [1] 2
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Upgrading CiviCRM (Moderator: Deepak Srivastava) »
  • db Upgrades and civicrm_activity_history

This forum was archived on 2017-11-26.