Author Topic: Activity by email (Read 5084 times)

xavier · November 04, 2009, 08:45:35 am

Hi,

Reached the conclusion that one of the major reason that activities are underused is that it's hard to transfer emails that are in your inbox to civicrm.

I've seen two pgm that deals with that, but haven't found any doc. Could someone points me to a resource ? Ok, let me try to explain what I understood from the code.

bin/EmailProcessor.php
That seems to take the FROM as the source, and everyone in the TO and CC and BCC as the target ?
So the idea is that if I send an email to John Doe and CC Alice and civimemory@example.org, the pgm parses the mailbox civimemory, and would create an activity with me as the creator, with John and Alice ?

- Why isn't it using the CC/BCC "relationship" as when you create an activity with civi ?

Is there possible to forward an email to civimemory so I can do the same "storage" as I do with emails I send.

bin/Email2Activity.php
That assumes that the mails are on a maildir. Not sure what is the difference between the previous one.

Donald Lobo · November 04, 2009, 01:04:00 pm

1. We are attempting to centralize all incoming email code to EmailProcessor.php. Email2Activity is a legacy script used for a specific client (HRD)

2. this is documented here: http://wiki.civicrm.org/confluence/display/CRMDOC/Autofiling+email+activities+via+EmailProcessor

3. This started off being writtten for PHP and hence was spec'ed to what they needed (collecting all the info in one place was more important). We should model the activity similar to the send email activity (which however does not track cc/bcc which we should fix). Wanna work on it and submit a patch?

lobo

demeritcowboy · November 04, 2009, 05:51:27 pm

Just a note that PHP here means Physician Health Program (of British Columbia). The EmailProcessor is used by them all the time to do exactly what you are trying to do. Let me know if there's anything missing from the docs that Lobo pointed to and I'll try to elaborate.

xavier · November 04, 2009, 10:38:10 pm

Hi,

I have several distinct use cases, could you tell what you cover already and the other makes sense for you ?

a) I'm sending an email from my mail client as I would from civi, but I copy archive@example.org, so that's stored in civi, like if I would have sent it from the civi web interface

b) I receive an email, I want to log it, so I forward it to archive@. The creator of the activity isn't me but the sender, I got the activity(and the other that are TO or CC) assigned.

c) The same as B, but I forward a mail I sent (because I forgot to put archive@ in CC or BCC

d) Someone reply to all to an mail I sent with archive@ in CC
Not sure if I want that to be logged by default. Does it work in your experience or do you end up with a lot of crap ?
May be we should limit who can use this mail to activity to only the members of a group (say staff) ?

Lastly, how do you deal with spam and can you easily make a difference between "normal" activities and activities auto created from a mail ?

demeritcowboy · November 05, 2009, 06:36:50 am

a) I'm sending an email from my mail client as I would from civi, but I copy archive@example.org, so that's stored in civi, like if I would have sent it from the civi web interface

Yes. Bcc will also work if you don't want the recipients to see that address. In this case archive@example.org emails are most easily retrieved using a POP configuration in CiviMail setup.

b) I receive an email, I want to log it, so I forward it to archive@. The creator of the activity isn't me but the sender, I got the activity(and the other that are TO or CC) assigned.

"Forward" no. "Bounce" or "Redirect" yes, but few email clients support this. Thunderbird has a Bounce plugin, and Microsoft Exchange Server also supports this (but Outlook without Exchange Server doesn't unless you buy a third-party add-on). I remember something about squirrelmail supporting bounce but I remember having mixed results looking into that.

So in this case I would use the IMAP setup if your mail server and client support that. Each user's email account needs to be configured in CiviMail setup, but once that's done, then the user can "file" emails into Civi by moving them into the CiviMail folder (or whatever you decide to call it).

c) The same as B, but I forward a mail I sent (because I forgot to put archive@ in CC or BCC

Same as b. I never tested this scenario but I don't see why it wouldn't work.

d) Someone reply to all to an mail I sent with archive@ in CC
Not sure if I want that to be logged by default. Does it work in your experience or do you end up with a lot of crap ?
May be we should limit who can use this mail to activity to only the members of a group (say staff) ?

Lastly, how do you deal with spam and can you easily make a difference between "normal" activities and activities auto created from a mail ?

Use bcc and make it a goofy address. So far no spam that I know of, but note that at the beginning you will probably get a lot of duplicate contacts created by the processor, usually because of people's alternate emails that aren't in civi already.

xavier · November 05, 2009, 10:09:31 am

Quote from: demeritcowboy on November 05, 2009, 06:36:50 am

Yes. Bcc will also work if you don't want the recipients to see that address. In this case archive@example.org emails are most easily retrieved using a POP configuration in CiviMail setup.

Better than IMAP as well ?

Quote from: demeritcowboy on November 05, 2009, 06:36:50 am

b) I receive an email, I want to log it, so I forward it to archive@. The creator of the activity isn't me but the sender, I got the activity(and the other that are TO or CC) assigned.

"Forward" no. "Bounce" or "Redirect" yes, but few email clients support this. Thunderbird has a Bounce plugin, and Microsoft Exchange Server also supports this (but Outlook without Exchange Server doesn't unless you buy a third-party add-on). I remember something about squirrelmail supporting bounce but I remember having mixed results looking into that.

Ok, so it needs some code, but what about the principle ?

Say that if you parse a mail that is "FWD: xxx" and that the sender is in the contact db, and that no CC or BCC ? Would it make sense to then parse the "original" forwarded email and process it like if it'd be directly BCCed as in a) ?

Quote from: demeritcowboy on November 05, 2009, 06:36:50 am

So in this case I would use the IMAP setup if your mail server and client support that. Each user's email account needs to be configured in CiviMail setup, but once that's done, then the user can "file" emails into Civi by moving them into the CiviMail folder (or whatever you decide to call it).[/b]

Ok, so civicrm needs to know the password of "real" accounts. I love the idea of the folder IMAP, but for security reason, don't want to have the pwd of the mail account stored in civicrm, so can't use that solution.

Quote from: demeritcowboy on November 05, 2009, 06:36:50 am

Lastly, how do you deal with spam and can you easily make a difference between "normal" activities and activities auto created from a mail ?

Use bcc and make it a goofy address. So far no spam that I know of, but note that at the beginning you will probably get a lot of duplicate contacts created by the processor, usually because of people's alternate emails that aren't in civi already.

Don't know where you find your users, but I know mine, and if they have this address for archiving, that's for sure they will forget that it should be in the BCC, and it will be in the TO or CC.

What do you think about protecting this address, either by being able to configure the group id of authorized users (eg. staff) or limit what domain can send emails to this address (eg @example.org) ?

Also, you experience shows that you probably need to be able to find all contacts that have been created through the mail to activity gateway.

What do you think about configuring a tag ("automatically created") and automatically associate it to the contacts ?

It would be good to be able to identify as well the activities autocreated. What do you think ?

X+

demeritcowboy · November 05, 2009, 07:40:52 pm

Quote from: xavier on November 05, 2009, 10:09:31 am

Quote from: demeritcowboy on November 05, 2009, 06:36:50 am

Yes. Bcc will also work if you don't want the recipients to see that address. In this case archive@example.org emails are most easily retrieved using a POP configuration in CiviMail setup.

Better than IMAP as well ?

Not sure what you mean by better, but it doesn't have to be either/or, you can set up both.

Quote from: xavier on November 05, 2009, 10:09:31 am

Quote from: demeritcowboy on November 05, 2009, 06:36:50 am

b) I receive an email, I want to log it, so I forward it to archive@. The creator of the activity isn't me but the sender, I got the activity(and the other that are TO or CC) assigned.

"Forward" no. "Bounce" or "Redirect" yes, but few email clients support this. Thunderbird has a Bounce plugin, and Microsoft Exchange Server also supports this (but Outlook without Exchange Server doesn't unless you buy a third-party add-on). I remember something about squirrelmail supporting bounce but I remember having mixed results looking into that.

Ok, so it needs some code, but what about the principle ?

Say that if you parse a mail that is "FWD: xxx" and that the sender is in the contact db, and that no CC or BCC ? Would it make sense to then parse the "original" forwarded email and process it like if it'd be directly BCCed as in a) ?

Briefly, no. There's too many difficulties. You need the original headers to parse it properly.

Quote from: xavier on November 05, 2009, 10:09:31 am

Quote from: demeritcowboy on November 05, 2009, 06:36:50 am

So in this case I would use the IMAP setup if your mail server and client support that. Each user's email account needs to be configured in CiviMail setup, but once that's done, then the user can "file" emails into Civi by moving them into the CiviMail folder (or whatever you decide to call it).[/b]

Ok, so civicrm needs to know the password of "real" accounts. I love the idea of the folder IMAP, but for security reason, don't want to have the pwd of the mail account stored in civicrm, so can't use that solution.

My understanding is that in 3.1 these passwords will be encrypted. Maybe that will help.

Quote from: xavier on November 05, 2009, 10:09:31 am

Quote from: demeritcowboy on November 05, 2009, 06:36:50 am

Lastly, how do you deal with spam and can you easily make a difference between "normal" activities and activities auto created from a mail ?

Use bcc and make it a goofy address. So far no spam that I know of, but note that at the beginning you will probably get a lot of duplicate contacts created by the processor, usually because of people's alternate emails that aren't in civi already.

Don't know where you find your users, but I know mine, and if they have this address for archiving, that's for sure they will forget that it should be in the BCC, and it will be in the TO or CC.

What do you think about protecting this address, either by being able to configure the group id of authorized users (eg. staff) or limit what domain can send emails to this address (eg @example.org) ?

Also, you experience shows that you probably need to be able to find all contacts that have been created through the mail to activity gateway.

What do you think about configuring a tag ("automatically created") and automatically associate it to the contacts ?

It would be good to be able to identify as well the activities autocreated. What do you think ?

X+

Don't have an answer about spam problems, but regarding finding the created contacts later what we did was create a required custom field with a default value, then back-updated the existing contacts to have this value. Then going forward manually created users will have this value, and ones created by the processor will be blank, which you can locate via search builder (which is admittedly not user friendly but it does work for this).

xavier · November 15, 2009, 04:55:03 am

Hi,

First, I've modified the script so it can run directly from php-cli too (instead of having to use wget/curl) on the cron. lobo, can you tell me when I can push it when you branch 3.1 and let the trunk be 3.2 ?

Otherwise, I'd need some clarification about how to make it work both for mail to activity and bounce.

If I don't put emailtoactivity as a parameter, it automatically process the mailboxes for the bounce, and ignore the mails for activities.

Is there a way of saying what mailbox should be processed as bounce recipient, and what other is for activity ? Otherwise, don't understand how to make the two working on the same configuration. What am I missing ?

X+

xavier · November 15, 2009, 06:55:54 am

Ok,

Bit further down. I'm sending the mail to autofile@example (my mail account for that), it's properly processing, but :

1) It creates a autofile@example as a contact and put it as the with
Was expecting it to skip the autofile as a contact.

2) The parsing isn't that good and you have both the html and text version

as -ALTERNATIVE ITEM 0-
and -ALTERNATIVE ITEM 1-

(with the html part being raw html)

Is it "normal" ? Wouldn't it make sense to only keep the html msg part ?

Otherwise, is there a way to keep the message unprocessed ? Whould be easier for the testing ?
X+

Donald Lobo · November 15, 2009, 08:27:33 am

1. any specific reason it should skip autofile?

2. we need to be be able to specify in the mail setting (or some other config) whether its email2activity or bounces or both. wanna add a patch for that

3. i think it makes sense to keep both, maybe keeping it as an unprocessed message is better. However either way we'd need some way of allowing folks to see the email in a easy manner

lobo

demeritcowboy · November 15, 2009, 03:00:30 pm

Regarding ALTERNATIVE parsing, I thought this was in jira already but I guess it's just an item on our civicase phase 3 page.
It's also a problem because having the html text in the actual body causes IDS to crash on the kind of emails where people have replied several times and so the whole thread is in the email. e.g. if you try to reply to it from within civi, or use the file on case button.

xavier · November 15, 2009, 03:27:39 pm

Quote from: Donald Lobo on November 15, 2009, 08:27:33 am

1. any specific reason it should skip autofile?

Yes, autofile isn't part of the conversation, it hasn't anything assigned/sent. That's just a technical interface.

Quote from: Donald Lobo on November 15, 2009, 08:27:33 am

2. we need to be be able to specify in the mail setting (or some other config) whether its email2activity or bounces or both. wanna add a patch for that

Yep, but don't understand enough how that's supposed to work still

Can we add a "bounce" boolean column before codefreeze so that's easier to take implement in the 3.2.xxx and simply change the code ?

Quote from: Donald Lobo on November 15, 2009, 08:27:33 am

3. i think it makes sense to keep both, maybe keeping it as an unprocessed message is better. However either way we'd need some way of allowing folks to see the email in a easy manner

Not sure either. Send an alert to someone ? Not quite sure what is the case that triggered it

xavier · November 15, 2009, 03:31:50 pm

Quote from: demeritcowboy on November 15, 2009, 03:00:30 pm

Regarding ALTERNATIVE parsing, I thought this was in jira already but I guess it's just an item on our civicase phase 3 page.

I changed the code. One thing missing: display (in the view) the html result instead of the escaped <tag> . Might be that I create the activity with some flag saying the body is in txt ?

It works for mail sent. Shall I add something on the activity add to make it work and the body being html ?

Quote from: demeritcowboy on November 15, 2009, 03:00:30 pm

It's also a problem because having the html text in the actual body causes IDS to crash on the kind of emails where people have replied several times and so the whole thread is in the email. e.g. if you try to reply to it from within civi, or use the file on case button.

I'm sure being able to handle reply/forward will be a big improvment. Not sure how that's supposed to work

I'm checking with ez to see if they have something to parse forwarded or reply to mails already

Donald Lobo · November 15, 2009, 07:49:13 pm

i dont think it is smart for us to make any DB changes in 3.1 at this stage for a potential future change related to activity email tracking. its already in alpha

regarding the skipping the from email address, i'm not too sure if its worth the trouble. Basically your attaching the email to everyone on the from/to/cc list

lobo

CiviCRM Community Forums (archive)

News:

Author Topic: Activity by email (Read 5084 times)

xavier

Activity by email

Donald Lobo

Re: Activity by email

demeritcowboy

Re: Activity by email

xavier

Re: Activity by email

demeritcowboy

Re: Activity by email

xavier

Re: Activity by email

demeritcowboy

Re: Activity by email

xavier

Re: Activity by email

xavier

Activity by email: creating the contact with the autofile.

Donald Lobo

Re: Activity by email

demeritcowboy

Re: Activity by email

xavier

Re: Activity by email

xavier

Re: Activity by email

Donald Lobo

Re: Activity by email