CiviCRM Community Forums (archive)

I've imap2soap up and running but my messages are getting sent to the junk folder. Looking at the code it is this routine

    unless ($to =~ m/^\s*(bounce|confirm|optout|reply|subscribe|unsubscribe|resubscribe)/i)
      {
        verbose "Junk: Not a CiviMail message $n -- To: $to";
        junk($imap, $msg);
        next;


imap2soap: Junk: Not a CiviMail message 1 -- To: <unsubscribe.1.1.1.7ed166ffca97b1a3e7624301a7b2c240497868ac@mydomain.com>

Unless I'm mistaken it's looking for key words in the to string using a regex at the very beginning of the string. However my $to variable starts with a <. Hmmm

If I change the regex to look anywhere in the string it proceeds to the next step and then hits this

HTTP::Message content not bytes at /usr/lib/perl5/site_perl/5.8.8/SOAP/Transport/HTTP.pm line 187

Searching this indicates that it might be a bug. I've got SOAP::Lite  0.710.05 installed. Don't really understand why it is looking around the message contents as a unsubscribe / optout message may well be empty....

Also the reply to address does not match this format

          type.domain-id.job-id.queue.id.hash-recipient=recipient-domain@yourCiviMail-mail-domain

which is stated in "What's this return path thing and why do I need it?" at http://wiki.civicrm.org/confluence/display/CRMDOC/Alternative+configuration+with+imap2soap.pl+-+no+Amavis+or+root+access. However I see the format I get in lots of other postings. Did this change sometime?

Hi Fen,

Many thanks for your support here! Most appreciated.

I've installed the new script that you've provided and am moving forward. First, there is a small bug introduced in the latest version on line 104 where $text needs to be in brackets otherwise the output is "1" rather than the actual message.

sub verbose
{
  my ($text) = @_;
  return if $opt{q};
  printf STDERR "imap2soap: $text\n";
}

However when I run the script I get this output

my_user@mydomain.org.uk [~/bin]# imap2soap.pl
imap2soap: Initializing http://www.mydomain.org.uk/administrator/components/com_civicrm/civicrm/extern/soap.php
length = 519
env = <?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><authenticate xmlns="urn:civicrm"><c-gensym3 xsi:type="xsd:string">admin</c-gensym3><c-gensym5 xsi:type="xsd:string">tdotwhnf</c-gensym5></authenticate></soap:Body></soap:Envelope>
imap2soap: Initialization complete
imap2soap: Script completed.
my_user@mydomain.org.uk [~/bin]# imap2soap.pl
imap2soap: Initializing http://www.mydomain.org.uk/administrator/components/com_civicrm/civicrm/extern/soap.php
length = 519
env = <?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><authenticate xmlns="urn:civicrm"><c-gensym3 xsi:type="xsd:string">admin</c-gensym3><c-gensym5 xsi:type="xsd:string">tdotwhnf</c-gensym5></authenticate></soap:Body></soap:Envelope>
imap2soap: Initialization complete
imap2soap: Processing message 1/20 (id 83), load average = 0
imap2soap: Junk: Not a CiviMail message 1 -- To: <webmasternn@mydomain.org.uk>
imap2soap: Processing message 2/20 (id 84), load average = 0
imap2soap: Junk: Not a CiviMail message 2 -- To: <unsubscribe.1.1.1.7ed166ffca97b1a3e7624301a7b2c240497868ac@mydomain.org.uk>
imap2soap: Processing message 3/20 (id 85), load average = 0
imap2soap: Junk: Not a CiviMail message 3 -- To: <unsubscribe.1.2.5.11a14472111fd0955da0aac29b2c923efc946126@mydomain.org.uk>
imap2soap: Processing message 4/20 (id 86), load average = 0
imap2soap: Junk: Not a CiviMail message 4 -- To: <unsubscribe.1.2.5.11a14472111fd0955da0aac29b2c923efc946126@mydomain.org.uk>
imap2soap: Processing message 5/20 (id 87), load average = 0
imap2soap: Junk: Not a CiviMail message 5 -- To: <unsubscribe.1.3.6.ac45765a56007f4030901593929a7836af23a88a@mydomain.org.uk>
imap2soap: Processing message 6/20 (id 88), load average = 0
imap2soap: Junk: Not a CiviMail message 6 -- To: <optOut.1.3.6.ac45765a56007f4030901593929a7836af23a88a@mydomain.org.uk>
imap2soap: Processing message 7/20 (id 89), load average = 0
imap2soap: Junk: No To: header in message 7
imap2soap: Processing message 8/20 (id 90), load average = 0
imap2soap: Junk: Not a CiviMail message 8 -- To: my_user@mydomain.org.uk
imap2soap: Processing message 9/20 (id 91), load average = 0
imap2soap: Junk: Not a CiviMail message 9 -- To: <test@mydomain.org.uk>
imap2soap: Processing message 10/20 (id 92), load average = 0
imap2soap: Junk: Not a CiviMail message 10 -- To: <test@mydomain.org.uk>
imap2soap: Processing message 11/20 (id 93), load average = 0
imap2soap: Junk: Not a CiviMail message 11 -- To: <test-request@mydomain.org.uk>
imap2soap: Processing message 12/20 (id 94), load average = 0
imap2soap: Junk: Not a CiviMail message 12 -- To: test-request@mydomain.org.uk
imap2soap: Processing message 13/20 (id 95), load average = 0
imap2soap: Junk: Not a CiviMail message 13 -- To: <test@mydomain.org.uk>
imap2soap: Processing message 14/20 (id 96), load average = 0
imap2soap: Junk: Not a CiviMail message 14 -- To: test-bounces@mydomain.org.uk
imap2soap: Processing message 15/20 (id 97), load average = 0
imap2soap: Junk: Not a CiviMail message 15 -- To: test-bounces@mydomain.org.uk
imap2soap: Processing message 16/20 (id 98), load average = 0
imap2soap: Junk: Not a CiviMail message 16 -- To: <test-request@mydomain.org.uk>
imap2soap: Processing message 17/20 (id 99), load average = 0
imap2soap: Junk: Not a CiviMail message 17 -- To: <cavingsec@mydomain.org.uk>
imap2soap: Processing message 18/20 (id 100), load average = 0
imap2soap: Junk: Not a CiviMail message 18 -- To: <optOut.1.4.8.3e07819a3a9a8cb3b797404cadb8e9066d93fa03@mydomain.org.uk>
imap2soap: Processing message 19/20 (id 101), load average = 0
imap2soap: Junk: Not a CiviMail message 19 -- To: <webmasternn@mydomain.org.uk>
imap2soap: Processing message 20/20 (id 102), load average = 0
imap2soap: Junk: Not a CiviMail message 20 -- To: <unsubscribe.1.1.1.7ed166ffca97b1a3e7624301a7b2c240497868ac@mydomain.org.uk>
imap2soap: Script completed.

As you can some some of the To addresses should be being picked up by the regex on line 303

unless ($to =~ m/^\s*(bounce|confirm|optout|reply|subscribe|unsubscribe|resubscribe)/i)

but I think this is looking at the beginning of the string. Since some of my To: are encapsulated in <> this causes it to bork.

I'm not a programmer or particularly good at regex so I looked into it and figured the ^ is used to match from the beginning of the string. Taking the ^ out through this

imap2soap: Processing message 1/20 (id 103), load average = 0
imap2soap: Junk: Not a CiviMail message 1 -- To: <webmasternn@mydomain.org.uk>
imap2soap: Processing message 2/20 (id 104), load average = 0
length = 702
env = <?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><mailer_event_unsubscribe xmlns="urn:civicrm"><c-gensym8 xsi:type="xsd:string">5e7927500227ce715bea4e75797fd278c9f98525</c-gensym8><c-gensym10 xsi:type="xsd:int">1</c-gensym10><c-gensym12 xsi:type="xsd:int">1</c-gensym12><c-gensym14 xsi:type="xsd:string">7ed166ffca97b1a3e7624301a7b2c240497868ac</c-gensym14></mailer_event_unsubscribe></soap:Body></soap:Envelope>
HTTP::Message content not bytes at /usr/lib/perl5/site_perl/5.8.8/SOAP/Transport/HTTP.pm line 190

Perl 5.8.8 is the latest supported by cPanel

The header of the msg it is failing at is

Content-Language:       en-gb
Content-Type:     multipart/alternative; boundary="----=_NextPart_000_008C_01C8AF80.96823710"
Date:     Tue, 6 May 2008 13:53:39 +0100 [06/05/08 13:53:39 BST]
Delivery-date:     Tue, 06 May 2008 13:53:40 +0100
Envelope-to:     unsubscribe.1.1.1.7ed166ffca97b1a3e7624301a7b2c240497868ac@mydomain.org.uk
From:     Test User <my@email.com>
MIME-Version:     1.0
Message-ID:     <008b01c8af78$34bdcf10$9e396d30$@com>
Received:     

    * from fhw-relay07.plus.net ([212.159.14.215]) by host.mydomain.org.uk with esmtps (TLSv1:AES256-SHA:256) (Exim 4.68) (envelope-from <my@email.com>) id 1JtMfs-0004Z1-6F for unsubscribe.1.1.1.7ed166ffca97b1a3e7624301a7b2c240497868ac@mydomain.org.uk; Tue, 06 May 2008 13:53:40 +0100
    * from [212.159.47.84] (helo=SpeleoPC) by fhw-relay07.plus.net with esmtp (Exim) id 1JtMfr-0005RT-TB for unsubscribe.1.1.1.7ed166ffca97b1a3e7624301a7b2c240497868ac@mydomain.org.uk; Tue, 06 May 2008 13:53:40 +0100

Return-path:     <my@email.com>
Subject:     [No Subject]
To:     unsubscribe.1.1.1.7ed166ffca97b1a3e7624301a7b2c240497868ac@mydomain.org.uk
X-Mailer:     Microsoft Office Outlook 12.0
X-Plusnet-Relay:     5fc1490f1e2c10e3430abe61c0af4713
X-Spam-Bar:     
X-Spam-Flag:     NO
X-Spam-Score:     
X-Spam-Status:     No, score=
thread-index:     AciveDSA87efUxidSaGj9gwFBA5ksA==

Hope this makes sense.

Henry

Hi Henry -

A couple things:

1) I've just uploaded a new version of imap2soap.pl that allows an optional '<' at the beginning of the To string - it's better than removing the '^' from the regular expression

2) I assume the following is all on one line, otherwise it would not be an RFC 822-compliant mail message (though that probably dates my - I suspect RFC 822 has been superseded...):
Received:     

    * from fhw-relay07.plus.net ([212.159.14.215]) by host.mydomain.org.uk with esmtps (TLSv1:AES256-SHA:256) (Exim 4.68) (envelope-from <my@email.com>) id 1JtMfs-0004Z1-6F for unsubscribe.1.1.1.7ed166ffca97b1a3e7624301a7b2c240497868ac@mydomain.org.uk; Tue, 06 May 2008 13:53:40 +0100
    * from [212.159.47.84] (helo=SpeleoPC) by fhw-relay07.plus.net with esmtp (Exim) id 1JtMfr-0005RT-TB for unsubscribe.1.1.1.7ed166ffca97b1a3e7624301a7b2c240497868ac@mydomain.org.uk; Tue, 06 May 2008 13:53:40 +0100
3) according to this thread <http://www.nntp.perl.org/group/perl.libwww/2008/04/msg7162.html> it appears that libwww-perl-5.811 fixes a problem with Unicode strings in messages - could this be an issue here?

4) I suspect (but am not certain without further investigation) that HTTP::Message is being invoked by line #335:
my $body = SOAP::Data->type(string =>$imap->body_string($msg)); - perhaps this need some additional error checking...

Perhaps we'll have to see is there's a way to handle Unicode strings (assuming this is the problem).

If you can update to libwww-perl-5.811, please try this and let me know how it goes (though I notice I'm running older versions on my machines...).

Hi Fen,

Progress!!! loaded up your latest imap2soap.pl and that sorted the "<" issue. Was still borking with a "HTTP::Message content not bytes at /usr/lib/perl5/site_perl/5.8.8/SOAP/Transport/HTTP.pm line 187" error message but it was calling the error from somewhere else. Some google research suggested just commenting out the offending call statement so, what the heck, lets try that. a quick grep found the offending file.

Line 114 of /usr/lib/perl5/site_perl/5.8.8/HTTP/Message.pm got the treatment.

sub _set_content {
    my $self = $_[0];
    if (_is_utf8($_[1])) {
#        Carp::croak("HTTP::Message content not bytes")
    }
    if (!ref($_[1]) && ref($self->{_content}) eq "SCALAR") {
${$self->{_content}} = $_[1];

since it is an erronous mssage and since it now works I'm happy with not doing whatever it was doing.

All seems to be running now.

BTW perl 5.8.11 was not really an option for me as I've read to many posts advising against not using the cPanel supported version and I'm not clever enough to run two versions.

Many thanks for your assistance here!!

Thanks - this is finally incorporated into the trunk - grab your new copy at https://svn.civicactions.net/repos/civicrm/scripts/imap2soap.pl