CRM-8744 breaks sending emails to SMTP servers requiring TLS & authentication

[ken]

CRM-8744 stops Civi sending emails to SMTP servers requiring TLS and authentication. I've attached a patch to fix the problem.

To reproduce on 3.4.6,
* Go to Administer CiviCRM > Global Settings > Settings - Outbound Email
* Enter credentials for an SMTP server that insists on TLS and authentication
* Click on Save & Send Test Email
* This fails with the message "SMTP server does not support authentication"

The offending code is /sites/all/modules/civicrm/packages/Net/SMTP.php at lines 602ff ...

Code: [Select]

        if ($tls && version_compare(PHP_VERSION, '5.1.0', '>=') &&
            extension_loaded('openssl') && isset($this->_esmtp['STARTTLS']) &&
            // CiviCRM: CRM-8744
            ($this->_esmtp['STARTTLS'] == true) &&
            strncasecmp($this->host, 'ssl://', 6) !== 0) {
            /* Start the TLS connection attempt. */

$this->_esmtp['STARTTLS'] is always false. It is set at line 541ff ...

Code: [Select]

        foreach ($this->_arguments as $argument) {
            $verb = strtok($argument, ' ');
            $arguments = substr($argument, strlen($verb) + 1,
                                strlen($argument) - strlen($verb) - 1);
            $this->_esmtp[$verb] = $arguments;
        }

In the case of $argument = 'STARTTLS' ...
* $verb gets set to 'STARTTLS'
* $arguments gets set to false
* so $this->_esmtp['STARTTLS'] = false

To fix this, the test should revert to isset($this->_esmtp['STARTTLS'])

In this case, 'false' does not mean 'does not support STARTTLS' but rather 'the server DOES support STARTTLS'. The isset() test is correct.

Ken

[Donald Lobo]

ken:

we borrowed that patch from here:

http://www.pear-forum.org/post-4935.html

any chance u can follow up there and other places and figure out why the differences. I've also asked lcdweb to comment

lobo

[ken]

Lobo,

I looked at that link while looking at CRM-8744. It looks like that PEAR-patch was intended to fix the very issue that it has created for me!

Looking at the debug code on that page, my guess is that the guy's server supports logging in with or without TLS. See ...

DEBUG: Recv: 250-AUTH PLAIN LOGIN CRAM-MD5
DEBUG: Recv: 250-STARTTLS

... so the PEAR auth() function doesn't use STARTTLS but uses one of the AUTH methods provided by the server in-the-clear.

The PHP substr() function only returns a string or 'false', but never 'true'., so the code as it stands cannot work.

Ken

[lcdweb]

I had suggested this patch, because we were seeing socket connection problems such that we'd only make it through about 500 of 5000 emails before it crashed -- resulting in a huge amount of pre-delivery bounces (i.e. connection-related bounces between civimail and the SMTP delivery).

I haven't traced the code -- but we are running SMTP over TLS/587 -- and I can confirm that we definitely have TLS based connections going on. (and our connection related issues completely went away -- we've had no more connection related bounces)

[ken]

@lcdweb,

Does your code for SMTP.php match the code that I quoted?

If so, I can't see how $this->_esmtp['STARTTLS'] could ever take on the value 'true'. (No other code modifies $this->_esmtp or mentions 'STARTTLS'.)

Reading the code, the only way I can explain what is happening is that your SMTP server offers both STARTTLS and AUTH verbs, and the connection is not a TLS connection (because this PEAR fix stops that) but rather one in-the-clear. (As my SMTP server only offers STARTTLS, the code fails saying authorisation is not supported. The server offers the AUTH verb after TLS is negotiated.)

As experiment always beats theory, I can't deny this is working for you, but I can't understand how it does that!

Ken

[ken]

Just to clarify my previous comment, this is what happens when I telnet to my SMTP server (my input in bold) ...

$ telnet example.com 587
Trying NNN.NNN.NNN.NNN...
Connected to example.com.
Escape character is '^]'.
220 example.com ESMTP + stunnel
EHLO localhost
250-EHLO localhost Welcome
250 STARTTLS

The PEAR-fix causes this to fail as it won't let me start a TLS session, and as no authorisation options are listed, I can't log in and send emails.

However if (as was the case for http://www.pear-forum.org/post-4935.html) the EHLO response had of been ...

250-EHLO localhost Welcome
250-AUTH PLAIN LOGIN CRAM-MD5
250 STARTTLS

... then the PEAR-fix wouldn't let me start a TLS session, but as authorisation options are listed for in-the-clear communications, I could log in and send emails in-the-clear.

[lcdweb]

I'm less certain about the patch now.

It's been a while since I did the debugging on our system, but I definitely traced it to the TTLS authorization at this point. But now if I throw some debug code to log activity, it doesn't look like the TTLS authentication is triggered within that condition statement. It reverts to the _authLogin method, which is just basic authentication. I know we're passing it to our SMTP provider over their TLS port, and as TLS authentication -- maybe the _authLogin method adapts to that.

If that's the case -- I still think there's a problem, but it must be deeper in the TTLS authentication routine. We had consistent, reproducible problems maintaining the TLS connection. And others had reported similar problems. I'll try to dig at it some more tomorrow.

[davideps]

Hello. I upgraded to CRM 4.0.7 from an early version. I was able to email out previously, but now get this error:

authentication failure [SMTP: No supported authentication methods (code: 250, response: hellskitchen.mr.itd.umich.edu Hello localhost SIZE=104857600 AUTH GSSAPI STARTTLS)]

I'm guessing this is the problem discussed in this thread. Is the patch provided above still the best way to fix this?

thank you,
-david

[ken]

David,

Your server does seem to support GSSAPI authentication in the clear, if you wanted to try that.

Assuming that you really want to use a secure connection, I'd suggest trying my patch at comment #4 (on a test server) to see if the previous functionality returns.

Ken

[zkrebs]

@lcdweb,

Does your code for SMTP.php match the code that I quoted?

If so, I can't see how $this->_esmtp['STARTTLS'] could ever take on the value 'true'. (No other code modifies $this->_esmtp or mentions 'STARTTLS'.)

Reading the code, the only way I can explain what is happening is that your SMTP server offers both STARTTLS and AUTH verbs, and the connection is not a TLS connection (because this PEAR fix stops that) but rather one in-the-clear. (As my SMTP server only offers STARTTLS, the code fails saying authorisation is not supported. The server offers the AUTH verb after TLS is negotiated.)

As experiment always beats theory, I can't deny this is working for you, but I can't understand how it does that!

Ken

I applied this patch today , because after upgrading to CiviCRM 3.4.8 my outgoing email stopped, saying my server did not support authentication. Am using TLS. Now, it "seems" to work. I am not entirely sure how or why, but am thankful I at least was able to send a test email.

[TravelingPharaoh]

I had the same problem described above after upgrading to CiviCRM 4.  The fix does work with Gmail.  Now I think we have an additional condition unnecessarily.  This is how I read the code in question

Code: [Select]

       if ($tls && version_compare(PHP_VERSION, '5.1.0', '>=') &&
            extension_loaded('openssl') && isset($this->_esmtp['STARTTLS']) &&
            // CiviCRM: CRM-8744
            (isset($this->_esmtp['STARTTLS']) ) &&
            strncasecmp($this->host, 'ssl://', 6) !== 0) {

Pseudo code is
       if TLS AND php version is greater than 5.1.0 AND OpenSSL is loaded AND STARTTLS is set AND STARTTLS is set AND the hostname has 'ssl'

Because we are using ANDs we don't need the second isset the final code should look like

Code: [Select]

[code]
       if ($tls && version_compare(PHP_VERSION, '5.1.0', '>=') &&
            extension_loaded('openssl') && isset($this->_esmtp['STARTTLS']) &&
            strncasecmp($this->host, 'ssl://', 6) !== 0) {



[bcaldwell]

Thanks TravelingPharaoh! I've just applied the code changes you suggested and it solved the problem for me.

[diegov]

I just like to report that I needed to use this patch to make Amazon SES SMTP work with CiviCRM 4.1.1 on Joomla! 2.5.2.

Thanks to ken !

[podfish]

Are you still required to use this patch, or has it been fixed?  Seems to be broken in 4.2.x still.

[podfish]

Confirm:  Commenting out

Code: [Select]

($this->_esmtp['STARTTLS'] == true) && unborks amazon's SES.  I am new here, but it really looks like this is a patch that keeps getting reapplied which attempts to repeat the same thing the isset() function above it which works JUST FINE all by itself.  Should I submit a bug, or is this going to break something in older pear/php versions?

The Podfish