CiviCRM Community Forums (archive)

Found a set of shell and SMTP commands(see below) you can use to send a mail, but not sure how to make a shell script out of them because the SMTP commands that are executed after the connection with smtp.gmail.com has been made can't be executed using a shell script (see pictures).
$ openssl s_client -connect smtp.gmail.com:465 -crlf -ign_eof
...............

         ###Server Response
         ###Connection to the server will be established
..............
EHLO localhost
### Server Response
AUTH LOGIN
###Server Response
OpenSSLbase64username
###Server Response
OpenSSLbase64password
###Server Response
MAIL FROM: <abcdabcd@gmail.com>
###Server Response
RCPT TO: <abcd1234@yahoo.com>
###Server Response
DATA
###Server Response
Subject: Test email through smtp.google.ca with openssl
Message body.
.
###Server Response
QUIT

Also, the gmail smtp server overwrites the from address everytime and the mail gets sent from the gmail id that has been used to log in to the gmail server. However this shouldn't be a problem with the CiviCRM mail server, I think. Can we perform tests to ensure that?

And if we can't use a shell script to execute these commands then I should probably try writing a javascript/php script for it. What do you think I should do? Will the already existing codes in CiviMail be of any use here?
Another question, what are we going to do about pictures and other attachments?

By the way, this is the WebdriverIO code that log into a gmail account and takes the screenshot of the first email in the Inbox. I've only learnt to open the first mail for now, I'll learn to open any specific mail in a few days. There's a problem with this code. The waiting times don't prove sufficient sometimes and the program returns with errors. What waiting times do you think we should use?
path = require ('path')
webdriverio = require ('webdriverio')
var location
var options = {
    desiredCapabilities: {
        browserName: 'firefox'
    }
};
webdriverio
        .remote(options)
        .init()
        .url ('https://gmail.com')
        .setValue ('input[type="email"]', 'username123')
.click('#next')
.windowHandleMaximize()
.waitFor('input[type="password"]',5000)
        .setValue ('input[type="password"]', 'password123')
        .click ('#signIn')
.waitFor('input[aria-label="Search"]',20000)
.click('#:3w')
.waitFor('img[alt="In new window"]',10000)
.saveScreenshot("test")
.close();


I'm impressed that you figured out how to use s_client to establish a secure connection to an SMTP server. It's good to have enough understanding of SMTP that you could send one or two messages by hand.

In practice, of course, one wouldn't want to use that technique on a day-to-day basis for writing code -- most programming languages have a library which can handle that. In NodeJS, the "Nodemailer" library seems like a good option -- http://www.nodemailer.com/ .

And if we can't use a shell script to execute these commands then I should probably try writing a javascript/php script for it. What do you think I should do? Will the already existing codes in CiviMail be of any use here?

I would strongly vote for JS. The code which sends the test mailing over SMTP should be part of the "Renderer". Since the "Renderer" (and "Manager") are already using NodeJS for other reasons, it's more consistent to also implement this in NodeJS.

There were a couple strong/debatable assertions in there, so I'll try to defend those with arguments:

Assertion 1. The code which sends the test mailing over SMTP should be part of the "Renderer"

I've added a section at the bottom of the wiki about delivering test messages.

http://wiki.civicrm.org/confluence/display/CRM/Email+Preview+Infrastructure

Assertion 2. The Manager and Webmail Renderer should be internally consistent -- using one language (JS/NodeJS)

We have three components -- the Composer, Manager, and Renderer. CiviMail already has a composer UI (written in PHP+AngularJS), and there are a number of other open-source email projects which have composer UIs (written in PHP or Python or Ruby or JS or C# or Java). However, the Manager+Renderer components are pretty novel (ie no known open-source implementations). I think they can be useful for Civi and for other open-source projects, but that means they should be designed as essentially standalone components. As standalone components, it's best if they're "internally consistent" and focus one language because:

• One language makes it easier to attract other contributors. (A prospective contributor only needs to understand Javascript.)
• One language only requires one dependency-management tool. If a sysadmin or developer wants to download our stuff along with dependencies, he runs one command "npm install". (Rather than two commands, "npm install" and "composer install".)

Another question, what are we going to do about pictures and other attachments?

We'll have to change the protocol. There are two possible changes, either:

a) Add fields for passing the list of attachments.

b) Instead of passing separate JSON fields for "subject", "body_text", and "body_html", we could pass one big field that contains the fully-rendered email message.

We will need to deal with these, but for the moment I think we should defer and first get a full process working using the simplified model (subject/body_text/body_html).

Perhaps you could add a section to http://wiki.civicrm.org/confluence/display/CRM/Email+Preview+Infrastructure which documents general features we still need to tackle?

The waiting times don't prove sufficient sometimes and the program returns with errors. What waiting times do you think we should use?

Given that personal Internet connectivity can be flaky, I think it's OK to go as high as 30-60 seconds. However, if it's consistently taking longer than 30 seconds, then something is wrong. My guesses (in order of top-guess to worst-guess):

• The "timeout" error is not really a "timeout" error -- it may be an edge-case that we haven't recognized. For example, maybe gmail.com shows different markup 10% of the time for some reason that we don't know. The only way to identify this would be (a) to modify the code to record a log of all the pages/screens, and then (b) run the script several times, and finally (c) compare the logs to see if gmail.com appears the same each time.
• The Internet connection is really bad.
• The activity is getting throttled (by gmail.com, by an ISP, or by some corporate firewall) -- eg If you run 10 tests per minute, and if each test does a login on gmail.com, then the Gmail folks might find that obnoxious. If this is the problem, then we could impose some throttling on ourselves (eg 5 tests minute -- or whatever rates puts us below a threshold that would annoy the upstream folks) or we could optimize our logic (eg only login when NodeJS starts; don't login for each individual test message). However, I wouldn't spend time developing on this until we rule-out the simpler explanations.

Suggested tasks

I think these would be some good steps to build on what you've done so far:

• Take your current code using WebdriverIO. Extend the code to read all settings (eg the gmail username+password) from a JSON file.
• Make a git repo for your current code using WebdriverIO. Post it on github.com.
• Extend the code to also send a test mailing over SMTP (eg with nodemailer).

Ideally, one could run the code using a process like this:

## Download the code
$ git clone https://github.com/utkarshsharma/webmail-renderer
$ cd webmail-renderer
$ npm install
## Create a config file
$ cp config.json.template config.json
$ vi config.json   (<== Put in gmail credentials ==>)
## Run the renderer
$ npm main.js


Things I've done so far:
I can now send email using nodemailer from the js script. I can also open any email identified by the unique from address using webdriverio.

Minor problems that need to be addressed now:
1. smtp.gmail.com overwrites from addresses. We need a different smtp server. This is the list of servers that nodemailer supports
https://github.com/andris9/nodemailer-wellknown#supported-services
2. The security of gmail accounts needs to be reduced manually for nodemailer to be able to send emails through them.

A little more on what the Renderer script does now:
The engine that I now have, picks up the email (Subject, Text, HTML) from a json file, picks up the sending gmail account's (with security reduced manually) details from another json file, sends an email from it, then it picks up the receiving gmail account's details from yet another json file, opens it, takes the screenshot and displays it. I haven't yet uploaded it to git though. I'll upload it as soon as you it is approved.

Things to do now:
1. I want to now know what all I should do to make my midterm submission ready.
2. And I also need to start working towards building the composer and the manager. I'll appreciate some guidance on how I can do that.

The changes including the nodemailer part have been pushed to github.
I haven't yet implemented the 4 points you suggested in your last post.
I have a few doubts regarding them:

The URL should be configurable (config.json / config.json.template)

I am sorry I didn't understand what you mean by that. Do you mean I should enable write permissions for that file? How do we do that?
Edited
Got it. The URL to the file should be configurable by the user. I'll add another field in the config.json.template for the user to enter path to the email json file.

Poll the address periodically (eg once every minute) and run the preview.

Is this for the case the message hasn't reached before we start the webdriverio script?
If that is what it meant, I just have to keep looking every minute for the email from the unique email id and not proceed to the code which takes the screenshot, until I can find such an email. Am I correct?

Got it. Thank you so much for the great explanation. I'll try implementing all that in the coming few days.
Loading the email from a remote file is done.
However, phantomjs is posing some problems. But it shouldn't be a really big issue. I'll get around it in a day or two hopefully.
I haven't yet started working on extending the code to another email client. How should I go about it? One after the other or parallely (if that's possible)?

Yahoo is working as well! (single-threaded)
Now polling and PhantomJS remain 

It's great that Yahoo/Selenium and Gmail/Phantom is working. Could you 'git push' (and generally do so more regularly)?

Is there a particular reason we want to use PhantomJS?

There are two reasons I've pushed for Phantom.

The first reason is "npm" and simple installation. If you put PhantomJS in "package.json", then you can download everything in one command ("npm install") which works on basically any desktop or server. I don't know a comparatively simple/effective way to download and configure Selenium+Firefox. (Simpler installation means simpler documentation / less support / more successful installs.)

The second reason is based on some experience using similar tools in Civi's testing infrastructure. (That's a slightly different problem space; easier in that we control the entire webapp, but harder in that we have many more scripts/scenarios to test.) Anyway:

• For Javascript unit-testing, we use Node+Jasmine+Karma+PhantomJS. This is a newer/smaller suite, but so far it's performed reliably. (It starts consistently; given the same inputs, it always produces the same outputs.)
• For end-to-end/click-through testing, we use PHPUnit+Selenium+Firefox, and it's been unstable for a long time (despite several projects to try to workaround). If you leave selenium-server.jar running continuously, it breaks down every 1-4 weeks. ("Breaks down" => the daemon is running and answering requests but mostly returns errors.) If you stop/restart periodically, selenium-server.jar often hangs during the restart. (Aside: Found a possible explanation last week -- https://code.google.com/p/selenium/issues/detail?id=8237#makechanges ).

So... I'm partial to Phantom over Selenium, and it's really unfortunate that Phantom doesn't run consistently on the Yahoo Mail. But we don't want it to become a big drag on the project. Let's get a couple eyes on the Yahoo Mail script (just in case). If that doesn't turn up anything, then we could do one of:

• Try the Yahoo Mail script using Selenium+Chrome (aka "*googlechrome"). This puts it about halfway between the Selenium+Firefox and Phantom configurations -- and may help identify/narrow the problem. (eg If the inconsistency involves a Webkit-vs-Gecko issue, then we'd expect PhantomJS and Chrometo fail the same way.)
• Flag the Yahoo Mail script as experimental. (The main goal here was to ensure that the app has enough abstraction to support multiple webmail systems, and that goal is met even if the Yahoo support is unstable.)
• Just accept that the installation/administration requires Selenium and Firefox for now.