How to specify the recipients for a new mailing, and initiate send?

[rivimey]

Hi, I have been able so far to create a new civi Mailing object and populate it, but confusingly I can't see a parameter in that to specify the mail destination group.

For context, I am dealing with Civi using pure REST api from a remote server. I have a solution to getting a custom template onto the server; the new problem is setting a schedule and delivery group, and initiating the send. I am using the python-civicrm library from github as the intermediary on the client.

I presume send happens as a result of setting the schedule -- i.e. I don't need an API call to say 'send mailing'?  Is setting 'sheduled date' == 'now' safe or should I set a date of 'now + 1min' or similar?

So that leaves setting the delivery group. We already have groups defined, and I want to specify the group by name (and preferably be able to verify in advance that a group name is a valid destination, perhaps by doing a group name -> id lookup).

Can anyone offer pointers?

[mfb]

I have the same question.

I passed in groups[include][]=866 and this seems to have added group 866 to the mailing. Yay! However, the mailing has zero recipients.

If I re-use this mailing thru the web UI, then the group is there and populates the recipients correctly.

[mfb]

Ok I figured out my trouble. The API user needed the "CiviCRM: view all contacts" permission. Also, I had to manually turn off url_tracking parameter, as I had fatal DB errors with too-long URLs.

[mfb]

I find something very strange which must be a bug in Civi 4.5: If I set the mailing scheduled_date parameter, two jobs are created and the mailing is sent to each recipient twice!

[rivimey]

I find something very strange which must be a bug in Civi 4.5: If I set the mailing scheduled_date parameter, two jobs are created and the mailing is sent to each recipient twice!

It sounds like it could be related to the issue I had a while ago: when you create a new Mailing, there are two places where a schedule can be attached -- with in the Mailing code and in the caller of that code, where a chained mailingjob create is automatically added. I suspect that your double is because one job is created in the Mailing code, and then a chained create  causes a second job to be created.

Personally, I regard the automatic addition of a chained mailing job to be a bug in itself, given that you can do this yourself if that is what you wanted.

[rivimey]

I have the same question.

I passed in groups[include][]=866 and this seems to have added group 866 to the mailing. Yay! However, the mailing has zero recipients.

Thanks for the thought.

Are you using the REST api or something else... I'm not sure where groups[] is...

Regards
Ruth

[totten]

I ran into some similar issues when using the API for the new Angular-based CiviMail UI in v4.6 - it felt a little quirky. One can use Firebug to inspect the AJAX calls for a live example of using the API. For a full list of fields used by the new UI, check the client-side code for "crmMailingMgr":

https://github.com/civicrm/civicrm-core/blob/4.6.alpha4/js/angular-crmMailing/services.js#L153
https://github.com/civicrm/civicrm-core/blob/4.6.alpha4/js/angular-crmMailing/services.js#L274
https://github.com/civicrm/civicrm-core/blob/4.6.alpha4/js/angular-crmMailing/services.js#L305
  (This uses both Mailing.create for saving drafts and Mailing.submit for final submission. Mailing.submit is new in 4.6; it's just a thin wrapper around Mailing.create which copes with some permissioning edge-cases. If all the users are administrative or if all the flow-control is done server-side, then Mailing.create will be fine.)

I presume send happens as a result of setting the schedule -- i.e. I don't need an API call to say 'send mailing'?  Is setting 'sheduled date' == 'now' safe or should I set a date of 'now + 1min' or similar?

This is correct. FWIW, there should be no real difference between 'now' and 'now+1min'. In either case, the mailing is only scheduled. The agent which sends is driven via cron, so it won't actually go out until the next cron run, and the cron schedule is configured by the site-admin.

I passed in groups[include][]=866 and this seems to have added group 866 to the mailing. Yay! However, the mailing has zero recipients.

The mailing BAO won't automatically compute the full list of recipients until you set the "scheduled_date". Note lines 1646 and 1658 (which are largely unchanged in 4.6):

https://github.com/civicrm/civicrm-core/blob/4.5.5/CRM/Mailing/BAO/Mailing.php#L1646

It sounds like it could be related to the issue I had a while ago: when you create a new Mailing, there are two places where a schedule can be attached -- with in the Mailing code and in the caller of that code, where a chained mailingjob create is automatically added. I suspect that your double is because one job is created in the Mailing code, and then a chained create  causes a second job to be created.

Personally, I regard the automatic addition of a chained mailing job to be a bug in itself, given that you can do this yourself if that is what you wanted.

That definitely confused me, too. I found it easiest to opt-out of the auto-API-chained job by passing an extra parameter {'api.mailing_job.create': 0}; instead, I relied only on the "scheduled_date". This worked better because computing recipients is linked to 'scheduled_date".

This is a bit of a shame. At a high-level, I appreciate the idea of using API chaining to manage jobs and recipients -- it gives more flexibility to the API consumer. However, this particular implementation doesn't really get "there" and just invites bugs. AFAICS, the only way to get "there" is to either break backward compatibility or write messy code.

Are you using the REST api or something else... I'm not sure where groups[] is...

It took me a while to trace that one. There's actually a series of related parameters which are all undeclared and all managed in the BAO: groups[include], groups[exclude], mailings[include], mailings[exclude].

https://github.com/civicrm/civicrm-core/blob/4.5.5/CRM/Mailing/BAO/Mailing.php#L1608
https://github.com/civicrm/civicrm-core/blob/4.6.alpha4/CRM/Mailing/BAO/Mailing.php#L1682

FWIW, the behavior of 'groups' and 'mailings' seemed to be a bit quirky when performing multiple save/load cycle, so I changed the implementation a bit in 4.6. (That won't matter if you're doing an integration with send+forget behavior in 4.5; however, if you're doing an interactive UI like 4.6's, then it's a problem.)

[Chris Burgess]

The auto-addition of the mailing job was just mirroring what we needed to get the emails out the door for the client that we added Mailing API for. I agree it seems like extra work to have to add the mailing job ... probably cleanest API would be if we can create Mailing objects via API, and CiviCRM automatically handles whether Mailing Job things need creation based on whether the Mailing is scheduled for delivery (and any other considerations).

EDIT: 2012 thread where Mailing API was introduced: http://forum.civicrm.org/index.php?topic=24075.0

[rivimey]

Hi

I have identified why my attempt to use groups/include doesn't work. The URL encoding of the dict I set up results in 'groups=include' which isn't quite right. I'm using the Python requests module for this, though I don't think that's the problem as AFAIK HTTP POST params are single-level.

Is there a syntax like groups_include I could use.

[totten]

Hmm, I believe the REST interface relies on PHP's default parsing mechanism to convert a parameter string like "entity=foo&action=bar&whiz=bang" into an array-structure. I don't know its rules very well, so I just avoid it and send JSON whenever possible. (The frontend AJAX API will accept a json blob like 'entity=MyEntity&action=MyAction&json={"first_name":"Alice"}', but I'm not sure if the backend REST API will.)

You can probably figure out how to get a URL string which produces the desired array-structure by experimenting with variations on this command:

php -r 'parse_str("subject=New+mailing&groups[include][]=gid_1&groups[include][]=gid2", $args);print_r($args);'

(Note: Change print_r() to var_dump() to get more exacting details. print_r() is a bit easier to read.)

(Incidentally, the above command is my second guess, and I think that's producing the desired arrays, but I haven't tested.)

[Chris Burgess]

I recall running into similar issues working on CiviCRM - NationBuilder sync from Python for a client, and the solution was to submit JSON as totten recommends.

[totten]

This PR includes a commit to remove the api.mailing_job.create default: https://github.com/civicrm/civicrm-core/pull/5121 . I'll try to update "API changes" after it's merged.

[rivimey]

Thanks for your help, guys.

I have been able to use json as suggested and it works fine.

I have found a problem within the python civicrm lib I'm using: it specifically excluded the 'json' key. I've submitted a bug report to maintainer tallus:
    https://github.com/tallus/python-civicrm/issues/16

As indicated in that report, there is a niggle it would be nice to clarify. In my testing, having fixed the 'json' key exclude, I tried to send all parameters using the json key, but the request was rejected because 'created_id', 'name' and 'subject' -- the required keys for Mailing -- weren't set. I infer that only http params are checked when looking for the required params, although all (http + json) get processed.

Would it be possible to modify the code so that all-json calls were possible? The alternative solutions are either to only json-encode params with an array value, or that callers need to know which params are required and which are optional, which is unfortunate.

In case it helps people, the call is made using this code:

    params = {
        u'name': template.name,
        u'subject': template.subject,
        u'created_id': template.creator_id,
        u'api.mailing_job.create': 0
    }

    # parameters that are json-encoded into the request.
    config = {
        u'from_email': template.from_email,
        u'from_name': template.from_name,
        u'body_html': template.html,
        u'body_text': template.plain,
        u'url_tracking': u'1',
    }

    if enable_mailingjob and len(groupids) > 0:
        # these cannot be in the http parameters.
        config[u'groups'] = {u'include': groupids,
                             u'exclude': []}
        config[u'scheduled_date'] = u'now'

    params[u'json'] = json.dumps(config)

    try:
        results = civicrm.create(u'Mailing', **params)

[rivimey]

Sorry to bump this, but I'm having more problems in trying to do this.

Essentially, I can create a scheduled mailing with the appropriate recipients, but if let it actually send the mail, it is sent to each recipient twice, using a different message-id for each.

More info: CiviCRM 4.4.11. Cron is scheduling mails via cli.php and is being run every 5 minutes. I can see that the duplicate emails have different message-ids, and the sendmail log strongly suggests that sendmail is being called twice (rather than sending one input twice). The recipient group in my test had 2 recipient email addresses.

I'm using the code in the previous post to send the request, via REST, except that I'm not setting u'api.mailing_job.create': 0 as this didn't seem to work when just creating a template (no schedule): it is created with a mailingjob and thus in the scheduled state, but with no recipients, which makes it useless. Instead, I am using:

    # parameters that will be put in the http request.
    params = {
        u'name': template.name,
        u'subject': template.subject,
        u'created_id': template.creator_id,
    }

    # parameters that are json-encoded into the request.
    config = {
        u'from_email': template.from_email,
        u'from_name': template.from_name,
        u'body_html': template.html,
        u'body_text': template.plain,
        u'url_tracking': u'1',
    }

    if enable_mailingjob and len(groupids) > 0:
        # these cannot be in the http parameters.
        config[u'groups'] = {u'include': groupids,
                             u'exclude': []}
        config[u'scheduled_date'] = u'now'

    params[u'json'] = json.dumps(config)

    try:
        results = civicrm.create(u'Mailing', **params)

        infomsg(settings, u'Template Created on:', results[0][u'created_date'])
        infomsg(settings, u'Template Scheduled for:', results[0][u'scheduled_date'])

        # CiviCRM creates a MailingJob record for us, which is not
        # always wanted: this code deletes it again.
        the_mailingjob = results[0]['api.mailing_job.create']['values']
        if (not enable_mailingjob) and len(the_mailingjob) == 1:
            delete_mailingjob(settings, civicrm, the_mailingjob[0]['id'])

        return True

    except CivicrmError as e:
        print(u'Mail template creation failed: ' + e.message)
        return False



My next step is to try increasing the cron run interval, in case the issue documented in https://issues.civicrm.org/jira/browse/CRM-12856 is causing the problem. Other suggestions very much appreciated.

Ruth

[rivimey]

Just wondering... could this be because of the following sequence:

1. Request comes in with scheduled_date set
2. CiviMail prepares incoming template, notices scheduled_date, creates Mailing job for it. [Mailing.php:1608]
3. As the (same) request processing completes, the shim code adds a chained call (api/v3/Mailing.php:66) that requesting the creation of a second Mailing job, which is then created via API chaining.

Now we have 2 MJs, populated with the same template and data. Both get executed. I will say, though, that only 1 mailing job is reported to the REST caller. I'm not sure if this makes the above sequence impossible or just that Civi isn't set up to consider more than one MJ per call.

I have tried setting api.mailing_job.create = 0 in the request, but this seems to be ignored (in the sense that for a create-but-not-scheduled call sequence, I still get a mailingjob created. Does API chaining work through REST in call cases? Is the problem that api.mailing_job.create is beaten by api.mailing_job.create/api.default when called through REST?

IMO the line at api/v3/Mailing.php:66 should be deleted. It is not helpful at all.
  $params['api.mailing_job.create']['api.default'] = 1;

When the user creates a scheduled mailing you get the job anyway, and if you don't set a scheduled date, having a job attached makes the template unusable via the GUI because the jobs is considered complete.