Saturday, 20 May 2017

Email to SMS gateway with Amazon SES and SNS with Lambda (for notifications and alerts)

The purpose of this was to get a service set up that will send an SMS with only the subject of a notification  email message to a few specific, subscribed users.  In this case to those responsible for major incidents,  receiving from newrelic and solrwinds.

I'm sure there are existing services that can do this, but we already used AWS, and it would keep the billing simple.  Also, we had limited control of the format of the email, so only needed the subject line to be sent by SMS.  It seemed clear that it wouldn't be hard to do in AWS - these notes show how to do it.

As this is not something I'm likely to do a lot, I'll use the AWS console - as this is far from static, the details here may change.

First - sort your SMS limit

By default, AWS has a limit of $1 / month spend on SMS - this is very nice of them, you don't accidentally want to spend a lot, but if you want to be able to start using it, contact support (top right n the console) and get that increased immediately.  You can still specify a "soft limit" after they've increased your limit.

Set up the SNS topic and subscribe your SMS 

  • Go to the Simple Notification Service , Topics, Create new topic .
    Enter topic name and display name (the display name will appear in SMS messages) 
  • Make a note of the topic ARN. 
  • Create subscription, Protocol SMS, Endpoint is your phone number (with international prefix, e.g. +44 in the UK) 
  • You will probably now receive some SMS messages about the fact that you're subscribed.
  • You can also add an email subscription at the same time, as your SMS limit may be reached, and you still want to check it works.

Set up email reception

For myself, we already had a hosted zone set up in Route 53.  If you don't have one, it may well be a good idea to set that  up, I'm not covering that here.   If you have a hosted zone (e.g. a subdomain of your normal domain), proceed from here, otherwise you can set up the MX record for a subdomain yourself elsewhere.


  • Go to Route 53, click on hosted zones, and select the zone for which you want to receive the email.
  • Check if you already have an MX record set up (you can change the drop down from Any type to MX to show it).
  • If not, Create Record Set
  • Leave the name blank (unless you want to receive mail for a subdomain, e.g. if the main domain already has one)
  • Change the Type to MX
  • under value, enter something like:
10 inbound-smtp.eu-west-1.amazonaws.com

Processing incoming email

Now you need to set up a lambda function to handle incoming email.  You can have incoming email go directly  to SMS, but it will send everything including all headers, which is almost certainly not what you want.

We will use a lambda function to process the email, sending notification of the subject to the SNS.

Before creating a Lambda function, first create an IAM role that you'll use.  This is because the Lambda function  needs permissions to use SNS.

Create an IAM role

  • Go to IAM, click on roles 
  • Create new role 
  • Select role type, AWS Service role, AWS Lambda 
  • Skip the attach policies (i.e. click next step) 
  • Give the role a name (e.g. ses_to_sns) 
  • Create role 
  • Then find the role and edit it.  Under Permissions, go to Inline Policies and create one. 
  • Custom Policy, give it a name, and add this in (this policy is much more open than you need it, but it should work)
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction"
            ],
            "Resource": "arn:aws:lambda:*"
        },
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*"
            ]
        },
        {
            "Action": [
                "sns:*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "ses:*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Create the lambda function 

  • Go to Lamda 
  • Create a Lambda function 
  • Blank function.  Skip configure triggers (we'll do this from SES), click Next 
  • Give it a name, and copy this code into the function code (note, you have to use the correct TopicArn:
var AWS = require('aws-sdk');
AWS.config.region = 'eu-west-1';

exports.handler = function(event, context, callback) {
    console.log('forward subject matcher');

    var sesNotification = event.Records[0].ses;
    console.log("SES Notification:\n", JSON.stringify(sesNotification, null, 2));

    var sns = new AWS.SNS();

    sns.publish({
        Message: sesNotification.mail.commonHeaders.subject ,
        TopicArn: 'arn:aws:sns:xxxxxwhatever'
    },  function(err, data) {
        if (err) {
            console.log(err.stack);
            return;
        }

        console.log('push sent');
        console.log( sesNotification.mail.commonHeaders.subject);
        context.done(null, 'Function Finished!');
    });
};

Set up the email receiving rule set

  • Go to SES
  • Create rule
  • Under recipient, specify the recipient you want to use in the subdomain you set up the MX record for (e.g. notifications@aws.example.com), and add, next
  • Under "Add action", select Lambda and select the name of the lambda function you created above.
  • Do NOT select an SNS topic (that's already done in the Lambda function itself)
  • Give the rule a name and go through the rest of the process.

Test it all

That should be it - send an email to the address you configured - you should receive an SMS containing the display name of the SNS topic and the subject line of the email.

Using it for alerts and notification

You will want to change some setting for the emails you are sending.  To do this, go to the Simple Notification Service in the AWS console (SNS), and select Text Messaging (SMS) and Manage text messaging preferences.

As this is for alert messages, you will want to change the message type to transactional (this is more expensive but higher priority), and the default sender ID (which shows up in the SMS).  You can also choose to have reports saved to S3.  Also set a reasonable account spend limit (not too low, you probably don't want it to silently fail, which is what will happen if you hit your limit).

Now all you need to do is set up your alerts to include this email, and subscribe anybody who needs to receive the alerts to the SNS topic.

And hope nothing ever goes wrong that you need to be alerted of.

Moving on - linux etc

It's now been more than 18 months since I last posted anything here.  The danger is, of course, that the amount I've done in the meantime means I can never catch up.  This is just an attempt to move past that, and hopefully allow me to get some more down that I want to record.

A very brief history since my last post

My client has a large number of development projects, but when I started had only one SIT and one UAT environment. These were traditionally hosted, and very expensive. My initial task was to try and build a second sit environment in AWS.

The core portal consisted of a number of Linux servers, with mostly Java applications deployed on jboss. It took some time to appreciate the size and complexity of the system. In addition, there were a number of services running on Windows servers, but fortunately I did not, initially, have any involvement in this, and they were also not initially included in our environments. I also didn't have more than the minimum involvement in the Oracle servers behind all of this.

It was also immediately clear that the actual requirement would be for multiple dev and test environments. This means the build process needed to be automated.

The production environment would almost certainly not be migrated to public cloud in at least the medium term. That means that too much reliance on cloud native services would prevent any automation applying to production. Where there is no conflict, native aws services could be used, and hopefully we could use more in future.

I evaluated a number of solutions before deciding on Ansible as the tool of choice. 

In time, our small team took over responsibility for live support of the production web portal. In time we got to rebuild and migrate the live system to a new data centre. By this time we could do the complete build, configuration and deployment of almost 80 servers in 40 distinct roles using Ansible.

The key technologies I've now been using over the past while therefore include:
Linux
Aws
Ansible
Java applications
Apache
VMware
Windows server (also SQL and SSIS)

Hopefully more coming soon.