A Serverless Weekend Project

22 Feb, 2018

Background

I am currently preparing for AWS Developer Associate certification, from what people have been saying there is a considerable content overlap between it and AWS Solution Architect Associate (AWS SAA) which I recently done.

Due to that, I feel a more practical / hands on project would help me with my preparation. I have been thinking for awhile about a project that is small enough that can be done on the weekend and finally thought of one.

From my AWS SAA preparation, I have compiled up a list of questions and their answers on Evernote that I review from time to time. For this toy project, I thought it would be awesome, if I can get these questions emailed to me in a regular interval, hence I won’t forget to review them as I check my emails all the time.

On a high level for this project, I would need to:

Setup a DynamoDB table for question and answer
Setup a Lambda function to pick a random question from the table
Send email through SES
Need a scheduler to trigger the lambda at a certain interval

I am so excited on building this project as:

It is actually a useful project, not a throw away weekend project. It will help my exam prep.
It gives me opportunity to work on a serverless solution
It gives me a chance to program in Golang

I managed to finish 90% of the project on the weekend, I am surprised on how easy it is to wire things up in AWS (once you know what to look for, again that SAA cert preparation is really useful).

I will outline what I have done in this post - if I can do it, you can too :)

DynamoDB

Firstly, I created a table aws-questions with questionId as the key, each record would have two fields: question and answer. Nothing too fancy, here’s an example of a record:

{
  questionId: 2,
  question: "What do you need to do when you see 'Origin policy cannot be ready at the remote resource'?",
  answer: "You need to enable CORS on the API gateway"
}

Lambda

The next part, we want to create a lambda function. I called my function sendAWSQuestions, I picked Golang as the language for the runtime.

And then, I was asked to create a role. At this stage, I stopped configuring my lambda function, opened a new tab and went to IAM section of AWS.

Role and Policies

Let’s think about the access requirements for the lambda function. The lambda needs to access DynamoDB and SES.

I could create a role for the lambda and attach the AWS pre-made DynamoDB and SES policies. However I want to practice setting up policies with minimal privilege in mind.

Here is the DynamoDB policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "dynamodb:BatchGetItem",
                "dynamodb:BatchWriteItem",
                "dynamodb:PutItem",
                "dynamodb:DeleteItem",
                "dynamodb:GetItem",
                "dynamodb:Scan",
                "dynamodb:Query",
                "dynamodb:UpdateItem"
            ],
            "Resource": "arn:aws:dynamodb:*:*:table/aws-questions"
        }
    ]
}

Here is the SES policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ses:SendEmail",
                "ses:SendTemplatedEmail",
                "ses:SendRawEmail"
            ],
            "Resource": "*"
        }
    ]
}

As you can see, the above policies are limited to certain actions and certain resource. I could have tighten up the SES policy but I reckon that will do.

Then I created a role and attached these 2 policies to it.

Back to Lambda

Ok, so now back to Lambda, I needed to write a Golang program to retrieve a record from DynamoDB and send the record via email.

The big challenge for me is how to be sure my Golang program works before I upload it to Lambda. I ended up creating 2 programs which are pretty much identical, one is the local version and the other one is the Lambda version.

The two programs are identical except, the Lambda version imports lambda package and the main function calls lambda start:

func main() {
	lambda.Start(Handler)
}

This workflow is probably sub optimal, but I don’t any better yet.

The other challenge that I face is the permission side of things, locally I use my AWS user that has admin rights, so no issue whatsoever. But on Lambda, the function has a limited permissions and I ran into permission related issues quite often and had to tweak the policies above. Here an example of permission error:

{
  "errorMessage": "Error retrieving item AccessDeniedException: User: arn:aws:sts::034077927509:assumed-role/AWS-Questions-Lambda/sendAWSQuestions is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:ap-southeast-2:034077927509:table/aws-questions\n\tstatus code: 400, request id: V4N5O5A2CKQH8872PF8OA7MNS7VV4KQNSO5AEMVJF66Q9ASUAAJG",
  "errorType": "errorString"
}

One thing that I learned quickly is, always handle error when doing things with Lambda, I cut corner with my code and assume that I won’t need to handle certain errors - that has bit me.

Here is the final code - yes, it’s ugly:

package main

import (
	"errors"
	"fmt"
	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-lambda-go/lambda"
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/awserr"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/dynamodb"
	"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
	"github.com/aws/aws-sdk-go/service/ses"
	"log"
	"math/rand"
	"strconv"
	"time"
)

// Item represent a record in aws-questions table
type Item struct {
	Question string `json:"question"`
	Answer   string `json:"answer"`
}

const (
	TableName = "aws-questions"
	Sender    = "[email protected]"
	Recipient = "[email protected]"
	Subject   = "AWS Question"
)

// Handler for AWS Lambda
func Handler(request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
	sess, err := session.NewSession(&aws.Config{Region: aws.String("ap-southeast-2")})
	svc := dynamodb.New(sess)

	// Get a random id from the table
	pickedIndex, pickIndexError := getRandomRecordID(svc)
	if pickIndexError != nil {
		fmt.Println(pickIndexError.Error())
		return events.APIGatewayProxyResponse{
			Body:       pickIndexError.Error(),
			StatusCode: 500,
		}, nil
	}

	// Get an item from the table based on the random id above
	result, err := svc.GetItem(&dynamodb.GetItemInput{
		TableName: aws.String("aws-questions"),
		Key: map[string]*dynamodb.AttributeValue{
			"questionId": {
				N: aws.String(strconv.Itoa(pickedIndex)),
			},
		},
	})

	if err != nil {
		log.Printf("Error retrieving from dynamoDB", err.Error())
		var ErrRetrievingItem = errors.New("Error retrieving item " + err.Error())
		return events.APIGatewayProxyResponse{}, ErrRetrievingItem
	}

	item := Item{}
	err = dynamodbattribute.UnmarshalMap(result.Item, &item)

	_, mailError := sendEmail(item)
	if mailError != nil {
		fmt.Println(mailError.Error())
		return events.APIGatewayProxyResponse{
			Body:       mailError.Error(),
			StatusCode: 500,
		}, nil
	}

	return events.APIGatewayProxyResponse{
		Body:       "Mail sent",
		StatusCode: 200,
	}, nil

}

func getRandomRecordID(svc *dynamodb.DynamoDB) (int, error) {
	input := &dynamodb.ScanInput{
		TableName: aws.String(TableName),
		Select:    aws.String("COUNT"),
	}
	scanResult, err := svc.Scan(input)

	if err != nil {
		log.Printf("Error getting count dynamoDB", err.Error())
		return 0, err
	}

	s1 := rand.NewSource(time.Now().UnixNano())
	r1 := rand.New(s1)
	return (r1.Intn(int(*scanResult.Count)) + 1), nil
}

func sendEmail(item Item) (bool, error) {
	// No SES service in ap-southeast-2, hence using us-east-1
	sess, err := session.NewSession(&aws.Config{Region: aws.String("us-east-1")})
	svc := ses.New(sess)
	HTMLBody := "<b>Question: </b><p>" + item.Question + "</p> <b>Answer:</b><p>" + item.Answer + "</p>"

	input := &ses.SendEmailInput{
		Destination: &ses.Destination{
			ToAddresses: []*string{
				aws.String(Recipient),
			},
		},
		Message: &ses.Message{
			Body: &ses.Body{
				Html: &ses.Content{
					Data: aws.String(HTMLBody),
				},
			},
			Subject: &ses.Content{
				Data: aws.String(Subject),
			},
		},
		Source: aws.String(Sender),
	}

	result, err := svc.SendEmail(input)
	fmt.Println(result)

	if err != nil {
		if aerr, ok := err.(awserr.Error); ok {
			switch aerr.Code() {
			case ses.ErrCodeMessageRejected:
				fmt.Println(ses.ErrCodeMessageRejected, aerr.Error())
			case ses.ErrCodeMailFromDomainNotVerifiedException:
				fmt.Println(ses.ErrCodeMailFromDomainNotVerifiedException, aerr.Error())
			case ses.ErrCodeConfigurationSetDoesNotExistException:
				fmt.Println(ses.ErrCodeConfigurationSetDoesNotExistException, aerr.Error())
			default:
				fmt.Println(aerr.Error())
			}
		} else {
			fmt.Println(err.Error())
		}
		return false, err
	}
	return true, nil
}

func main() {
	lambda.Start(Handler)
}

The complete code can be found on: aws-question-of-the-day

SES

Setup SES to receive email, you need to register and verify an email address that you own for this.

Scheduling

The final part is scheduling, I needed a cron like program to trigger my lambda function on regular interval. After some Googling, apparently I could use AWS CloudWatch for this, I was quite surprised as I thought CloudWatch is mainly for monitoring use.

The following guide helps me to set it up AWS CloudWatch Scheduled Events.

Finally

When it is all wired up - this is what I get:

“Email”

SUCCESS!!

“Success!”

I am interested to know how much this experiment will cost me, hopefully it’s nothing if everything is under free tier.

All in all, it felt good to finally scratch my itch with building a serveless solution.