/ Meteor Continuous Integration on AWS

Part 1: Setting up CodeStar for Meteor

Our goal. We want to have working Continuous Integration flow for our Meteor application. A push to our repository should eventually result in an updated application in our cloud. Besides that, we want to run our application under a custom domain using a SSL certificate (HTTPS enabled). To top it of, we need to know it if and where something went wrong during the process. Our CI flow needs to have the following things covered:

  1. A push to our repository triggers the CI flow for any given branch
  2. Before the build process, unit tests and integration tests are executed. Any failure must stop the process and report back to us showing where the failure happened
  3. The build process needs to run in the cloud on a server that is spun up on demand. We don't want it to run forever. The build server has all the specs required to successfully build a Meteor application for our final production architecture
  4. A second testing process happens after the build to run any UI and acceptance tests. Same story as the first test. We want to know it whenever and where something fails.
  5. A deployment process takes the finished and tested build and puts it into cloud servers that are continuously monitored. We wan't to know if, why and when they fail.
  6. We wan't autoscaling and loadbalancing on demand. The loadbalancers must support SSL certificates for secure connections over a custom domain

Its a lot right? Developing CI tools takes some time, but if done carefully they will save you huge amounts of time. If done in a rush they might even backfire! The practices and conventions in this guide are my years of trial and errors and included with the practices of other people.

Take note that the CI process itself will take some time as well to run all the tests, spinning up build slaves etc, but we should not care. Its the developers time that we eventually save and the human error that we are trying to diminish here! Exactly for this reason I've split up the guide into separate articles. You can finish the first article and its result will already be beneficial. If after a week you have again some extra time, you can follow the next article etc. Just don't rush this!

Why AWS CodeStar?

AWS has a new service called CodeStar. CodeStar is a Continuous Integration platform that uses the different AWS services under the hood to streamline the code, build, test and deployment phases. I found AWS CodeStar a very nice choice to automate my Meteor deployments into AWS EC2, because I have most of my tools running in AWS. I'm sharing the way I did it in the form of an extensive guide to help the rest of the Meteor community. This guide also involves some best practices when it comes to structuring your app for Continuous Integration in general.

This is part one. It covers the initial CodeStar setup for Node applications and how to extend it with a build step to run the Meteor build and Npm install commands. The follow ups cover the code part for the build commands, the deployment in Elastic Beanstalk, Test stages and setting up a domain with ssl enabled.

I'm assuming that you are familiar with AWS and have an account that allows you to access developer tools, deploy ec2 instances and configure IAM roles.

AWS CodeStar can best be compared to a managed version of Jenkins that has all the tooling needed to automate deployments on the AWS infrastructure. Its a wrapper around lower laying services of AWS like CodeCommit, CodeBuild, CodeDeploy, CodePipeline and application containers like Elastic Container Service (ECS) and Elastic Beanstalk. Although it uses these tools by default, choosing other tools is also quite easy. For example, you can replace CodeCommit with Github or Elastic Beanstalk with plain Ec2 servers.

Setting up a CodeStar project for NodeJS

To get started, navigate to the AWS CodeStar service and click "Create a new project". You will now be able to select project templates to help you get started. Our end goal is to have the whole deployment process automated from a git commit to a running production ready application. The built version of Meteor is a plain NodeJS app so this part is purely about setting up CodeStar for NodeJS. However we do take into account the versions and settings recommended by the official Meteor Guide about custom deployments.

  1. Select the NodeJS template with Elastic Beanstalk. It should look like the below image:
    AWS CodeStar Nodejs Elastic Beanstalk template

  2. Fill in a descriptive project name. Be sure that the name is unique and identifiable. All AWS resources (like EC2 servers, s3 buckets etc) for this project will include this name. If the project is for another customer, be sure to prepend it with the customer name that as is written in your invoices. This makes managing costs in AWS easier.

  3. Select a keypair This is should be the key pair that you will use to SSH into your servers. If you don't have one, go and create a one using the AWS guide on creating key pairs.

  4. Wait until the project has been provisioned During this phase AWS is creating AWS resources using existing services. Elastic Beanstalk is being setup, The AWS Developer Tools projects are created, S3 buckets are being set-up, cloudwatch alarms and logs generated and a dashboard is being prepared in CodeStar to provide an overview.

  5. Connect your tools to CodeStar. When your project has finished provisioning, a message will appear "You must connect to your project’s repository before you can start working on the code.". Click on the top right side on "Connect tools". If you use Visual Studio or Eclipse then select either of them. I'm going to use the Command Line tools since they are most common to each development environment.

  6. Follow the steps outlined in the Wizard. Just a small mention about the connection method. HTTPS is usually recommended for git, because its the most likely port to be open on your cloud's firewalls. If you have complete control, SSH might be a better choice. If you want more information on this. Please read this nice Github article about choosing between using SSH vs HTTPS for Git

You are now finished setting up the default AWS CodeStar project and have successfully connected your tools to it. Its now time to extend the project to support a build stage suitable to build Meteor projects.

Extending AWS CodePipeline with a build stage for Meteor

Lets explore our project for a moment. In your CodeStar project, on the left menu there's the Pipeline icon. Clickig it will lead you to CodePipeline showing your project's Continuous Integration stages. The first stage is your git hook. That's the trigger that starts the whole Continuous Integration pipeline whenever you commit to a certain branch. In this case a commit to your git Master branch will start the pipeline. The second stage is the deploy step that triggers elastic beanstalk to deploy and run any new code.

Lets click the edit button. Now add a stage between the CodeCommit and Elastic Beanstalk step. Name it 'BuildMeteor'. It should look like below image:
Adding Meteor build stage to AWS CodePipeline

Then click the 'action' button to configure the stage. On the right side a panel opens. We are now going to configure this stage to use the buildspec.yml in our project's codebase.

  1. Action category: build
  2. Action name: 'build-meteor'
  3. Build provider: AWS CodeBuild
  4. Select 'Create a new project'.
  5. Project name: Your project name but lowercase dash separated (slug form)
  6. Environment Image: Use an image managed by AWS CodeBuild
  7. Operating System: Ubuntu
  8. Runtime: NodeJS
  9. Version: aws/codebuild/nodejs 4.4.7 (For Meteor 1.5 and below. 6.3.1 for Meteor 1.6 and up)
  10. Build specification: Use the buildspec.yml in the source code root directory
  11. Service Role: Create a service role in your account
  12. Save build project
  13. Build artifact: select your project name from step 5 appended with SourceArtifact
  14. Output artifact: name it built-meteor-app
  15. Click add action
  16. Save pipeline actions and confirm the action.

So we've now successfully configured the build step. It will use AWS's internal EC2 servers to start a build using an Ubuntu Linux image that runs NodeJS 4x. This is nice, because our Elastic Beanstalk also runs a Linux distribution. This allows us to trigger NPM install on the build server and therefore it simplifies our deployment step with Elastic Beanstalk.

Adding IAM Service Role policies

We are just 2 steps away from finishing the project. We only need to extend 2 IAM Service Roles with some policies. The first one will grant the CodePipeline service access to the underlying services like CodeCommit, CodeBuild, CodeDeploy and S3. The second one will grant the CodeBuild service access to the CodeStar project and its resources like the S3 bucket to fetch the source code from the previous stage.

Configuring the CodePipeline Service Role

We are going to grant the CodePipeline project Service Role access to the new CodeBuild project. AWS has a pre-defined CodeStar service policy with all the permissions needed to control any needed services. Lets use that one and save ourselves a lot of time!

To do this, navigate back to the IAM Roles list and select the role that looks like this name: CodeStarWorker-{name}-CodePipeline. Where {name} should be replaced by your CodeStar project name. Click on it and then click on the blue 'Attach Policy' button. Search for the policy named AWSCodeStarServiceRole. Select it. The result should look like the below picture. Finish this step by clicking the blue 'Attach Policy' button.

Picture of result Attaching AWSCodeStarServiceRole

Configuring the CodeBuild Service Role

Search in IAM Roles for a role that is similar to the following name: code-build-{name}-service-role where name is your build project's name. Now add the AWSCodePipelineReadOnlyAccess policy. The result should look like the below image:

AWS CodePipeline Read Only policy

Now we need to add another policy. This will again be a custom one like we did for the CodePipeline Role. This time to allow CodeBuild to do 4 things:

  • Adding logs to cloudwatch
  • Listening to commits on CodeCommit
  • Pulling the source code from S3
  • Pushing a build artifact to S3

Create the new policy and call it CodeBuildDefault. Then add the following policy document:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudWatchLogsPolicy",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "CodeCommitPolicy",
            "Effect": "Allow",
            "Action": [
                "codecommit:GitPull"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "S3GetObjectPolicy",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "S3PutObjectPolicy",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Now attach this policy to your code-build-{name}-service-role.

Note that these policies are not tied to any project. They will be able to read other CodePipeline projects. On small businesses this is usually not an issue. If you don't want that, then please read about Controlling Access using AWS Policies to tailor it to your account's needs

So far the required steps in AWS to have a working build stage. In the next article about Structuring your Meteor application code for CI we are going to structure the project's code itself and include the necessary build and NPM commands to prepare your Meteor project for production deployments!

All articles of the continuous integration guide for MeteorJS