/ Meteor Continuous Integration on AWS

Part 3: Configuring Elastic Beanstalk for CI

I assume that you've gone trough the first 2 articles and have a finished build artifact (which is a tar file of your built Meteor application) in AWS S3. There should also be a failing deployment stage for Elastic Beanstalk. In this article we are going to focus on preparing this stage to run our Meteor build using Elastic Beanstalk.

If you did not yet read the previous to articles. Here's Part 1 about Setting up AWS CodeStar and Part 2 about Meteor's code structure and the buildsteps.

Why Meteor is not running with Elastic Beanstalk?

There are a number of reasons why Elastic Beanstalk is not yet able to run the build. Here's a short summary of the required obstacles to overcome:

  • Elastic Beanstalk still uses the repository code and not the build artifact.
  • The command used to start the application needs to be configured
  • The environment variables need to be adjusted in the Elastic Beanstalk Console

Before we proceed, lets recap our goal. We want to have working Continuous Integration flow for our Meteor application. A push to our repository should eventually result in an updated application in our cloud. Besides that, we want to run our application under a custom domain using a SSL certificate (HTTPS enabled). To top it of, we need to know it if and where something went wrong during the process.

To reach our above goal, we need to consider some extra points:

  • We need to work with loadbalancer and autoscaling group
  • We need to name our environments to always be unique and identifiable
  • We need to enable the right settings to allow HTTPS

Exploring Elastic Beanstalk

This chapter does not contain any steps, but instead gives you a gentle introduction into Elastic Beanstalk. It also includes some powerful practices that will help you to prevent some common mistakes and gotchas down the road.

Elastic Beanstalk is an easy to use application service. It handles the deployment, monitoring, scaling and loadbalancing of applications using other AWS services like Cloudwatch. It supports multiple platforms like Python, PHP, .net and Docker. Below is a list of the services it uses under the hood. It will give you an idea about what you would normally have to do if you decide to go manual.

  • Ec2 instances It spins up Ec2 virtual instances to run the application. This includes the configuration of EBS Volumes, naming, tagging, and allocating public IP's.
  • Ec2 autoscaling and loadbalancing. It helps you set up the scaling and loadbalancing of your application. The Elastic Beanstalk interface allows you to configure on what metrics it needs to scale
  • Cloudwatch. Used for logging and monitoring. The Elastic Beanstalk console allows you to easily set up cloudwatch triggers that increase or decrease the number of instances based on for example CPU usage or response times.
  • S3. Elastic Beanstalk uses S3 buckets to store application code in the form of a tar file. You can upload a tar file using the Elastic Beanstalk console. After the upload it will trigger the environment to download and use the new code.

Naming resources in AWS

Conventions are basically result of agreements. Reinventing the wheel on this might prove to be a bad idea. I would go for standards that exist for ages (software ages, hehe). They work almost all the time and allow you to tailor it a little bit to your own needs.

Elastic Beanstalk works by creating 'applications'. Each application can have multiple environments. How these environments are used is up to you. Over time I've noticed that some approaches work and others don't, so in that sense I'm quite opinionated in how to set up an environment using Elastic Beanstalk. If you have alternatives, please share and discuss them on Meteor's forum! I'm not saying that my approach works for each use-case, just that it works for most.

I always name the applications after the customer focused name. In our case the name of the project in CodeStar that we've created in part one. For example, this tutorial's project is called 'tutorial'. The environments in this setup are essentially so called deployment environments, meaning that they are part of the application's release cycle. A production ready application typically has 3 deployment environments: Development, Staging and Production. However there are many flavors. Again these 3 are most common. Below a short description of each environment type:

  • Local: This environment does not exist in the cloud. It runs only on the developer's machine
  • Development: This runs in the cloud and contains the merged codebase from different developers. Might want to downscale or shutdown this environment during weekends to save costs!
  • Testing (optional): This environment might suit human testers. It allows them to use a test protocol to see if any new features are working as expected. If you don't have dedicated testers or a test protocol, this environment might not be needed.
  • Staging: Pre production. This is the final step before the application will be run in production. Manual tests are done by testers and stakeholders to double check any requested features and a final go-ahead should be given. This environment contains data that matches production data as closely as possible. Use this environment for any demo's!
  • Production: This is where the production application runs. This environment should never go down. No tests are allowed to be done here!

Naming and using environments like above will prevent bugs and unstable deployments. It also creates a predictable release cycle towards your stakeholders. In Elastic Beanstalk environment names must be unique for your entire cloud! This means that you cannot call them just 'tutorial', 'development' or 'production'. I've created a solid convention over time and applied fixes for any gotchas that I've faced along the way. Its simple, but very powerful and works in most cases:

Naming convention: {customer}-{application}-{server-type}-{deployment-environment}:

  • customer is the name of your customer like in your invoice. Some developers don't have apps per customer. Just use your company's name or your nickname. You can leave out this part if you will never have any customer specific entities on your AWS account.
  • application is the name of the application. In my case 'tutorial'
  • server-type is the type of the application. I'm using this to identify if its a web server, a crawler or mongo database or whatever. I will provide a list of examples later on in an article about naming conventions in AWS, but heres a small list of examples:
    • web Servers that run Meteor or any other application facing users
    • bastion Servers that act as a bastion host
    • mongo Servers that run a mongo database
    • api Servers that run an api like a REST API or DDP API
    • worker Servers that run a generic worker process supporting the application
  • deployment-environment: production, staging, development, testing etc.

An example for our tutorial application would be to have 3 environments named like this: cloudspider-tutorial-web-dev, cloudspider-tutorial-web-sta and cloudspider-tutorial-web-production. This prevents colliding names and also results in a searchable, predictable, maintainable and scale-able cloud infrastructure! Another nice benefit is that the generated URL in your environments will have that name. In my case: http://cloudspider-tutorial-web-dev.eu-west-1.elasticbeanstalk.com.

Replacing the default environment

In the previous chapter I've explained why naming is important. However our default Elastic Beanstalk environment has been set up differently. Lets change this to match our final result. To do this, go to the AWS Elastic Beanstalk Console. We'll leave the current environment intact for now and just create our own.

  1. Create a new application. On the top right corner of the page, there is button for that.
  2. Name the application using the naming convention. In my case: cloudspider-tutorial. Optionally describe the application with more words.
  3. New environment. Click 'Create Web Server'
  4. Select platform. Preconfigured NodeJS
  5. Select environment type. Loadbalancing, Autoscaling and click 'Next'
  6. Select source. Sample application and click 'Next'
  7. Name your environment according to our conventions: In my case cloudspider-tutorial-web-dev. Optionally describe it and click 'Next'
  8. VPC or not? It depends on your own requirements. I usually have 1 VPC per customer. For the sake of this tutorial I'm just clicking 'Next'.
  9. Instance type t2.micro should be fine. Especially for small Meteor apps. Ofcourse this is up to you if you expect many users, etc. etc.
  10. EC2 Keypair Its the same as the one in Part 1 of this guide when you were setting up the CodeStar project. Its the one that enables you to SSH into your servers. More info on the AWS guide on creating key pairs.
  11. Email address. Elastic Beanstalk will register an SNS topic to this address. Whenever something happens on the environment, this email will get the alert.
  12. Health Check URL. Put a '/' (slash) in there. Its the URI that the loadbalancer will use to check if your application is still running.
  13. Click next
  14. Environment Tags. If done right, they will make your DevOps live easier, because it allows you to split invoices, filter and group things in AWS list views like Ec2's instances table. I'm using 5 tags on this screen:
  • Customer: Cloudspider
  • Application: Tutorial
  • Platform: Meteor
  • Environment: Development
  • Type: Web
  1. Permissions. Just click "Next'
  2. Summary. Just double check and then click 'Launch'

Lets just wait until the environment is Green. It should result in a sample NodeJS application being visible when you click the link above the environment.

Showing Elastic Beanstalk Green Environment NodeJS

Connecting the new environment to CodeStar

Our Elastic Beanstalk Environment is running now, but its not connected to our CodeStar project. So lets connect that first by following the below steps:

  1. Navigate to the CodePipeLine of your project
  2. Click on the grey 'Edit' button
  3. There is the third stage called 'Application' with 1 action called 'EBStack'.
  4. Lets remove that one. Why? Because the name is not descriptive enough. Setting up a new one is very easy and done within 1 minute. Click on the circle with the cross icon within the EBStack action to remove it. Note: there are 2 remove icons. Make sure that you don't remove the Application stage!
  5. Lets create a new action in the application stage. Click the pencil icon on the top right of the application stage.
  6. Click on the Action square (with the plus sign)
  7. Action category: Deploy
  8. Action name DeployMeteor
  9. Deployment provider AWS Elastic Beanstalk
  10. Application name: Select or type the Elastic Beanstalk application that you've created earlier on. Mine is cloudspider-tutorial.
  11. Environment name: The Elastic Beanstalk environment. Mine is: cloudspider-tutorial-web-dev
  12. Input Artifact: Select or type the name of the output artifact of the previous stage (Build stage). Which in our case is built-meteor-app.
  13. Click the blue 'Update' button.
  14. Click the grey 'Save pipeline changes'. (Above the Pipeline project page)

Lets just click the blue 'Release change' button. We still expect this to fail, because we need to configure some additional things in our Elastic Beanstalk environment to make it work with Meteor's bundle. However, this allows us to check if the pipeline is working as expected. The expected outcome is a red environment and a red failed MeteorDeploy stage in our pipeline.

Errors are scary, but awesome, because in this case it will tell us what went 'wrong'. The expectation is that Elastic Beanstalk will tell us that it tried to start our bundle, but that it wasn't able to. What we don't want is any other error about for example permissions. To see the error click on the 'Details' link on the failing MeteorDeploy stage. Like below image:

Image showing an AWS Pipeline stage containing an pointing to the Details link

The error looks like below image:

Image showing an Elastic Beanstalk error containing the expected error

We now know that the pipeline is working. Lets verify if it has used the correct build artifact. Navigate to the Elastic Beanstalk service index page. Click on your application name. It should show 3 links. Click on 'Application versions'. Now check if there is a version with a description that is prep-ended by built-meteor-app-. Click on the link on the Source column to download the artifact. Check if the contents represent the built meteor bundle.

This step might feel like overkill, but its best to understand if and where it goes wrong. At least you know where to find what Elastic Beanstalk tried to run and you now know for sure that up until this step everything is working as expected.

In part 4 of the Meteor & CI guide I'm going to explain how to tweak the Elastic Beanstalk environment to run Meteor bundles.

All articles of the continuous integration guide for MeteorJS