/ Meteor Continuous Integration on AWS

Part 2: Meteor's code structure and build commands

I assume that you are using the AWS Developer Tools for Continuous Integration with Meteor on AWS. In the first article we've set up a CodeStar project and included a build stage for our Meteor project. We are now going to structure the code and include the build commands to build Meteor as recommended in the Meteor Guide about custom deployments.

Before we get started, a little bit of naming. I'm referring to the Meteor part as the 'source'. Its the part that contains only unbuilt application. With 'CI Code' I'm referring to the Continuous Integration code that runs tests, builds the source code and deploys it onto a server. Then we have a thing called the 'build' which is basically the application as it is in deployed state (after being built). 'repository' or 'repo' is the source code combined with the CI code as stored into our git repository. This does not include the build! Then there is the 'project' which covers everything from code up until the infrastructure on AWS where it runs and gets build.

Its important to have naming conventions about common things. Aligning them from the start will prove beneficial as soon as your team grows, but also removes the overhead of having to explain every word that makes our developers and domain jargon.

Splitting Source code from CI Code

A good practice is to always keep source code separate from CI code. This allows us to easily change infrastructure or environment without changing the application. To do that its a common practice to put the source code in a subfolder called src. All non-application code resides in the project's root. This makes it easier for CI code to reference the source code and prevents a need for cherry-picking files that should be built.

  1. Create a project using your preferred IDE based on the cloned repository. I'm using PHPStorm. This means clicking on 'file', open and then selecting the folder of my cloned repository (default clone means that the name is the same as the repo).

  2. Remove the package.json and app.js files from the project root as well as the public/ folder. Its boilerplate code from CodeStar. Our approach is a little bit different.

  3. Run meteor create src in your project root. This will create a meteor application in a folder called src.

  4. Configure your IDE to start the Meteor project. Make sure that the root is the project's root and not just the src folder. Also make sure that your IDE starts meteor from the src folder. For Webstorm and PHPStorm there are plugins available to easily run Meteor. You can find a comprehensive guide about it on the Jetbrains Meteor configuration guide. Eventually the configuration boils down to what is shown on the below picture:
    Configuring PHPStorm to run meteor

  5. Rule out any environment specific files using .gitignore We want to prevent environment specific code to end up in our repository at all costs! Create a .gitignore file in your project root and fill in any environment specific files like node_modules and .idea. If your project has any pem files or env var files. Rule them out as-well. Never put any env vars in your codebase! If you do need them in the same file, rule them out using .gitignore.

  6. Create a buildspec.yml file. This file will contain any build information to build the Meteor application using AWS CodeBuild. We are going to fill this file in the next chapter.

Right now, your project folder should have at least the following file structure:

.ebextensions/
src/
src/.meteor
src/package.json
src/client/
src/client/main.js
src/client/main.html
src/client/main.css
src/server/main.js
src/node_modules/*
src/.gitignore
.gitignore
buildspec.yml
README.md

The buildspec.yml

The buildspec.yml is used by AWS CodeBuild. It allows the developer to issue commands that install, build and prepare the application for deployment. In our case we are dealing with Meteor specifically. This means that we need to run the meteor build at some point. After that we need to run the cd progams/server && npm install --production command as described in the Meteor Guide about custom deployment. The below example does that:

version: 0.2

phases:
  install:
    commands:
      - echo Installing Meteor
      - curl https://install.meteor.com/ | sh
  pre_build:
    commands:
      - echo Displaying folder structure
      - ls
  build:
    commands:
      - echo Build started on `date`
      - cd src && meteor build .. --allow-superuser --directory
      - cd ../bundle/programs/server && npm install --production
  post_build:
    commands:
      - cd ../../../
      - echo Build completed on `date`
artifacts:
  base-directory: bundle # Bundle contains the built code
  files:
    - ./**/* # Include all files 

Take note that this file does not contain any environment specific information. This is important, because now there is no possibility for environment related bugs when it comes to the CI code. Another detail is the --allow-superuser tag. AWS CodeBuild's build servers run the commands with the root user, meaning that we need to tell Meteor that running it as root is fine and deliberate.

Testing the build process

You should be able to build locally using the exact same steps. The goal of the local build process is to create a local directory containing the built version of the app. This app can then be tested using integration or acceptance tests. And its good to see if it actually builds correctly right? On our Continuous Integration process in AWS CodeStar its also the code that will eventually run on the servers.

You can test the build steps by simply executing the commands in the buildspec.yml files.

  1. Open a command line. (or git bash would be ideal in Windows)
  2. Execute: cd src && meteor build .. --allow-superuser --directory to build the Meteor app
  3. Execute: cd ../bundle/programs/server && npm install --production to install any NPM dependencies

If things went well, it should produce a built version of your Meteor application in a folder called 'bundle'. Although you need the bundle for testing, you don't want to push that to your git repository, since your local machine is likely different then the one that runs the finished application in the cloud! Just to be sure: This means that you should ignore it in your .gitignore file!

Now navigate to your bundle dir and try to run the plain nodejs application. Keep in mind that Meteor 1.5 and lower runs on Node 4x. It will not work in Node 6 or higher. Meteor 1.6 will support Node 6x, but is at the time of writing not yet released. So go ahead and start Meteor using the below command from your project's root:

cd bundle && node main.js

If your application starts, you are ready to deploy!

Commit, build and deploy to AWS

Its actually quite easy from this point on, because you've already connected your tools earlier. Just commit your code and push it to your repository. To do it, use the following commands in your project root:

git add .
git commit -m "Included Meteor and buildspec.yml"
git push

If the setup was successful, the pipeline will be triggered starting with CodeCommit pushing it into S3. The build step will then be triggered and create an so called artifact which is put into S3. The result is a green Source stage and a green Build stage.

I did not yet cover the last stage. Its very likely going to fail. I will cover that in the next article.

Lets double check our S3 bucket to see if the outcome is what we expect:

  1. Go to S3 and search for a bucket named like this: aws-codestar-{region}-{account_id}-{project_name}-pipeline.
  2. Navigate to a folder named {project_name}-Pipeline
  3. Navigate to a folder named built-mete
  4. Download the tar file and check the contents for the following files and directories:
  • programs/
  • server/
  • .node_version.txt
  • main.js
  • README
  • star.json

The next article will be about Configuring the deployment stage in AWS with Elastic Beanstalk and CodeStar.

All articles of the continuous integration guide for MeteorJS