Streamlined Continuous Integration Using AWS Code Build and Code Pipeline

8 min readJun 18, 2024

Introduction

In modern software development, Continuous Integration (CI) plays a crucial role in automating build and test processes, ensuring efficient code integration and quality assurance.

This project explores CI practices using AWS CodeBuild and AWS CodePipeline to automate the build, test, and validation of code changes.

This article focuses exclusively on the CI aspect. For Continuous Deployment (CD), please refer to the documentation here.

Feel free to connect with me on LinkedIn to discuss this post, or ask any questions.

Tools and Technologies Used

This project utilizes a range of tools and technologies to streamline Continuous Integration (CI) with AWS CodeBuild and AWS CodePipeline:

AWS CodeBuild: AWS’s fully managed build service for compiling source code, running tests, and producing software packages.
AWS CodePipeline: A continuous integration and continuous delivery service that orchestrates workflows for building, testing, and deploying code changes.
GitHub: A version control repository to store and manage source code, triggering automated builds in AWS CodeBuild upon code commits.
Docker: A platform for developing, shipping, and running applications inside containers, providing consistency across development, testing, and deployment environments.
Docker Hub: A cloud-based repository for Docker container images used in the application.
AWS Systems Manager: Facilitates secure storage and management of Docker Hub credentials using Parameter Store, ensuring sensitive information is stored centrally and accessed securely.

These tools and technologies together automate build and test processes, ensure efficient integration of code changes, and enhance overall development productivity.

Prerequisites

Detailed steps for setting up each prerequisite will be provided in the documentation.

AWS Account:

GitHub Repository:

Have a GitHub repository where your application source code is stored. This repository will trigger automated builds in AWS CodeBuild upon code commits.

AWS IAM Role:

Create an IAM role with necessary permissions to allow AWS CodePipeline and AWS CodeBuild to access AWS resources securely.

AWS CLI:

Install and configure the AWS Command Line Interface (CLI) to interact with AWS services from your terminal.

Docker Hub Account (Optional):

If using Docker containers, have a Docker Hub account to store and manage Docker images used in your application.

Access to AWS Systems Manager (Optional):

Ensure access to AWS Systems Manager Parameter Store if storing sensitive credentials like Docker Hub credentials securely.

Setting Up Git Repository

Fork the GitHub Repository: Fork the GitHub repository I’ve prepared for this demo. You can find it here.

Create IAM Role and Add Permissions

Navigate to the AWS IAM Console.
In the left-hand menu, click on “Roles” under “Access management”.
Click on the “Create role” button.
Select the service that will use this role, choose “AWS service” .

Select “CodeBuild” for the service.

Search for and select the following policies to attach to the role: AmazonEC2FullAccess and AmazonSSMFullAccess.

Provide a name and description for the role, then click “Create role”.

Optionally, add tags to your IAM role for better organization and management.
Review everything then click “Create role”.

Configure AWS Systems Manager Parameter Store

To securely manage and access Docker credentials such as username, password, and Docker registry URL within AWS environments, the AWS Systems Manager Parameter Store provides a robust solution.

By storing these sensitive values as SecureString parameters, AWS ensures encryption using AWS Key Management Service (KMS), maintaining the confidentiality and integrity of your credentials.

This setup is crucial for integrating Docker-based workflows with AWS services like AWS CodeBuild and AWS CodePipeline, enabling seamless and secure CI/CD pipelines.

Navigate to the AWS Systems Manager Console.
Click on “Parameter Store” in the left-hand menu
Click on the “Create parameter” button.
Enter parameter details: Enter a name for your parameter, such as /docker/username, /docker/password, and /docker/registry-url.
Optionally, provide a description to document the purpose of the parameter.

Choose “SecureString” as the parameter type. This encrypts the parameter value with an AWS Key Management Service (KMS) key, ensuring secure storage.
Optionally, you can choose a specific AWS KMS key to encrypt the parameter value. AWS manages the default KMS key if none is specified.
Create the value
Click on “Create parameter” to save.

The Docker credentials are securely stored in the AWS Systems Manager Parameter Store.

AWS CodeBuild

Navigate to the AWS CodeBuild Console.
Click on “Create build project” button.
Enter a name for your CodeBuild project.

Choose the source provider where your build will fetch the code. This could be AWS CodeCommit, GitHub, Bitbucket, or Amazon S3.
Specify the branch, tag, or revision ID for the source code version.

Define the build environment with details such as operating system, runtime version, and Docker configuration if applicable.

Choose an “existing service role.”
From the dropdown menu, select the IAM role you created in the previous steps. This role should have the necessary permissions, such as SSM Full Access, to access the Docker credentials stored in AWS Systems Manager and any other required resources.
Ensure you enable the privileged box.

Choose between using a buildspec file stored in your source code repository or providing a buildspec directly in the console.

Buildspec file

Copy and paste this buildspec file.

version: 0.2

env:
  parameter-store:
    DOCKER_REGISTRY_USERNAME: /myapp/docker-credentials/username
    DOCKER_REGISTRY_PASSWORD: /myapp/docker-credentials/password
    DOCKER_REGISTRY_URL: /myapp/docker-registry/url
phases:
  install:
    runtime-versions:
      python: 3.11
  pre_build:
    commands:
      - echo "Installing dependencies..."
      - pip install -r day-14/simple-python-app/requirements.txt
  build:
    commands:
      - echo "Running tests..."
      - cd day-14/simple-python-app/
      - echo "Building Docker image..."
      - echo "$DOCKER_REGISTRY_PASSWORD" | docker login -u "$DOCKER_REGISTRY_USERNAME" --password-stdin "$DOCKER_REGISTRY_URL"
      - docker build -t "$DOCKER_REGISTRY_URL/$DOCKER_REGISTRY_USERNAME/simple-python-flask-app:latest" .
      - docker push "$DOCKER_REGISTRY_URL/$DOCKER_REGISTRY_USERNAME/simple-python-flask-app:latest"
  post_build:
    commands:
      - echo "Build completed successfully!"
artifacts:
  files:
    - '**/*'
  base-directory: ../simple-python-app

Configure where build logs should be stored for monitoring and troubleshooting.
Specify the build artifacts to be generated and where they should be stored (e.g., S3 bucket).
Review the configuration details to ensure accuracy.
Click on “Create build project” to create the CodeBuild project.

The CodeBuild project is now created and ready to execute builds based on the specified configuration.

Click “Start Build”.
Use “Build Details” to see the status of the build.

By following these steps, you can create an AWS CodeBuild project that integrates with your source code repository and is configured to build Docker images or execute other build tasks as per your project requirements.

Verify the Image Was Pushed to Docker Hub

Log in to your Docker Hub account.
Navigate to the repository where you expected the image to be pushed.
Verify that the image is present with the correct tag and was pushed recently.

By following these steps, you can confirm that your Docker image was successfully pushed to Docker Hub, ensuring the build process was completed correctly.

AWS Code Pipeline

In this step, we will set up an AWS CodePipeline to automate the continuous integration (CI) process.

CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates.

By integrating with your source repository and CodeBuild project, CodePipeline will automatically detect changes in your code and trigger the build process.

For this article, we’ll focus on the CI part, ensuring that every code change is automatically built and tested.

The deployment (CD) part is covered in another documentation, which you can find here.

Open the AWS Management Console.
Navigate to the AWS CodePipeline service.
Click on “Create pipeline” to start the pipeline creation wizard.
Enter a name for your pipeline.

Select “New service role” to let AWS create a role for the pipeline or “Existing service role” if you have a preconfigured IAM role.

Select the source provider for your code repository (e.g., AWS CodeCommit, GitHub, Bitbucket).
Enter the name of your repository.
Specify the branch to use (e.g., main or master).

Choose AWS CodeBuild.
Select the region where your CodeBuild project is located.
Select the CodeBuild project you created earlier.
Add any necessary environment variables required for the build process.

Skip code deploy, since this guide focuses on CI, you might not need a deploy stage here.
Review all the stages and configurations.
Click on “Create pipeline” to finalize and create your CodePipeline.
Trigger the pipeline manually by clicking “Release change” or wait for an automated trigger from a new code commit.
Monitor the progress of the pipeline in the CodePipeline console. Each stage (Source, Build) will show its status.

Clean Up Resources

After completing the deployment, it’s important to clean up resources to avoid unnecessary charges. Ensure you:

1: Delete the CodePipeline: Select the pipeline you created and delete it.

2: Delete the CodeBuild Project: Select the build used in the pipeline and delete it.

3: Remove Secrets from AWS Systems Manager: Select and delete the parameters used to store Docker credentials.

4: Clean Up IAM Roles: Find the roles created for this project (e.g., CodePipeline and CodeBuild roles). Detach any policies attached to these roles and then delete the roles.

Verify Deletion:

Double-check that all resources have been deleted from the AWS Management Console to ensure there are no lingering resources that could incur charges.

Conclusion

In this project, we successfully set up a Continuous Integration (CI) pipeline using AWS CodePipeline and AWS CodeBuild, complemented by AWS Systems Manager for securely managing Docker credentials.

This setup ensures that every code change is automatically built and tested, streamlining our development workflow and maintaining code quality.

By leveraging these AWS services, we’ve created a scalable and efficient CI pipeline that supports rapid development and deployment.

For further information on the CD part of this project, you can refer here.

Connect with me

For more insights and future projects, feel free to connect with me on LinkedIn , GitHub and Medium.