Setting Up a Complete AWS Infrastructure Connected to GitLab | by Obafemi | Oct, 2024

Posted on by

Building automated CI/CD pipelines, streamlined workflow from development to production.

In this detailed tutorial, we will walk through the process of setting up a complete AWS infrastructure connected to GitLab. Our goal is to create four distinct environments (Development, Staging, Production, and Demonstration), where each server functions independently. These servers will communicate with each other only through a script used to transfer code between them.

Additionally, we’ll cover how to manage access control in a simple and flexible way, ensuring that team members can be added or removed quickly.

By the end of this tutorial, you will have a robust infrastructure on AWS that is integrated with GitLab for Continuous Integration and Continuous Deployment (CI/CD).

Prerequisites

Before we begin, ensure you have the following:

– An AWS account
– Basic understanding of AWS services such as EC2, IAM, and VPC
– A GitLab account with repository access
– AWS CLI installed and configured on your local machine
– Terraform or CloudFormation (it’s optional but recommended for automation)

1: Set Up AWS VPC and Subnets

The first step is to create a Virtual Private Cloud (VPC) and subnets to isolate your servers. This ensures that each environment (Development, Staging, Production, Demonstration) is logically separated.

1.1 Create a VPC

  • Log in to your AWS Management Console.
  • Navigate to VPC and click on Create VPC.
  • Set the name as ‘Project-VPC’ and choose a CIDR block (e.g., ’10.0.0.0/16’).
  • Enable DNS resolution and hostnames for better management.

1.2 Create Subnets for Each Environment

  • Under the VPC dashboard, select Subnets and create four subnets for Development, Staging, Production, and Demonstration. Each subnet should have its own CIDR block (e.g., ’10.0.1.0/24′ for Development).
  • Ensure each subnet is associated with the main ‘Project-VPC’.

1.3 Set Up Internet Gateway and Route Tables

  • Create an Internet Gateway and attach it to your Project-VPC.
  • Create a Route Table and associate it with each of the subnets, allowing public internet access for now.
  • Add a route to the Route Table with ‘0.0.0.0/0′ and point it to the Internet Gateway.

2: Launch EC2 Instances for Each Server

Next, we’ll create four EC2 instances, one for each environment.

2.1 Launch EC2 Instances

2.2 Configure Security Groups

For each instance, create a security group to manage access. Here’s what you need to configure:

– Allow SSH (port 22) only from trusted IP addresses (such as your office or home IP).
– Optionally, allow HTTP (port 80) or HTTPS (port 443) if you’re running web servers.

2.3 Attach Elastic IPs

  • For ease of access, allocate Elastic IPs for each EC2 instance.
  • Attach the Elastic IPs to their respective servers so that they can be accessed publicly if needed.

3: Set Up Access Management with IAM

Proper access management is crucial to ensure that your team can easily access the right servers.

3.1 Create IAM Groups and Roles

  • In the IAM console, create three groups: ‘developers’, ‘staging-team’, and ‘production-team’.
  • Attach the necessary policies to each group. For example:
    – ‘developers’ group can access the Development and Demonstration servers.
    – ‘staging-team’ group can access the Staging server.
    – ‘production-team’ group can access the Production server.

3.2 Create Users and Assign to Groups

  • Add users for each team member by going to IAM > Users.
  • Assign users to the relevant group based on their roles.
  • Enable multi-factor authentication (MFA) for better security.
  • Optionally, generate programmatic access credentials for team members who need CLI access.

Automate all your server, database and cloud storage backups.

4: Set Up GitLab and CI/CD Pipelines

4.1 Create a GitLab Project Repository

  • In GitLab, create a new project for your codebase. This repository will act as the central hub for your CI/CD pipelines.
  • Push your initial code to the repository.

4.2 Generate SSH Keys for GitLab and EC2 Instances

On your local machine, generate an SSH key pair that will be used for communication between GitLab and the EC2 instances.

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
  • Add the public key to your GitLab repository by going to Settings > CI/CD> SSH Keys.
  • Copy the same public key to the ‘~/.ssh/authorized_keys’ file on each of the EC2 instances (Development, Staging, Production, and Demonstration).

4.3 Define GitLab CI/CD Pipeline

In your GitLab repository, create a ‘.gitlab-ci.yml’ file to automate code deployments.

stages:
- deploy
deploy_dev:
stage: deploy
script:
- ssh -o StrictHostKeyChecking=no ec2-user@dev-server 'cd /var/www/app && git pull origin main'
only:
- dev
deploy_staging:
stage: deploy
script:
- ssh -o StrictHostKeyChecking=no ec2-user@staging-server 'cd /var/www/app && git pull origin main'
only:
- staging
deploy_prod:
stage: deploy
script:
- ssh -o StrictHostKeyChecking=no ec2-user@prod-server 'cd /var/www/app && git pull origin main'
only:
- prod

This pipeline will:
– Deploy code to the Development server when pushing to the dev branch.
– Deploy code to the Staging server when pushing to the staging branch.
– Deploy code to the Production server when pushing to the prod branch.

5: Automate Code Transfer Between Servers

The environments should communicate via a script that allows the transfer of code between Development, Staging, Production, and Demonstration. This script will be executed manually when ready to promote code from one environment to another.

5.1 Create a Transfer Script

On your Development server, create a script that transfers code to the Staging server, and so on.

#!/bin/bash
# Transfer code from Development to Staging
scp -i ~/.ssh/id_rsa -r /var/www/app ec2-user@staging-server:/var/www/app
# Transfer code from Staging to Production
scp -i ~/.ssh/id_rsa -r /var/www/app ec2-user@prod-server:/var/www/app
# Transfer code from Production to Demonstration 
scp -i ~/.ssh/id_rsa -r /var/www/app ec2-user@demo-server:/var/www/app

Make the script executable:

chmod +x transfer_code.sh

5.2 Test the Script

To ensure that the script works as expected, run the script and verify that the code is transferred between the servers.

Automate all your server, database and cloud storage backups.

6: Manage Access Control Dynamically

AWS IAM makes it easy to manage access. To add or remove team members quickly:

  • Go to IAM > Users and add a new user to the appropriate group (e.g., developers, staging-team, or production-team).
  • If a user needs to be removed, simply delete their IAM user, and their access will be revoked across all servers instantly.

For better control, you can also implement role-based access using IAM Roles, ensuring that users only have access to specific AWS resources required for their tasks.

And that’s it.

You’ve successfully set up a complete AWS infrastructure connected to GitLab with four distinct environments: Development, Staging, Production, and Demonstration. Each server functions independently and only communicates through a script for code transfer. Additionally, we’ve implemented flexible access management using AWS IAM, allowing quick addition or removal of team members.

This setup can now serve as the foundation for building scalable, secure, and automated CI/CD pipelines, ensuring a streamlined workflow from development to production. You can expand this infrastructure by adding more automation, monitoring, and security controls as your project evolves.

Note: There are affiliate links in the article and if you buy something, I’ll get a commission at no extra cost to you.

This content is free, and by using these links, You’ll be supporting my work & that means a whole lot to me.