init

README.md

@@ -0,0 +1,49 @@
+## 
+
+This repository contains:
+- A terraform module that is meant to create s3 buckets, configure them, and apply the policies that are tied to them.
+- A terraform module that can create self-hosted github actions runners on ec2 instances and a yml configuration file for the runners.
+- A python script to run a simple flask API that can run terraform with certain parameters.
+- A '.env' file containing the variables for region, tfstate bucket, and a pre-shared secret for the requests.
+- The docker compose file used to deploy Jenkins, Vault and Selenium for a local testing environment.
+
+### How to run the s3 bucket api
+The python script provides a simplified way to request buckets for developers without getting into terraform files or the AWS console. To set it up, the required python dependencies need to be installed:
+```bash
+pip install dotenv marshmallow flask python_terraform
+```
+Then, the script can just be executed and the API will run on a machine, listening to port 8080.
+Here is an example of how to perform a request with curl to create a bucket:
+```bash
+curl -X POST -H "Content-Type: application/json" \
+-d '{"bucket_name":"mybucket-python-testing-1234999",\
+"environment":"prod",\
+"encryption":"Enabled",\
+"versioning":"Enabled",\
+"api_key":"08de5837ac8129886fec0d53d1a8626c"}'\
+ localhost:8080/create_bucket
+```
+
+### Terraform for the API
+The terraform files follow a simple structure and handle variables that are passed through from the python script. The variables to fill the required parameters are taken from the .env file and the requests made to the API.
+It is required to run terraform init the first time before the API works on deploying a bucket.
+
+### Jenkinsfile
+The jenkinsfile from requirement 1 has been placed in this repo to be used as an example of a python deployment with the required tests. 
+My jenkins deployment failed to install plugins after the first run. Because of time constraints, I was not able to test the full flow before publishing.
+
+### AWS runner
+The setup for creating the runners is quite easy, as the only required thing is to add a GITHUB_KEY in the runner.tf file and apply. I tried to make the permissions on the runners as slim as possible.
+
+### Lessons learned, security considerations, and possible improvements
+- Authentication could be handled not by a pre-shared secret. A better way would be to tie the authentication to IAM, which would in turn also allow nice permission management.
+- This is a flask dev server, which is not stable and shouldn't be used anywhere. Transforming this into a proper service with WSGI is recommended.
+- HTTP is not a secure protocol. This could be easily mitigated by running behind a reverse proxy with HTTPS (Which is easier to manage than with flask or WSGI).
+- Many more variables can be added, like AWS account, user, region, ACL management, etc.
+- python_terraform is not great at reporting any errors, and does not handle errors in a "python way". Maybe not the best tool for the job. Subprocess and direct terraform commands could make this better.
+- Variables cannot go into the backend file. I needed to dynamically write this file on execution to manage the state file with s3.
+- Plugins should have been installed on the initial setup of Jenkins, as something (unknown) changed on my local system not allowing for plugins to be installed afterward. 
+- The github runner code is not tested at the moment.
+- The full jenkinsfile is not tested at the moment.
+
+

aws_runner/aws.tf

@@ -0,0 +1,3 @@
+provider "aws" {
+    region = "eu-central-1"
+}

aws_runner/backend.tf

@@ -0,0 +1,7 @@
+terraform {
+    backend "s3" {
+        bucket = "company-s3-tfstate-bucket-eu-central-1"
+        region = "eu-central-1"
+        key = "s3-prod-mybucket-python-testing-1234999"
+}
+}

aws_runner/main.yml

@@ -0,0 +1,71 @@
+name: CI/CD Pipeline
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  build:
+    runs-on: [self-hosted, linux]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.12' 
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+
+  test:
+    runs-on: [self-hosted, linux]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.12'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+
+      - name: Run tests
+        run: |
+          pytest tests/unit -n auto --junitxml=reports/unit.xml --cov=app --cov-report=xml:reports/coverage.xml
+          pytest tests/integration --junitxml=reports/integration.xml
+          pytest tests/e2e --junitxml=reports/e2e.xml
+
+  deploy:
+    runs-on: [self-hosted, linux]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.12'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+
+      - name: Deploy to AWS (Example with AWS CLI)
+        run: |
+          aws configure set aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws configure set aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws configure set region us-east-1
+          aws s3 cp ./build/ s3://my-bucket-name/ --recursive

aws_runner/roles.tf

@@ -0,0 +1,40 @@
+resource "aws_iam_role" "github_runner_role" {
+  name               = "github-runner-role"
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Effect    = "Allow"
+        Principal = {
+          Service = "ec2.amazonaws.com"
+        }
+        Action    = "sts:AssumeRole"
+      },
+    ]
+  })
+}
+
+resource "aws_iam_policy" "github_runner_policy" {
+  name        = "github-runner-policy"
+  description = "Policy for GitHub Self-Hosted Runner EC2 instances"
+  policy      = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Effect   = "Allow"
+        Action   = [
+          "s3:ListBucket",
+          "s3:GetObject",
+          "ec2:DescribeInstances",
+          "ec2:DescribeVolumes"
+        ]
+        Resource = "*"
+      },
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "github_runner_policy_attachment" {
+  role       = aws_iam_role.github_runner_role.name
+  policy_arn = aws_iam_policy.github_runner_policy.arn
+}

aws_runner/runner.tf

@@ -0,0 +1,27 @@
+resource "aws_security_group" "github_runner_sg" {
+  name        = "github-runner-sg"
+  description = "Security group for GitHub self-hosted runners"
+}
+
+resource "aws_instance" "github_runner" {
+  ami           = "ami-07d3c3e2c1184609e"  
+  instance_type = "t3.medium"
+  key_name      = "dummy-keypair"
+  security_groups = [
+    aws_security_group.github_runner_sg.name
+  ]
+  iam_instance_profile = aws_iam_instance_profile.github_runner_instance_profile.name
+  user_data = <<-EOF
+              #!/bin/bash
+              curl -o actions-runner.tar.gz -L https://github.com/actions/runner/releases/download/v2.297.0/actions-runner-linux-x64-2.297.0.tar.gz
+              tar xzf ./actions-runner.tar.gz
+              ./config.sh --url https://github.com/your-org/your-repo --token GITHUB_TOKEN
+              sudo ./svc.sh install
+              sudo ./svc.sh start
+              EOF
+}
+
+resource "aws_iam_instance_profile" "github_runner_instance_profile" {
+  name = "github-runner-instance-profile"
+  role = aws_iam_role.github_runner_role.name
+}

aws_runner/sg.tf

@@ -0,0 +1,16 @@
+resource "aws_security_group" "github_runner_sg" {
+  name        = "github-runner-sg"
+  description = "Security group for GitHub self-hosted runners"
+  ingress {
+    from_port   = 22
+    to_port     = 22
+    protocol    = "tcp"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+  egress {
+    from_port   = 0
+    to_port     = 0
+    protocol    = "-1"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+}

docker-lab/docker-compose.yml

@@ -0,0 +1,36 @@
+services:
+  jenkins:
+    image: jenkins/jenkins:lts
+    privileged: true
+    user: root
+    ports:
+      - "8080:8080"
+      - "50000:50000"
+    container_name: jenkins
+    volumes:
+      - ./jenkins_home:/var/jenkins_home
+      - /var/run/docker.sock:/var/run/docker.sock
+      - /usr/bin/docker:/usr/bin/docker
+    networks:
+      - jenkins
+  
+  selenium:
+    image: selenium/standalone-chrome:latest
+    ports:
+      - 4444:4444
+    networks:
+      - jenkins
+  
+  vault:
+    image: hashicorp/vault:latest
+    ports:
+      - 8200:8200
+    env:
+      - VAULT_DEV_ROOT_TOKEN_ID: root
+    networks:
+      - jenkins
+
+networks:
+  jenkins:
+    external: true
+    name: jenkins 

s3_api/Jenkinsfile

@@ -0,0 +1,149 @@
+def vaultConfig() {
+    return [
+        vaultUrl: '172.17.0.3:8200',
+        credentialsId: 'vault-approle',
+        engineVersion: 2
+    ]
+}
+
+def vaultSecrets() {
+    return [[
+        path: 'secret/data/companyTransferMoneyService',
+        secretValues: [
+            [envVar: 'API_KEY', vaultKey: 'api_key'],
+            [envVar: 'SELENIUM_GRID', vaultKey: 'selenium_grid_url']
+        ]
+    ]]
+}
+
+
+pipeline {
+    agent {
+        docker {
+            image 'python:3.12-slim'
+            label 'python3.12'
+            args '-v /tmp:/tmp'  // Optional: Mount /tmp for caching, etc.
+        }
+    }
+
+    environment {
+        PYTHON_VERSION = "3.12"
+        AWS_REGION = "eu-central-1"
+        S3_BUCKET = "Company-ci-executions"
+        SERVICE_NAME = "companyTransferMoneyService"
+    }
+
+    options {
+        timestamps()
+        buildDiscarder(logRotator(numToKeepStr: '20'))
+    }
+
+    stages {
+
+        stage('Checkout') {
+            agent { label 'ubuntu' }
+            steps {
+                checkout scm
+            }
+        }
+
+        stage('Setup Environment') {
+            steps {
+                sh '''
+                python3.12 -m venv venv
+                . venv/bin/activate
+                pip install --upgrade pip
+                pip install -r requirements.txt
+                mkdir -p reports
+                '''
+            }
+        }
+
+        stage('Unit Tests (Parallel)') {
+            parallel {
+                stage('Unit Batch 1') {
+                    steps {
+                        sh '''
+                        . venv/bin/activate
+                        pytest tests/unit -n auto \
+                          --junitxml=reports/unit1.xml \
+                          --cov=app --cov-report=xml:reports/coverage1.xml
+                        '''
+                    }
+                }
+                stage('Unit Batch 2') {
+                    steps {
+                        sh '''
+                        . venv/bin/activate
+                        pytest tests/unit -n auto \
+                          --junitxml=reports/unit2.xml \
+                          --cov=app --cov-report=xml:reports/coverage2.xml
+                        '''
+                    }
+                }
+            }
+            post {
+                always {
+                    junit 'reports/unit*.xml'
+                    publishCoverage adapters: [coberturaAdapter('reports/coverage*.xml')]
+                }
+            }
+        }
+
+        stage('Integration Tests') {
+            steps {
+                withVault(configuration: vaultConfig(), vaultSecrets: vaultSecrets()) {
+                    sh '''
+                    . venv/bin/activate
+                    export API_KEY=$API_KEY
+                    export ENV=staging
+                    pytest tests/integration \
+                      --junitxml=reports/integration.xml
+                    '''
+                }
+            }
+            post {
+                always {
+                    junit 'reports/integration.xml'
+                }
+            }
+        }
+
+        stage('E2E Tests (Selenium Grid)') {
+            steps {
+                withVault(configuration: vaultConfig(), vaultSecrets: vaultSecrets()) {
+                    sh '''
+                    . venv/bin/activate
+                    export API_KEY=$API_KEY
+                    export SELENIUM_GRID_URL=$SELENIUM_GRID
+                    # Start Flask app in background
+                    nohup python app.py &
+                    FLASK_PID=$!
+                    # Give Flask a few seconds to start
+                    sleep 5
+                    pytest tests/e2e/test_selenium.py \
+                    --junitxml=reports/e2e_selenium.xml
+                    kill $FLASK_PID
+                    '''
+                }
+            }
+            post {
+                always {
+                    junit 'reports/e2e_selenium.xml'
+                }
+            }
+        }
+
+        stage('Upload Reports to S3') {
+            steps {
+                withAWS(region: "${AWS_REGION}", credentials: 'aws-jenkins-credentials') {
+                    sh '''
+                    aws s3 cp reports/ \
+                      s3://${S3_BUCKET}/${SERVICE_NAME}/${BUILD_NUMBER}/ \
+                      --recursive
+                    '''
+                }
+            }
+        }
+    }
+}

s3_api/s3api.py

@@ -0,0 +1,62 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+import requests, json, os
+from dotenv import load_dotenv
+from marshmallow import Schema, fields, ValidationError
+from flask import Flask, request, Response, jsonify
+from python_terraform import Terraform
+
+load_dotenv()
+
+
+API_KEY = os.getenv('API_KEY')
+
+app = Flask(__name__)
+
+class BucketSchema(Schema):
+    environment = fields.Str(required=True)
+    bucket_name = fields.Str(required=True)
+    versioning = fields.Str(required=True)
+    encryption = fields.Str(required=True)
+    api_key = fields.Str(required=True)
+
+def create_bucket(ENVIRONMENT, ENCRYPTED, BUCKET_NAME, VERSIONING):
+    tfstate_bucket = os.getenv("TFSTATE_BUCKET", "company-s3-tfstate-bucket-eu-central-1")
+    tfstate_region = os.getenv("AWS_REGION", "eu-central-1")
+    with open("terraform/backend.tf", "w") as f:
+        f.write('terraform {\n')
+        f.write('backend "s3" {\n')
+        f.write(f'bucket = "{tfstate_bucket}"\n')
+        f.write(f'region = "{tfstate_region}"\n')
+        f.write(f'key = "s3-{ENVIRONMENT}-{BUCKET_NAME}"\n')
+        f.write('}\n}')
+    tf = Terraform(working_dir='terraform',
+                   variables={'ENCRYPTED': ENCRYPTED, 'VERSIONING': VERSIONING, 'BUCKET_NAME': BUCKET_NAME, 'ENVIRONMENT': ENVIRONMENT}
+                    )
+    return tf.apply(capture_output=True, skip_plan=True, auto_approve=True, var={'ENCRYPTED': ENCRYPTED, 'VERSIONING': VERSIONING, 'BUCKET_NAME': BUCKET_NAME, 'ENVIRONMENT': ENVIRONMENT})
+
+@app.route('/', methods = ['GET'])
+def ping():
+    return ["Pong"]
+
+@app.route('/create_bucket', methods = ['POST'])
+def bucket_data():
+    request_data = BucketSchema().load(request.json)
+    if request_data["api_key"] == API_KEY:
+        try:
+            ENVIRONMENT = request_data["environment"]
+            BUCKET_NAME = request_data["bucket_name"]
+            ENCRYPTED = request_data["encryption"]
+            VERSIONING = request_data["versioning"]
+            if create_bucket(ENVIRONMENT, ENCRYPTED, BUCKET_NAME, VERSIONING)[0] == 1:
+                return "Something went wrong trying to create the bucket"
+            return f"Creating bucket {BUCKET_NAME} in {ENVIRONMENT} with encryption={ENCRYPTED} and versioning={VERSIONING}"
+        except ValidationError as err:
+            return jsonify(err.messages), 400        
+    else:
+        return "Authentication error", 403
+
+
+if __name__ == '__main__':
+    app.run(port = 8080, host="0.0.0.0")

s3_api/terraform/aws.tf

@@ -0,0 +1,3 @@
+provider "aws" {
+    region = "eu-central-1"
+}

s3_api/terraform/backend.tf

@@ -0,0 +1,7 @@
+terraform {
+backend "s3" {
+bucket = "company-s3-tfstate-bucket-eu-central-1"
+region = "eu-central-1"
+key = "s3-prod-mybucket-python-testing-1234999"
+}
+}

s3_api/terraform/main.tf

@@ -0,0 +1,50 @@
+resource "aws_s3_bucket" "bucket" {
+  bucket = var.BUCKET_NAME
+
+  tags = {
+    Name        = var.BUCKET_NAME
+    Environment = var.ENVIRONMENT
+  }
+}
+
+resource "aws_s3_bucket_versioning" "versioning" {
+  bucket = aws_s3_bucket.bucket.id
+
+  versioning_configuration {
+    status = var.ENCRYPTED
+  }
+}
+
+resource "aws_s3_bucket_public_access_block" "block_public" {
+  bucket = aws_s3_bucket.bucket.id
+
+  block_public_acls       = true
+  block_public_policy     = true
+  ignore_public_acls      = true
+  restrict_public_buckets = true
+}
+
+data "aws_iam_policy_document" "bucket_policy" {
+  statement {
+    sid    = "AllowUserReadAccess"
+    effect = "Allow"
+
+    principals {
+      type        = "AWS"
+      identifiers = ["arn:aws:iam::848173547540:user/dummy_user"]
+    }
+
+    actions = [
+      "s3:GetObject"
+    ]
+
+    resources = [
+      "${aws_s3_bucket.bucket.arn}/*"
+    ]
+  }
+}
+
+resource "aws_s3_bucket_policy" "bucket_policy" {
+  bucket = aws_s3_bucket.bucket.id
+  policy = data.aws_iam_policy_document.bucket_policy.json
+}

s3_api/terraform/vars.tf

@@ -0,0 +1,19 @@
+variable "BUCKET_NAME" {
+  description = "Name of the bucket to create"
+  type        = string
+}
+
+variable "ENCRYPTED" {
+  description = "S3 encryption enabled?"
+  type        = string
+}
+
+variable "VERSIONING" {
+  description = "S3 versioning enabled?"
+  type        = string
+}
+
+variable "ENVIRONMENT" {
+  description = "Staging or production env?"
+  type        = string
+}

s3_api/tests/e2e/test_e2e.py

@@ -0,0 +1,19 @@
+import os
+import pytest
+from selenium import webdriver
+from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
+
+SELENIUM_GRID_URL = os.getenv('SELENIUM_GRID_URL')
+
+@pytest.mark.e2e
+def test_ping_endpoint():
+    if not SELENIUM_GRID_URL:
+        pytest.skip("No Selenium Grid URL set")
+
+    driver = webdriver.Remote(
+        command_executor=SELENIUM_GRID_URL,
+        desired_capabilities=DesiredCapabilities.CHROME
+    )
+    driver.get("http://host.docker.internal:8080/")
+    assert "Pong" in driver.page_source
+    driver.quit()