Git webhooks on the AWS Cloud

Quick Start Reference Deployment

QS

January 2021
Kirankumar Chandrashekar and Jay McConnell, AWS Quick Start team

Visit our GitHub repository for source files and to post feedback, report bugs, or submit feature ideas for this Quick Start.

This Quick Start was created by Amazon Web Services (AWS). Quick Starts are automated reference deployments that use AWS CloudFormation templates to deploy key technologies on AWS, following AWS best practices.

Overview

This Quick Start deploys the HTTPS endpoint you can use to configure a webhook to link your Git and AWS services. With a webhook in place, each time a Git user pushes a commit, your repository is automatically retrieved, zipped, and uploaded to an Amazon Simple Storage Service (Amazon S3) bucket. You can then configure AWS services such as AWS CodePipeline, AWS CodeBuild, and AWS CodeDeploy to use the S3 bucket as a source.

This guide describes the components that are deployed by the Quick Start, and contains links to launch the AWS CloudFormation template that automates the deployment.

Git webhooks on AWS

After deploying the Quick Start, you use the endpoint information it provides to configure a webhook in your Git service. A webhook sends an HTTPS POST request to the endpoint in response to a push action. The HTTPS POST request contains JavaScript Object Notation (JSON) data about the push and repository. After the request is accepted by Amazon API Gateway, it is passed to an AWS Lambda function that triggers an AWS CodeBuild project. The AWS CodeBuild project uses the information in the HTTPS POST request to retrieve the latest version of your repository.

For more information about the components that this Quick Start deploys, see the Architecture section later in this guide.

Cost

You are responsible for the cost of the AWS services used while running this Quick Start. There is no additional cost for using the Quick Start.

The AWS CloudFormation templates for Quick Starts include configuration parameters that you can customize. Some of the settings, such as the instance type, affect the cost of deployment. For cost estimates, see the pricing pages for each AWS service you use. Prices are subject to change.

After you deploy the Quick Start, create AWS Cost and Usage Reports to deliver billing metrics to an Amazon Simple Storage Service (Amazon S3) bucket in your account. These reports provide cost estimates based on usage throughout each month and aggregate the data at the end of the month. For more information, see What are AWS Cost and Usage Reports?

Architecture

Deploying this Quick Start builds the following environment in the AWS Cloud.

architecture_diagram
Figure 1. Git webhooks with AWS services Quick Start architecture

As shown in Figure 1, this Quick Start sets up a serverless AWS Cloud environment that includes the following components:

  • Amazon API Gateway to receive Git webhook requests and forward them to AWS Lambda.

  • An AWS Lambda function to process Git webhook requests from API Gateway and invoke an AWS CodeBuild project.

  • An AWS CodeBuild project to connect to your Git service, then retrieve, zip, and upload the latest version of your Git repository to Amazon S3.

  • An AWS Key Management Service (AWS KMS) key to encrypt/decrypt the SSH (Secure Shell) keys used by AWS CodeBuild to connect to your Git repository using SSH. The SSH key pair is generated by a Lambda-backed AWS CloudFormation custom resource when the stack is deployed.

  • Two Amazon S3 buckets: one for Git repository contents, and another for encrypted SSH keys. A Lambda-backed AWS CloudFormation custom resource deletes the contents of the S3 buckets when you delete the CloudFormation stack. If you need backups, copy the S3 buckets before deleting the stack.

  • The Quick Start deploys AWS Identity and Access Management (IAM) roles required by Lambda and API Gateway. The inline permissions attached to the roles are scoped using the least privilege model.

  • The AWS CodeBuild project must be able to communicate with your Git repository. For example, you can employ a SaaS-based Git service like GitHub to which CodeBuild can connect over the internet.

  • The Git repository S3 bucket this Quick Start deploys has versioning enabled, and all previous versions are retained indefinitely. To modify the retention period, see How do I create a lifecycle rule for an S3 bucket?

Planning the deployment

Specialized knowledge

This deployment requires a moderate level of familiarity with AWS services. If you’re new to AWS, visit Getting Started with AWS and Training and Certification. These sites provide materials for learning how to design, deploy, and operate your infrastructure and applications on the AWS Cloud.

AWS account

If you don’t already have an AWS account, create one at https://aws.amazon.com by following the on-screen instructions. Part of the sign-up process involves receiving a phone call and entering a PIN using the phone keypad.

Your AWS account is automatically signed up for all AWS services. You are charged only for the services you use.

Resource quotas

If necessary, request service quota increases for the following resources. You might need to request increases if your existing deployment currently uses these resources and if this Quick Start deployment could result in exceeding the default quotas. The Service Quotas console displays your usage and quotas for some aspects of some services. For more information, see What is Service Quotas? and AWS service quotas.

EC2 key pairs

Make sure that at least one Amazon EC2 key pair exists in your AWS account in the Region where you plan to deploy the Quick Start. Make note of the key pair name. You need it during deployment. To create a key pair, see Amazon EC2 key pairs and Linux instances.

For testing or proof-of-concept purposes, we recommend creating a new key pair instead of using one that’s already being used by a production instance.

IAM permissions

Before launching the Quick Start, you must sign in to the AWS Management Console with IAM permissions for the resources that the templates deploy. The AdministratorAccess managed policy within IAM provides sufficient permissions, although your organization may choose to use a custom policy with more restrictions. For more information, see AWS managed policies for job functions.

Deployment options

Deployment steps

Sign in to your AWS account

  1. Sign in to your AWS account at https://aws.amazon.com with an IAM user role that has the necessary permissions. For details, see Planning the deployment earlier in this guide.

  2. Make sure that your AWS account is configured correctly, as discussed in the Technical requirements section.

Launch the Quick Start

You are responsible for the cost of the AWS services used while running this Quick Start reference deployment. There is no additional cost for using this Quick Start. Prices are subject to change. See the pricing pages for each AWS service you use in this Quick Start for full details.
  1. Check the Region that’s displayed in the upper-right corner of the navigation bar, and change it if necessary. This Region is where the Quick Start infrastructure is built. The template for this Quick Start is launched in the US East (Ohio) Region by default.

  2. On the Select Template page, keep the default setting for the template URL, and then choose Next.

  3. On the Specify Details page, change the stack name if needed. Review the parameters for the template. Provide values for the parameters that require input. For all other parameters, review the default settings and customize them as necessary. For details on each parameter, see the Parameter reference section of this guide. After reviewing and customizing the parameters, choose Next.

  1. On the Configure stack options page, you can specify tags (key-value pairs) for resources in your stack and set advanced options. When you’re finished, choose Next.

  2. On the Review page, review and confirm the template settings. Under Capabilities, select the two check boxes to acknowledge that the template creates IAM resources and might require the ability to automatically expand macros.

  3. Choose Create stack to deploy the stack.

  4. Monitor the status of the stack. When the status is CREATE_COMPLETE, the Git webhooks deployment is ready.

  5. Use the values displayed in the Outputs tab for the stack, as shown in Figure 2, to view the created resources.

cfn_outputs
Figure 2. Git webhooks outputs after successful deployment

Configuring Git services

After deploying the Quick Start, set up a webhook in your Git repository.

To configure a webhook, you need GitPullWebHookApi and PublicSSHKey. You can find these on the Outputs tab on the AWS CloudFormation console after deploying the Quick Start.

outputs_tab
Figure 3. Outputs tab on the AWS CloudFormation console
  • GitPullWebHookApi is the URL endpoint that receives the HTTP POST request from the Git service.

  • PublicSSHKey is the public SSH key used to connect to your Git repository. This key can be configured as a read-only machine user or as a deployment key in your Git repository.

The instructions for setting up webhooks and deployment keys vary by Git service. For more information, see your Git service documentation.

Configuring AWS services

After deploying the Quick Start, configure the AWS services in your workload to use the Git repository S3 bucket as a source.

As shown in Figure 3, the Outputs tab in the AWS CloudFormation console includes OutputBucketName. This output is an Amazon S3 key that forms the base of the path to the .zip file of your repository code. The S3 key has the following format:

S3://output-bucket-name/git-user/git-repository/git-user_git-repository.zip

Here, git-user is the owner or path prefix of the repository. In some Git services, this may be an organization name. However, some Git services do not return a Git user or organization for a repository. In these cases, you can omit the git-user parts of the path.

The instructions vary for linking an AWS service to an Amazon S3 object. For links to AWS service documentation, see AWS services, later in this guide.

Adding an API secret after deployment

You can launch this Quick Start without an API Secret parameter. If your Git service provides an API secret when you create a webhook, you can update the stack with the API secret later.

To update the stack with an API secret, do the following:

  1. In the AWS CloudFormation console, select the stack you want to update.

  2. In the stack details pane, choose Update.

  3. Choose Use current template.

  4. On the Specify stack details page, change the API Secret parameter setting, then choose Next.

  5. On the Configure stack options page, choose Next.

  6. Select I acknowledge that this template may create IAM resources.

  7. Choose Update stack. When the status is UPDATE_COMPLETE, the stack is updated with the API secret.

Test the deployment

Before putting a webhook into production, test your deployment by doing the following:

  1. Modify a file in your repository.

  2. Commit and push the changes.

  3. Wait a few minutes, and then check the Git repository S3 bucket for a new (or updated) object with a key that matches your repository path.

S3_new_object
Figure 4. Checking for a new or updated object in your S3 bucket after a commit

Best practices

The architecture built by this Quick Start supports AWS best practices for security.

SSH keys

This Quick Start deploys a private SSH key pair that is encrypted with an AWS KMS key and uploaded to Amazon S3. AWS CodeBuild decrypts the private SSH key and uses it to authenticate your Git service before cloning the repository.

We don’t recommend sharing SSH keys among multiple services, or launching another instance of this Quick Start to clone and store another repository in Amazon S3. Each repository should use unique SSH keys.

Webhook security

Git services provide different ways to authenticate an endpoint, such as webhook secrets, source-IP-address allow listing, personal access tokens, and OAuth2. We recommend that you set up at least one of these security mechanisms to protect your webhook API endpoint.

For more information about how this Quick Start uses endpoint security mechanisms, see the Parameter reference section of this guide. For specific guidance on how to configure security mechanisms for your Git service, refer to your Git service documentation.

FAQ

Q. I encountered a CREATE_FAILED error when I launched the Quick Start.

A. If AWS CloudFormation fails to create the stack, we recommend that you relaunch the template with Rollback on failure set to Disabled. (This setting is under Advanced on the Options page of the AWS CloudFormation console.) With this setting, the stack’s state is retained and the instance remains running so you can troubleshoot the issue.

When you set Rollback on failure to Disabled, you continue to incur AWS charges for this stack. Ensure that you delete the stack after troubleshooting.

For more information, see Troubleshooting AWS CloudFormation.

Troubleshooting

If commits to your repository do not show up in Amazon S3, do the following:

  • Check the security parameters and endpoint in your Git webhook configuration. See Configuring Git services earlier in this guide and consult your Git service documentation for help with configuring webhooks.

  • Check the AWS Lambda logs for errors. These are stored in Amazon CloudWatch Logs. For help with accessing them, see Accessing Amazon CloudWatch logs for AWS Lambda.

  • Check the AWS CodeBuild project logs for errors. To access them, do the following:

    1. Open the AWS CodeBuild console.

    2. On the Build history page, choose the Build run link for the project.

    3. On the Build status page, see the Build logs tab.

Additional resources

Parameter reference

Unless you are customizing the Quick Start templates for your own deployment projects, we recommend that you keep the default settings for the parameters labeled Quick Start S3 bucket name, Quick Start S3 bucket Region, and Quick Start S3 key prefix. Changing these parameter settings automatically updates code references to point to a new Quick Start location. For more information, see the AWS Quick Start Contributor’s Guide.

Parameters for deploying into your selected Region

Table 1. General settings
Parameter label (name) Default value Description

Output S3 bucket name (OutputBucketName)

Blank string

(Optional) Name for the S3 bucket where the Git repository .zip file is stored. If left blank, the Quick Start creates one for you.

Custom domain name (CustomDomainName)

Blank string

Domain name for the webhook endpoint. If left blank, API Gateway creates a domain name for you.

Table 2. Git pull settings
Parameter label (name) Default value Description

API secret (ApiSecret)

Blank string

API secret used to authenticate access to webhooks in GitHub Enterprise, GitLab, and other Git services. If a webhook payload header contains a matching secret, IP address authentication is bypassed. API secrets cannot contain commas (,), backward slashes (\), or quotes (").

Allowed IP addresses (AllowedIps)

18.205.93.0/25,18.234.32.128/25,13.52.5.0/25

Comma-separated list of allowed IP CIDR blocks. The default addresses listed are BitBucket Cloud IP ranges.

Exclude .git directory (ExcludeGit)

True

Choose False to omit the .git directory from the Git repository .zip file.

Table 3. AWS Quick Start configuration
Parameter label (name) Default value Description

Quick Start S3 bucket name (QSS3BucketName)

aws-quickstart

S3 bucket name for Quick Start assets. It can include numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen (-).

Quick Start S3 bucket Region (QSS3BucketRegion)

us-east-1

AWS Region where the Quick Start assets S3 bucket (QSS3BucketName) is hosted. Required when using your own S3 bucket.

Quick Start S3 key prefix (QSS3KeyPrefix)

quickstart-git2s3/

Key prefix for the Quick Start assets S3 bucket. A key prefix is similar to a directory name that enables you to store similar data under the same directory in an S3 bucket. It can include numbers, lowercase letters, uppercase letters, hyphens (-), and forward slashes (/).

Table 4. VPC configuration
Parameter label (name) Default value Description

VPC ID (VPCId)

Blank string

ID of the VPC in which the Lambda function runs.

VPC CIDR (VPCCidrRange)

Blank string

CIDR range of the VPC.

Subnet IDs (SubnetIds)

Blank string

SubnetIDs in which the Lambda function runs.

Hostname override (ScmHostnameOverride)

Blank string

Name to override the hostname in the header of a webhook JSON payload.

Send us feedback

To post feedback, submit feature ideas, or report bugs, use the Issues section of the GitHub repository for this Quick Start. To submit code, see the Quick Start Contributor’s Guide.

Quick Start reference deployments

GitHub repository

Visit our GitHub repository to download the templates and scripts for this Quick Start, to post your comments, and to share your customizations with others.


Notices

This document is provided for informational purposes only. It represents AWS’s current product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS’s products or services, each of which is provided “as is” without warranty of any kind, whether expressed or implied. This document does not create any warranties, representations, contractual commitments, conditions, or assurances from AWS, its affiliates, suppliers, or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.

The software included with this paper is licensed under the Apache License, version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the accompanying "license" file. This code is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either expressed or implied. See the License for specific language governing permissions and limitations.