Argo CD Add-on¶

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. The Argo CD add-on provisions Argo CD into an EKS cluster, and can optionally bootstrap your workloads from public and private Git repositories.

The Argo CD add-on allows platform administrators to combine cluster provisioning and workload bootstrapping in a single step and enables use cases such as replicating an existing running production cluster in a different region in a matter of minutes. This is important for business continuity and disaster recovery cases as well as for cross-regional availability and geographical expansion.

Please see the documentation below for details on automatic boostrapping with ArgoCD add-on. If you prefer manual bootstrapping (once your cluster is deployed with this add-on included), you can find instructions on getting started with Argo CD in our Getting Started guide.

Full Argo CD project documentation can be found here.

Usage¶

To provision and maintain ArgoCD components without any bootstrapping, the add-on provides a no-argument constructor to get started.

import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import * as blueprints from '@aws-quickstart/eks-blueprints';

const app = new cdk.App();

const addOn = new blueprints.addons.ArgoCDAddOn();

const blueprint = blueprints.EksBlueprint.builder()
  .version("auto")
  .addOns(addOn)
  .build(app, 'my-stack-name');

The above will create an argocd namespace and install all Argo CD components. In order to bootstrap workloads you will need to change the default ArgoCD admin password and add repositories as specified in the Getting Started documentation.

Functionality¶

Creates the namespace specified in the construction parameter (argocd by default).
Deploys the argo-cd Helm chart into the cluster.
Allows to specify ApplicationRepository selecting the required authentication method as SSH Key, username/password or username/token. Credentials are expected to be set in AWS Secrets Manager and replicated to the desired region. If bootstrap repository is specified, creates the initial bootstrap application which may be leveraged to bootstrap workloads and/or other add-ons through GitOps.
Allows setting the initial admin password through AWS Secrets Manager, replicating to the desired region.
Supports standard helm configuration options.

Setting an Admin Password¶

By default, the Argo CD add-on will create a new admin password for you. To specify your own, you can leverage the AWS Secrets Manager.

const argoCDAddOn = new ArgoCDAddOn({
    adminPasswordSecretName: `your-secret-name`
});
const addOns: Array<ClusterAddOn> = [ argoCDAddOn ];

The attribute adminPasswordSecretName is the logical name of the secret in AWS Secret Manager. Note, that when deploying to multiple regions, the secret is expected to be replicated to each region.

Inside ArgoCD, the admin password is stored as a bcrypt hash. This step will be performed by the framework and stored in the ArgoCD admin secret.

You can change the admin password through the Secrets Manager, but it will require rerunning the provisioning pipeline to apply the change.

Bootstrapping¶

The Blueprints framework provides an approach to bootstrap workloads and/or additional add-ons from a customer GitOps repository. In a general case, the bootstrap GitOps repository may contains an App of Apps that points to all workloads and add-ons.

In order to enable bootstrapping, the add-on allows passing an ApplicationRepository at construction time. The following repository types are supported at present:

Public HTTP/HTTPS repositories (e.g., GitHub)
Private HTTPS accessible git repositories requiring username/password authentication.
Private git repositories with SSH access requiring an SSH key for authentication.
Private HTTPS accessible GitHub repositories accessible with GitHub token.

An example is provided below, along with an approach that could use a separate app of apps to bootstrap workloads in different stages, which is important for a software delivery platform as it allows segregating workloads specific to each stage of the SDLC and defines clear promotion processes through GitOps.

import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { ArgoCDAddOn, ClusterAddOn, EksBlueprint, ApplicationRepository }  from '@aws-quickstart/eks-blueprints';

const secretStoreAddOn = new SecretsStoreAddOn();
const repoUrl = 'git@github.com:aws-samples/eks-blueprints-workloads.git'

const bootstrapRepo: ApplicationRepository = {
    repoUrl,
    credentialsSecretName: 'github-ssh-test',
    credentialsType: 'SSH'
}

const devBootstrapArgo = new ArgoCDAddOn({
    bootstrapRepo: {
        ...bootstrapRepo,
        path: 'envs/dev'
    }
});
const testBootstrapArgo = new ArgoCDAddOn({
    bootstrapRepo: {
        ...bootstrapRepo,
        path: 'envs/test',
    },

});
const prodBootstrapArgo = new ArgoCDAddOn({
    bootstrapRepo: {
        ...bootstrapRepo,
        path: 'envs/prod',
    },
    adminPasswordSecretName: 'ArgoCDAdmin',
});

const blueprint = EksBlueprint.builder()
    .version("auto")
    .addOns(secretStoreAddOn)
    .account(account);

blueprint.clone('us-east-1')
    .addOns(devBootstrapArgo)
    .build(app, 'argo-us-east-1');

blueprint.clone('us-east-2')
    .addOns(testBootstrapArgo)
    .build(app, 'argo-us-east-2');

blueprint.clone('us-west-2')
    .addOns(prodBootstrapArgo)
    .build(app, 'argo-us-west-1');

The application promotion process in the above example is handled entirely through GitOps. Each stage specific App of Apps contains references to respective application GitOps repository for each stage (e.g referencing the release vs work branches or path-based within individual app GitOps repository).

GitOps for EKS Blueprints AddOns¶

By default all AddOns defined in a blueprint are deployed to the cluster via CDK. You can opt-in to deploy them following the GitOps model via ArgoCD. You will need a repository contains all the AddOns you would like to deploy via ArgoCD, such as, eks-blueprints-add-ons. You then configure ArgoCD bootstrapping with this repository as shown above.

There are two types of GitOps deployments via ArgoCD depending on whether you would like to adopt the App of Apps strategy:

CDK deploys the Application resource for each AddOn enabled, and ArgoCD deploys the actual AddOn via GitOps based on the Application resource. Example:

    import * as blueprints from '@aws-quickstart/eks-blueprints';

    // enable gitops bootstrapping with argocd
    const prodBootstrapArgo = new blueprints.addons.ArgoCDAddOn({
        bootstrapRepo: {
            repoUrl: 'https://github.com/aws-samples/eks-blueprints-add-ons',
            path: 'add-ons',
            targetRevision: "main",
        },
    });
    // addons
    const addOns: Array<blueprints.ClusterAddOn> = [
        prodBootstrapArgo,
        new blueprints.addons.AppMeshAddOn(),
        new blueprints.addons.MetricsServerAddOn(),
        new blueprints.addons.AwsLoadBalancerControllerAddOn(),
    ];

    blueprints.EksBlueprint.builder()
        .version("auto")
        .addOns(...addOns)
        .enableGitOps(blueprints.GitOpsMode.APPLICATION)
        .build(scope, stackID);

CDK deploys the only Application resource for the App of Apps, aka bootstrap-apps, and ArgoCD deploys all the AddOns based on the bootstrap-apps. This requires the naming pattern between the AddOns and App of Apps matches. Example:

    import * as blueprints from '@aws-quickstart/eks-blueprints';

    // enable gitops bootstrapping with argocd app of apps
    const prodBootstrapArgo = new blueprints.addons.ArgoCDAddOn({
        bootstrapRepo: {
            repoUrl: 'https://github.com/aws-samples/eks-blueprints-add-ons',
            path: 'chart',
            targetRevision: "main",
        },
    });
    // addons
    const addOns: Array<blueprints.ClusterAddOn> = [
        prodBootstrapArgo,
        new blueprints.addons.AppMeshAddOn(),
        new blueprints.addons.MetricsServerAddOn(),
        new blueprints.addons.AwsLoadBalancerControllerAddOn(),
    ];

    blueprints.EksBlueprint.builder()
        .version("auto")
        .addOns(...addOns)
        .enableGitOps(blueprints.GitOpsMode.APP_OF_APPS)
        .build(scope, stackID);

Secrets Support¶

The framework provides support to supply repository and administrator secrets in AWS Secrets Manager. This support is evolving and will be improved over time as ArgoCD itself matures.

Private Repositories¶

SSH Key Authentication

Set credentialsType to SSH when defining bootstrap repository in the ArgoCD add-on configuration.

.addOns(new blueprints.addons.ArgoCDAddOn({
    bootstrapRepo: {
        repoUrl: 'git@github.com:aws-samples/eks-blueprints-workloads.git',
        path: 'envs/dev',
        credentialsSecretName: 'github-ssh-json',
        credentialsType: 'SSH'
    }
}))

Note: In this case the configuration assumes that there is a secret github-ssh-json define in the target account and all the regions where the blueprint will be deployed.

Define the secret in AWS Secret Manager as "Plain Text" that contains a JSON structure (as of ArgoCD 2.x) with the fields sshPrivateKey and url defined. Note, that JSON does not allow line break characters, so all new line characters must be escaped with \n.

Example Structure:

{
    "sshPrivateKey": "-----BEGIN THIS IS NOT A REAL PRIVATE KEY-----\nb3BlbnNzaC1rtdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAACFwAAAAdzc2gtcn\nNhAAAAAwEAAQAAAgEAy82zTTDStK+s0dnaYzE7vLSAcwsiHM8gN\nhq2p5TfcjCcYUWetyu6e/xx5Rh+AwbVvDV5h9QyMw4NJobwuj5PBnhkc3QfwJAO5wOnl7R\nGbehIleWWZLs9qq`DufViQsa0fDwP6JCrqD14aIozg6sJ0Oqi7vQkV+jR0ht/\nuFO1ANXBn2ih0ZpXeHSbPDLeZQjlOBrbGytnCbdvLtfGEsV0WO2oIieWVXJj/zzpKuMmrr\nebPsfwr36nLprOQV6IhDDo\n-----END NOT A REAL PRIVATE KEY-----\n",

    "url": "git@github"
}

Note: You can notice explicit \n characters in the sshPrivateKey.

url attribute is required and must specify full or partial URL for credentials template. For example git@github will set the credentials for all GitHub repositories when SSH authentication is used. For more information see Repository Credentials and SSH Repositories from official ArgoCD documentation.

To escape your SSH private key for storing it as a secret you can use the following command on Mac/Linux:

awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}'  <path-to-your-cert>

A convenience script to create the JSON structure for SSH private key can be found here. You will need to set the PEM_FILE(full path to the ssh private key file) and URL_TEMPLATE (part of the URL for credentials template) variables inside the script.

(important) Replicate the secret to all the desired regions.
Please see instructions for GitHub oou n details on setting up SSH access.

Username Password and Token Authentication

Set credentialsType to USERNAME or TOKEN when defining ApplicationRepository in the ArgoCD add-on configuration.
Define the secret in the AWS Secret Manager as "Key Value" and set fields url, username and password to the desired values (clear text). For TOKEN username could be set to any username and password field set to the GitHub token. Replicate to the desired regions.
Make sure that for this type of authentication your repository URL is set as https, e.g. https://github.com/aws-samples/eks-blueprints-workloads.git.

Example Structure for USERNAME and TOKEN authentication type:

{
    "username": "YOUR_GIT_USERNAME", 
    "password": "YOUR PASSWORD OR TOKEN",
    "url": "https://github.com/aws-samples"
}

Note: url value can be a path to the org, rather than an actual repository.

Admin Secret

Create a secret in the AWS Secrets Manager as "Plain Text" and set the value to the desired ArgoCD admin password.
Replicate the secret to all the desired regions.
Set the secret name in adminPasswordSecretName in ArgoCD add-on configuration.
You can change the secret value through AWS Secrets Manager, however, it will require to rerun cdk deploy with the minimal changeset to apply the change.

Alternatively to get started, the admin password hash can be set bypassing the AWS Secret by setting the following structure in the values properties of the add-on parameters:

import * as bcrypt from "bcrypt";

.addOns(new blueprints.addons.ArgoCDAddOn({
         ... // other settings
         values: {
           "configs": {
                "secret": {
                    "argocdServerAdminPassword": bcrypt.hash(<your password plain text>, 10) // or just supply <your bcrypt hash> directly
                }
            }
        }
     }))

For more information, please refer to the ArgoCD official documentation.

Known Issues¶

Destruction of the cluster with provisioned applications may cause cloud formation to get stuck on deleting ArgoCD namespace. This happens because the server component that handles Application CRD resource is destroyed before it has a chance to clean up applications that were provisioned through GitOps (of which CFN is unaware). To address this issue at the moment, App of Apps application should be destroyed manually before destroying the stack.
Changing the administrator password in the AWS Secrets Manager and rerunning the stack causes login error on ArgoCD UI. This happens due to the fact that Argo Helm rewrites the secret containing the Dex server API Key (OIDC component of ArgoCD). The workaround at present is to restart the argocd-server pod, which repopulates the token. Secret management aspect of ArgoCD will be improved in the future to not require this step after password change.

Troubleshooting¶

Dex Server crashing on startup with server.secretkey is missing.

It may be a byproduct of another failure. As a rule (unless ArgoCD secret is configured separately) the initial start of the ArgoCD server should populate a few fields in the in ArgoCD secret. If ArgoCD server fails to start or is waiting on some condition to become ready, these fields are not populated, causing cascading failures.

Make sure that all the secrets are mounted properly onto the ArgoCD server pod. It can be caused by an incorrect shape of the secret for private repositories (see "Private Repositories" section above). SSH secret is expected to have two fields (url and sshPrivateKey) and USERNAME/TOKEN is expected to have three fields (username, password, url).

Make sure your secret name (as defined in AWS Secrets Manager) does not conflict with ArgoCD reserved secret names, such as argocd-secret.