When users launch an AWS CloudFormation template, they get charged for AWS usage even if the stack fails and the instance runs for a very brief time. For this reason, you need to test your Quick Start thoroughly before we publish it. Testing becomes even more important for complex stacks and expensive stacks (such as those that use the X1 instance type).

Manual testing

Make sure to include the following in your testing:

  • Hitting AWS limits (for example, VPCs, Amazon EBS, Elastic IP addresses).
  • Stacks launched within the user’s VPC that don’t use our NAT gateway/instance or Internet gateway. This becomes an issue when an operating system doesn’t recognize the instances and repository registrations fail, which causes downstream failures.
  • Stack deletion. Make sure that deleting a CREATE_COMPLETE stack succeeds in a single try and doesn’t display a DELETE_FAILED status message.
  • Template validation. Launch your templates in the AWS CloudFormation console to check parameters and test the user experience. This review will make it easier to detect issues with parameter names, labels, defaults, and descriptions. For example:
    • If parameters appear in an “Other parameters” group in the console, they’re orphaned in the template. You’ll need to add these parameters to the appropriate group in the Metadata section of your template.
    • If you’ve defined default values that don’t comply with parameter constraints (AllowedValues or AllowedPatterns), the console will display an error message. You’ll need to fix the default value or revise the constraint.
    • The visual test pass will also help you identify cosmetic issues and inconsistencies such as typos in parameter labels and descriptions, or truncated descriptions.

Debugging

Debugging the Quick Start templates is never easy, because deployment can fail midway through stack creation. Here are some helpful approaches:

  • Disable the Rollback on failure option before stack creation so that you have a partially working stack in case of a failure and can find details about the failure in the logs.
    The user will still incur charges for whatever has been deployed up to the point of failure.
  • Use a debug/verbose flag as input to generate additional logging.
  • It’s always best to explicitly signal in order to properly control the flow of the deployment. You can do this by calling cfn-signal with the wait condition handle or resource.

For more information about troubleshooting your Quick Start templates, see the AWS CloudFormation documentation.

Testing with TaskCat

TaskCat is a tool that tests AWS CloudFormation templates. It deploys your AWS CloudFormation template in multiple AWS Regions and generates a report with a pass/fail grade for each region. You can specify the regions and number of Availability Zones you want to include in the test, and pass in parameter values from your AWS CloudFormation template. TaskCat is implemented as a Python class that you import, instantiate, and run.

TestCat was developed by the AWS Quick Start team to test AWS CloudFormation templates for Quick Starts. We’re pleased to make the tool available to all developers who want to validate their Quick Start templates during development, or test their custom AWS CloudFormation templates.

TaskCat supports all major Linux distributions and MacOS. (For Windows, use the Bash shell.) For installation instructions, see the TaskCat documentation.

For detailed usage information and API reference, see the TaskCat documentation.

Creating a parameters file for automated testing

When you’ve finished developing your templates, the Quick Start team runs them through our internal automated validation and integration system. The validation test requires the following:

  • A configuration file named config.yml in the ci subfolder of the GitHub repo. This file is generated automatically. Edit the file to specify your AWS CloudFormation templates and parameter files, and commit to the develop branch when you’re ready for the validation test. If you want to configure your test to run in specific regions, see the instructions in the Quick Start Automated Validation user guide. Make sure to always use the master template in any tests defined in the config.yml file.

  • An AWS CloudFormation parameters file in JSON format with a list of parameters and their defaults in the ci subfolder of the GitHub repo. For the following parameters that require user input, use these special values:

    To Use Comments
    Generate password $[alfred_genpass_x] Generates a password consisting of x characters, including a combination of uppercase letters [A-Z], lowercase letters [a-z], and numbers [0-9].
    Generate GUID $[alfred_genuuid] Generates a universally unique identifier (UUID).
    Get key-pair name $[alfred_getkeypair] Gets the current Amazon EC2 key pair name to be used for testing.
    Get Availability Zones $[alfred_getaz_x] Returns x number of Availability Zones from that region in the test account.
    Get single Availability Zone $[alfred_getsingleaz_x] Returns Availability Zone x from a 1-indexed array, from that region in the test account.
    Get license location $[alfred_getlicensebucket] Returns the name of the S3 bucket that contains the license file. This is a private bucket that’s owned by the Quick Start team. Example: "https://s3.amazonaws.com/$[alfred_getlicensebucket]/alfresco/one/license.key"
    Get license content $[getlicensecontent] Returns the contents of a license file from the S3 object key that follows this getter keyword. The license bucket is owned by the Quick Start team. Example: "$[alfred_getlicensecontent]/tableau/server/license.key"
    Get media location $[alfred_getmediabucket] Returns the name of the S3 bucket that contains installation media. This is a private bucket that’s owned by the Quick Start team. Example: "$[alfred_getmediabucket]/oracle/database/12"
    Copy parameter value $[alfred_getparamvalue_x] Copies the value of the parameter named x into another parameter that appears later in the parameter file.

Here’s an example of a parameters file that includes some of these user input values.

      [
          {
              "ParameterKey": "KeyPairName",
              "ParameterValue": "$[alfred_getkeypair]"
          },
          {
              "ParameterKey": "AdminPassword",
              "ParameterValue": "$[alfred_genpass_32]"
          },
          {
              "ParameterKey": "AdminUser",
              "ParameterValue": "StackAdmin"
          },
          {
              "ParameterKey": "AvailabilityZones",
              "ParameterValue": "$[alfred_getaz_2]"
          },
          {
              "ParameterKey": "CodeDeployDeploymentGroupName",
              "ParameterValue": "$[alfred_genuuid]"
          },
          {
              "ParameterKey": "CodeDeployNumberOfServers",
              "ParameterValue": "2"
          },
          {
              "ParameterKey": "DatabasePassword",
              "ParameterValue": "$[alfred_genpass_16]"
          },
          {
              "ParameterKey": "DatabasePasswordConfirm",
              "ParameterValue": "$[alfred_getparamvalue_DatabasePassword]"
          },
      ...
  • Do not add the QSS3BucketName and QSS3KeyPrefix names to the parameters file. These parameters are automatically modified by the validation/integration system directly in the template when launching tests.