Amazon AppFlow on the AWS Cloud

Quick Start Reference Deployment

August 2021
Anirudh Rautela, Avdesh Mankotia, and Karandeep Singh, Trantor Inc.
Dave May, AWS Infrastructure & Automation team

Visit our GitHub repository for source files and to post feedback, report bugs, or submit feature ideas for this Quick Start.

This Quick Start was created by Amazon Web Services (AWS). Quick Starts are automated reference deployments that use AWS CloudFormation templates to deploy key technologies on AWS, following AWS best practices.

Overview

This Quick Start deployment guide walks you through deploying Amazon AppFlow on the Amazon Web Services (AWS) Cloud for unidirectional data synchronization from Salesforce to Amazon Relational Database Service (Amazon RDS) PostgreSQL. This deployment requires extensive knowledge of AWS and Linux administration.

This Quick Start is for users who want an integration platform as a service (iPaaS) hosted on the AWS Cloud. Users can then deploy Amazon AppFlow through AWS CloudFormation.

Amazon AppFlow on AWS

Amazon AppFlow is a fully managed integration service for transferring data between software as a service (SaaS) applications (for example, Salesforce, Zendesk, Slack, and ServiceNow) and AWS services. This Quick Start deploys Amazon AppFlow for synchronization between Salesforce and an Amazon RDS PostgreSQL database.

AWS costs

You are responsible for the cost of the AWS services and any third-party licenses used while running this Quick Start. There is no additional cost for using the Quick Start.

The AWS CloudFormation templates for Quick Starts include configuration parameters that you can customize. Some of the settings, such as the instance type, affect the cost of deployment. For cost estimates, see the pricing pages for each AWS service you use. Prices are subject to change.

After you deploy the Quick Start, create AWS Cost and Usage Reports to deliver billing metrics to an Amazon Simple Storage Service (Amazon S3) bucket in your account. These reports provide cost estimates based on usage throughout each month and aggregate the data at the end of the month. For more information, see What are AWS Cost and Usage Reports?

Software licenses

No additional licenses are required to use this Quick Start.

Architecture

Deploying this Quick Start for a new virtual private cloud (VPC) with default parameters builds the following Amazon AppFlow environment in the AWS Cloud.

Figure 1. Quick Start architecture for Amazon AppFlow on AWS

As shown in Figure 1, the Quick Start sets up the following:

A highly available architecture that spans multiple Availability Zones.*
A VPC configured with public and private subnets, according to AWS best practices, to provide you with your own virtual network on AWS.*
In the public subnets, a managed network address translation (NAT) gateway to allow outbound internet access for resources in the private subnets.*
In the private subnets, an Amazon RDS PostgreSQL database.
Amazon Elastic Container Registry (Amazon ECR) to store Docker images.
Amazon Simple Storage Service (Amazon S3) to store Salesforce object data.
An Amazon AppFlow flow, created by an AWS Lambda function, to transfer Salesforce data to Amazon RDS and Amazon S3.

* The template that deploys the Quick Start into an existing VPC skips the components marked by asterisks and prompts you for your existing VPC configuration.

Figure 2. Quick Start flow diagram for Amazon AppFlow on AWS

As shown in Figure 2, this Quick Start sets up the following workflow:

Salesforce object data is transferred to Amazon AppFlow.
Amazon AppFlow relays Salesforce data to Amazon S3.
Amazon S3 relays object data to AWS Lambda, which sits inside a private subnet of your VPC.
AWS Lambda transfers your Salesforce data to Amazon RDS.

Planning the deployment

Specialized knowledge

This deployment requires a moderate level of familiarity with AWS services. If you’re new to AWS, see Getting Started Resource Center and AWS Training and Certification. These sites provide materials for learning how to design, deploy, and operate your infrastructure and applications on the AWS Cloud.

This Quick Start assumes familiarity with Amazon AppFlow, AWS CloudFormation, Amazon S3, Amazon RDS, and Salesforce.

AWS account

If you don’t already have an AWS account, create one at https://aws.amazon.com by following the on-screen instructions. Part of the sign-up process involves receiving a phone call and entering a PIN using the phone keypad.

Your AWS account is automatically signed up for all AWS services. You are charged only for the services you use.

Technical requirements

Before you launch the Quick Start, review the following information and ensure that your account is properly configured. Otherwise, deployment might fail.

Resource quotas

If necessary, request service quota increases for the following resources. You might need to request increases if your existing deployment currently uses these resources and if this Quick Start deployment could result in exceeding the default quotas. The Service Quotas console displays your usage and quotas for some aspects of some services. For more information, see What is Service Quotas? and AWS service quotas.

Resource	This deployment uses
VPCs	1
Elastic IP addresses	1
NAT gateways	1
Security groups	3
AWS Identity and Access Management (IAM) roles	3
Amazon ECR repositories	1
Amazon RDS PostgreSQL databases	1
Amazon S3 buckets	1
t2.micro instances	1

Supported Regions

This Quick Start supports the following Regions:

Americas

us-east-1, US East (N. Virginia)
us-east-2, US East (Ohio)
us-west-1, US West (N. California)
us-west-2, US West (Oregon)
ca-central-1, Canada (Central)
sa-east-1, South America (Sao Paulo)

Asia–Pacific

ap-northeast-1, Asia Pacific (Tokyo)
ap-northeast-2, Asia Pacific (Seoul)
ap-south-1, Asia Pacific (Mumbai)
ap-southeast-1, Asia Pacific (Singapore)
ap-southeast-2, Asia Pacific (Sydney)

Europe

eu-central-1, Europe (Frankfurt)
eu-west-1, Europe (Ireland)
eu-west-2, Europe (London)
eu-west-3, Europe (Paris)

Certain Regions are available on an opt-in basis. For more information, see Managing AWS Regions.

EC2 key pairs

Make sure that at least one Amazon EC2 key pair exists in your AWS account in the Region where you plan to deploy the Quick Start. Make note of the key pair name. You need it during deployment. To create a key pair, see Amazon EC2 key pairs and Linux instances.

For testing or proof-of-concept purposes, we recommend creating a new key pair instead of using one that’s already being used by a production instance.

IAM permissions

Before launching the Quick Start, you must sign in to the AWS Management Console with IAM permissions for the resources that the templates deploy. The AdministratorAccess managed policy within IAM provides sufficient permissions, although your organization may choose to use a custom policy with more restrictions. For more information, see AWS managed policies for job functions.

Prepare for the deployment

Prerequisites

To access Salesforce from Amazon AppFlow, your Salesforce edition must have API access. Salesforce editions, such as enterprise, unlimited, developer, and performance provide API access, but professional and essentials editions do not. Before proceeding, check the latest information about which editions provide API access.
Naming conventions are strict for objects, so use exact object names. For more information, see Object Reference for Salesforce and Lightning Platform. Note that object support may vary across Salesforce editions.

Create an Amazon AppFlow connection in Salesforce

Create a new connection in Amazon AppFlow.
1. Ensure that you are in the same Region in which you deploy this Quick Start.
2. Navigate to Connections in Amazon AppFlow.
3. Choose Salesforce and then Create connection.
  
  Figure 3. Amazon AppFlow connections
Choose Sandbox or Production, based on the Salesforce instance type, and add a new connection name. Then choose Continue.

Figure 4. Create a new connection
Add Salesforce credentials for the user account, and then choose Log In.

Figure 5. Entering Salesforce credentials
Choose Allow to allow access to basic information and data.

Figure 6. Allowing Amazon AppFlow access to Salesforce
Verify that the new connection name is listed in the connections list. Note that the connection name is required for deploying this Quick Start.

Figure 7. Amazon AppFlow connections

Deployment options

This Quick Start provides two deployment options:

Deploy Amazon AppFlow into a new VPC. This option builds a new AWS environment consisting of the VPC, subnets, NAT gateways, security groups, bastion hosts, and other infrastructure components. It then deploys Amazon AppFlow into this new VPC.
Deploy additional AppFlows for new Salesforce objects after deploying Amazon Appflow in a new VPC. This option provisions Amazon AppFlow in your existing AWS infrastructure.

The Quick Start provides separate templates for these options. It also lets you configure Classless Inter-Domain Routing (CIDR) blocks, instance types, and Amazon AppFlow settings, as discussed later in this guide.

Deployment steps

Confirm your AWS account configuration

Sign in to your AWS account at https://aws.amazon.com with an IAM user role that has the necessary permissions. For more information, see Planning the deployment, earlier in this guide.
Ensure that your AWS account is configured correctly, as discussed in the Technical requirements section.

Launch the Quick Start

If you deploy Amazon AppFlow into an existing VPC, ensure that your VPC has two private subnets in different Availability Zones for the workload instances and that the subnets are not shared. This Quick Start does not support shared subnets. The subnets require NAT gateways in their route tables to allow the instances to download packages and software without exposing them to the internet. Also ensure that the domain name in the DHCP options is configured, as explained in DHCP options sets for your VPC. Provide your VPC settings when you launch the Quick Start.

Each deployment takes about 20 minutes to complete.

Sign in to your AWS account, and choose one of the following options to launch the AWS CloudFormation template. For help with choosing an option, see Deployment options, earlier in this guide.

Deploy Amazon AppFlow into a new VPC on AWS

View template

Deploy additional AppFlows for new Salesforce objects after deploying Amazon Appflow in a new VPC

View template
Check the AWS Region that’s displayed in the upper-right corner of the navigation bar, and change it if necessary. This Region is where you build the network infrastructure. The template is launched in the us-east-1 Region by default. For other choices, see Supported Regions, earlier in this guide.
On the Create stack page, keep the default setting for the template URL, and then choose Next.
On the Specify stack details page, change the stack name if needed. Review the parameters for the template. Provide values for the parameters that require input. For all other parameters, review the default settings and customize them as necessary. For details about each parameter, see the Parameter reference section of this guide. When you finish reviewing and customizing the parameters, choose Next.
On the Configure stack options page, you can specify tags (key-value pairs) for resources in your stack and set advanced options. When you’re finished, choose Next.
On the Review page, review and confirm the template settings. Under Capabilities, select the two check boxes to acknowledge that the template creates IAM resources and might require the ability to automatically expand macros.
Choose Create stack to deploy the stack.
Monitor the status of the stack. When the status is CREATE_COMPLETE, the Amazon AppFlow deployment is ready.
To view the created resources, see the values displayed in the Outputs tab for the stack.

Test the deployment

Log in to your Salesforce account, and perform any operation (for example, create or update) on an object that is enabled for synchronization.
To create a bastion host for connecting to the Amazon RDS PostgreSQL database, see How can I connect to my Amazon RDS DB instance using a bastion host from my Linux/macOS machine?
Connect to Amazon RDS using a bastion host, and verify that data is written to PostgreSQL.

FAQ

Q. I encountered a CREATE_FAILED error when I launched the Quick Start.

A. If AWS CloudFormation fails to create the stack, relaunch the template with Rollback on failure set to Disabled. This setting is under Advanced in the AWS CloudFormation console on the Configure stack options page. With this setting, the stack’s state is retained, and the instance remains running so you can troubleshoot the issue. (For Windows, look at the log files in %ProgramFiles%\Amazon\EC2ConfigService and C:\cfn\log.)

When you set Rollback on failure to Disabled, you continue to incur AWS charges for the stack. Ensure that you delete the stack when you finish troubleshooting. For more information, see Troubleshooting AWS CloudFormation.

Q. I encountered a size-limitation error when I deployed the AWS CloudFormation templates.

A. Launch the Quick Start templates from the links in this guide or from another S3 bucket. If you deploy the templates from a local copy on your computer, or from a location other than an S3 bucket, you might encounter template-size limitations. For more information, see AWS CloudFormation quotas.

Parameter reference

Unless you are customizing the Quick Start templates for your own deployment projects, keep the default settings for the parameters labeled Quick Start S3 bucket name, Quick Start S3 bucket Region, and Quick Start S3 key prefix. Changing these parameter settings automatically updates code references to point to a new Quick Start location. For more information, see the AWS Quick Start Contributor’s Guide.

Parameters for deploying into a new VPC

Table 1. Network configuration
Parameter label (name)	Default value	Description
VPC name (`VPCName`)	`Requires input`	VPC name.
VPC CIDR (`VPCCIDR`)	`10.192.0.0/16`	CIDR block for the VPC.
Public subnet 1 CIDR (`PublicSubnet1CIDR`)	`10.192.10.0/24`	CIDR block for public subnet 1, located in Availability Zone 1.
Public subnet 2 CIDR (`PublicSubnet2CIDR`)	`10.192.11.0/24`	CIDR block for public subnet 2, located in Availability Zone 2.
Private subnet 1 CIDR (`PrivateSubnet1CIDR`)	`10.192.20.0/24`	CIDR block for private subnet 1, located in Availability Zone 1.
Private subnet 2 CIDR (`PrivateSubnet2CIDR`)	`10.192.21.0/24`	CIDR block for private subnet 2, located in Availability Zone 2.
Permitted IP range (`RemoteAccessCIDR`)	`Requires input`	CIDR IP range that is permitted to access the instances.

Table 2. Database configuration
Parameter label (name)	Default value	Description
Database instance name (`DBInstanceName`)	`Requires input`	Name of the database instance.
Database name (`DBName`)	`MyDatabase`	Name of the Amazon PostgreSQL database.
Database primary user name (`DBUser`)	`postgres`	Database administrator account name.
Database primary password (`DBPassword`)	`Requires input`	Database administrator account password.
Database allocated storage (`DBAllocatedStorage`)	`5`	Size of the database in gigabytes (GB).
Database instance class (`DBInstanceClass`)	`db.t2.micro`	Name of the compute and memory capacity class of the database instance.
Database storage type (`DBStorageType`)	`gp2`	The database storage type.
Database backup retention period (`DBBackupRetentionPeriod`)	`7`	Number of days for which to retain automatic database snapshots.
Multi-AZ deployment (`DBMultiAZ`)	`True`	Choose whether to provision a Multi-AZ Amazon RDS instance.
Database auto-minor version upgrade (`DBAutoMinorVersionUpgrade`)	`False`	Enter "true" or "false" to set up auto-minor version upgrade (e.g., 9.6.8 to 9.6.11).

Table 3. ECR configuration
Parameter label (name)	Default value	Description
Amazon Elastic Container Registry name (`ECRRepoName`)	`Requires input`	Name of the Amazon Elastic Container Registry.

Table 4. S3 bucket configuration
Parameter label (name)	Default value	Description
S3 bucket name for Salesforce objects (`S3BucketName`)	`Requires input`	Name of the S3 bucket that stores data for Salesforce objects.
S3 bucket name for Lambda zip files (`LambdaZipBucket`)	`Requires input`	Name of the S3 bucket that stores your Lambda zip files.

Table 5. Amazon AppFlow configuration
Parameter label (name)	Default value	Description
Salesforce connector name (`Connection`)	`Requires input`	Name of the Salesforce connector.
Salesforce objects name (`Objects`)	`Requires input`	Name of the Salesforce objects (e.g., Account, Contact, Lead, Opportunity).

Table 6. AWS Quick Start configuration
Parameter label (name)	Default value	Description
Quick Start S3 bucket name (`QSS3BucketName`)	`aws-quickstart`	S3 bucket name for the Quick Start assets. This string can include numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen (-).
Quick Start S3 key prefix (`QSS3KeyPrefix`)	`quickstart-amazon-appflow/`	S3 key prefix for the Quick Start assets. Quick Start key prefix can include numbers, lowercase letters, uppercase letters, hyphens (-), periods (.) and forward slash (/).
Quick Start S3 bucket Region (`QSS3BucketRegion`)	`us-east-1`	AWS Region where the Quick Start S3 bucket (QSS3BucketName) is hosted. When using your own bucket, you must specify this value.
Quick Start GitHub URL (`GitHubUrl`)	`https://github.com/aws-quickstart/quickstart-amazon-appflow.git`	GitHub URL for the Quick Start assets.
GitHub branch name (`GitHubBranch`)	`main`	Name of the GitHub branch that the code pulls from.

Parameters for deploying additional AppFlows for new Salesforce objects after deploying Amazon Appflow in a new VPC

Table 7. Network configuration
Parameter label (name)	Default value	Description
Private subnet 1 ID (`PrivateSubnet1ID`)	`Requires input`	ID of the private subnet in Availability Zone 1 of your existing VPC (e.g., subnet-fe9a8b32).
Private subnet 2 ID (`PrivateSubnet2ID`)	`Requires input`	ID of the private subnet in Availability Zone 2 of your existing VPC (e.g., subnet-fe9a8b32).
Lambda security group ID (`LambdaSecurityGroupID`)	`Requires input`	ID of your existing Lambda security group.

Table 8. Database configuration
Parameter label (name)	Default value	Description
Database name (`DBName`)	`Requires input`	Name of an existing Amazon PostgreSQL database.
Database primary user name (`DBUser`)	`Requires input`	Name of an existing database administrator account name.
Database primary password (`DBPassword`)	`Requires input`	Enter an existing database administrator password.
Database connection string (`RDSEndpoint`)	`Requires input`	Name of an existing database connection string.

Table 9. IAM role configuration
Parameter label (name)	Default value	Description
Custom resource (IAM role) ARN (`CustomResourceRoleARN`)	`Requires input`	Existing IAM role Amazon Resource Name (ARN) for accessing custom resources.

Table 10. Amazon AppFlow configuration
Parameter label (name)	Default value	Description
Salesforce connector name (`Connection`)	`Requires input`	Name of the Salesforce connector.
Salesforce objects name (`Objects`)	`Requires input`	Name of the Salesforce objects (e.g., Account, Contact, Lead).

Table 11. S3 bucket configuration
Parameter label (name)	Default value	Description
S3 bucket name for Salesforce objects (`S3BucketName`)	`Requires input`	Name of an existing S3 bucket for storing Salesforce objects.
S3 bucket name for Lambda zip files (`LambdaZipBucket`)	`Requires input`	Name of an existing S3 bucket for storing Lambda zip files.

Table 12. Amazon ECR configuration
Parameter label (name)	Default value	Description
Amazon Elastic Container Registry image URI (`ImageURI`)	`Requires input`	Existing image URI.

Table 13. AWS Quick Start configuration
Parameter label (name)	Default value	Description
Quick Start S3 bucket name (`QSS3BucketName`)	`aws-quickstart`	S3 bucket name for Quick Start assets. This string can include numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen (-).
Quick Start S3 key prefix (`QSS3KeyPrefix`)	`quickstart-amazon-appflow/`	S3 key prefix for Quick Start assets. This string can include numbers, lowercase letters, uppercase letters, hyphens (-), periods (.) and forward slashes (/).
Quick Start S3 bucket Region (`QSS3BucketRegion`)	`us-east-1`	Region where the Quick Start S3 bucket (QSS3BucketName) is hosted. When using your own bucket, you must specify this value.

Send us feedback

To post feedback, submit feature ideas, or report bugs, use the Issues section of the GitHub repository for this Quick Start. To submit code, see the Quick Start Contributor’s Guide.

Quick Start reference deployments

See the AWS Quick Start home page.

GitHub repository

Visit our GitHub repository to download the templates and scripts for this Quick Start, to post your comments, and to share your customizations with others.

Notices

This document is provided for informational purposes only. It represents AWS’s current product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS’s products or services, each of which is provided “as is” without warranty of any kind, whether expressed or implied. This document does not create any warranties, representations, contractual commitments, conditions, or assurances from AWS, its affiliates, suppliers, or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.

The software included with this paper is licensed under the Apache License, version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the accompanying "license" file. This code is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either expressed or implied. See the License for specific language governing permissions and limitations.