Amazon Marketing Cloud Insights on AWS

Partner Solution Operational Guide

QS

November 2022
Arpan Shah, Ronak Parikh, and Chaitra Nayak, Amazon Ad Tech Solutions
Ramesh Jetty, Brian Maguire, and Gerry Louw, AWS Solutions Architecture | Bret Pontillo, Noah Paige, Naomi Joshi and Anmol Gandhi, AWS Professional Services | Eric Beard, AWS Partner Core | Chip Reno and Clark Fredricksen, AWS Advertising & Marketing Industry
Troy Ameigh, AWS Integration & Automation

Visit our GitHub repository to view source files, report bugs, submit feature ideas, and post feedback about this Partner Solution. To comment on the documentation, refer to Feedback.

This Partner Solution was created by the Amazon Ad Tech Solutions team in collaboration with Amazon Web Services (AWS). Partner Solutions are automated reference deployments that help people deploy popular technologies on AWS according to AWS best practices. If you’re unfamiliar with AWS Partner Solutions, refer to the AWS Partner Solution General Information Guide.

Overview

This guide covers the information you need to use your deployed Amazon Marketing Cloud (AMC) Insights on AWS solution to do such things as build custom analytics for your data lake and use the multitenant and workflow-management features.

For deployment instructions, refer to the Amazon Marketing Cloud Insights on AWS Partner Solution Deployment Guide.

About the deployed microservices

This solution deploys three microservices, which you can access from the platform manager in the deployed Amazon SageMaker instance.

  • Platform Management Service (PMS)

  • Tenant Provisioning Service (TPS)

  • Workflow Management Service (WFM)

The PMS microservice uses the platform management library to interact with the other two microservices: Tenant Provisioning Service and Workflow Management Service.

The TPS microservice is an Amazon Ad Tech solution that adds functionality for managing multiple AMC customers. It provides a centralized location to manage customers, supporting multitenancy and an ability to onboard customers without the need for custom solutions. Customers are mapped one-to-one with AMC instances. When a customer is onboarded, if an Amazon Simple Storage Service (Amazon S3) bucket doesn’t already exist for the associated AMC instance, one is deployed.

With the WFM microservice, you can do the following:

  • Synchronize workflows and their schedules in the Workflow Library service with multiple AMC instances.

  • Send workflow requests to an Amazon Simple Queue Service (Amazon SQS) queue rather than directly to the AMC endpoint to prevent timeout failures when there are many requests in a short time.

  • Schedule with dynamic, relative time windows rather than using AMC’s scheduling feature, which allows only predefined scheduled reporting, such as daily or weekly.

  • Track the status of all workflow runs for customer AMC instances, whether they’re submitted through WFM or other means, such as Postman. You can also track historical runs for troubleshooting or performance monitoring.

Hydrate your data lake

Hydrate the data lake by populating it with data from your AMC instance. When you’re finished, business stakeholders can use Amazon Athena to access the data returned by the workflow run.

  1. Sign in to the AWS Management Console, and open the Amazon SageMaker console.

  2. In the navigation pane, under Notebook, choose Notebook Instances.

  3. Choose Open JupyterLab next to the deployed instance named aws-quickstart-platform-manager-notebooks. (The prefix, aws-, may differ depending on the value set in ddk.json.) This notebook instance opens in a new tab.

  4. Open -Getting_Started-.ipynb, and walk through using the TPS and WFM microservices.

If your AMC S3 bucket contained data before deployment of this solution, the data lake does not pick up that data automatically. You must run workflows after deployment to start this process.

Query your data with Athena

As part of the solution, an Athena workgroup is automatically created. When it’s used, your query results are stored in the -athena S3 bucket.

  1. Open the Athena console.

  2. From the sidebar, choose Query editor.

  3. From the top menu, choose Workgroup, wfm-.

You are now ready to query your tables with Athena.

Build a QuickSight dashboard

This section walks through how to build an Amazon QuickSight dashboard with AMC data from your data lake. For more information on using QuickSight features, refer to What is Amazon QuickSight?

Create a QuickSight account

To build your first dashboard, you must create a QuickSight account. If you do not have a QuickSight account already, create one by following these steps:

  1. From the AWS Management Console, search for and choose QuickSight.

  2. On the QuickSight page, choose Sign up for QuickSight.

  3. Keeping the default Enterprise edition, scroll down and choose Continue.

  4. Enter a QuickSight account name and a notification email address.

  5. Scroll down and choose Finish. The QuickSight console opens.

Authorize QuickSight to access AWS services

For QuickSight to access Athena, Amazon S3, and AWS Lake Formation, a QuickSight administrator must configure the AWS resource permissions. These permissions apply to all QuickSight users. If you’re a QuickSight administrator (in which case you see the Manage QuickSight option when you open the menu from your profile at the upper right), you can authorize QuickSight access to AWS services by doing the following two procedures.

Authorize QuickSight to access Athena and Amazon S3

  1. In the QuickSight console, choose your profile name (upper right) and choose Manage QuickSight.

  2. Navigate to Security & Permissions.

  3. Under QuickSight access to AWS services, choose Manage.

  4. Find Athena in the list. Clear the box by Athena, and then select it again to enable Athena. Then choose Next.

  5. Under S3 Bucket, choose -stage to grant QuickSight read access.

  6. Choose Finish, and save your settings.

Authorize QuickSight to access Lake Formation database and tables

  1. Find the Amazon Resource Names (ARNs) of the QuickSight users and groups that need access to Lake Formation data. Collect these ARNs, ensuring that the users are QuickSight authors or administrators.

    • To construct ARNs manually, use this string, replacing the bracketed items:

      arn:aws:quicksight:<REGION_NAME>:<ACCOUNT_ID>:user/default/<QUICKSIGHT_USERNAME>

    • To collect ARNS programmatically, run the following command in your terminal (Linux or macOS) or at your command prompt (Windows), replacing the information in brackets:

      aws quicksight list-users --region <aws-region> --aws-account-id <account-id> --namespace default

  2. Open the Lake Formation console as the data-lake administrator.

  3. Choose Databases, and select the database to which you want to grant your QuickSight user access. Then, for Actions, choose Grant.

  4. Select SAML users and groups, and enter the QuickSight user ARN.

  5. Choose Named data catalog resources.

  6. For Tables, select All tables, or select individual tables to which you want to grant your user access. Then for Table permissions, choose Select and Describe. Then choose Grant.

  7. Repeat the preceding steps to grant permissions to other users or groups.

Create a dataset in QuickSight

After you’ve authorized QuickSight to access AWS services, as described earlier, you can create custom datasets in QuickSight using Athena as follows:

  1. In the QuickSight console, in the navigation pane, choose Datasets, and then choose New dataset.

  2. Create an Athena connection profile.

    1. Under FROM NEW DATA SOURCES, choose the Athena data source card.

    2. For Data source name, enter a descriptive name.

    3. For Athena Workgroup, choose [primary].

    4. Choose Validate connection to test the connection.

    5. Choose Create data source.

  3. Choose your table.

    1. On the Choose your table screen, under Catalog, choose AwsDataCatalog.

    2. Do one of the following:

      • Select the database and table manually from the dropdown.

      • Choose Use custom SQL to pull data in with a Structured Query Language (SQL) query.

    3. Choose Select or Confirm Query, depending on the option chosen earlier.

    4. Choose Visualize.

Now you can create, publish, and share your custom dashboard.

Delete deployed resources

When you no longer need the architecture that was deployed by this solution, delete the resources from your AWS account so that you’re no longer charged for them. These resources include S3 buckets, AWS CloudFormation stacks, DataOps Development Kit (DDK) bootstrap, AWS CodeCommit repos, AWS Key Management Service (AWS KMS) keys, AWS Lambda layers, and SQS queues and rules.

To delete all these resources, follow these steps:

  1. Look into Makefile.

    $ cd quickstart-amazon-marketing-cloud
$ cat MakeFile

  2. Verify that the following functions are passing the correct stack names. Replace the information in brackets.

    • The delete_repositories function is passing -d <AMC_REPO_NAME> (default: ddk-amc-quickstart).

    • The delete_bootstrap function is passing --stack-name <BOOTSTRAP_STACK_NAME> (default: DdkDevBootstrap).

  3. Enter the following command:

    $ make delete_all

Some CloudWatch general log groups may remain in your account with logs specific to this solution’s resources. Examples:

  • /aws/sagemaker/NotebookInstances

  • /aws-glue/jobs/error

  • /aws-glue/jobs/output

Feedback

To submit feature ideas and report bugs, use the Issues section of the GitHub repository for this Partner Solution. To submit code, refer to the Partner Solution Contributor’s Guide. To submit feedback on this deployment guide, use the following GitHub links:

Notices

This document is provided for informational purposes only. It represents current AWS product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS products or services, each of which is provided "as is" without warranty of any kind, whether expressed or implied. This document does not create any warranties, representations, contractual commitments, conditions, or assurances from AWS, its affiliates, suppliers, or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.

The software included with this paper is licensed under the Apache License, version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at https://aws.amazon.com/apache2.0/ or in the accompanying "license" file. This code is distributed on an "as is" basis, without warranties or conditions of any kind, either expressed or implied. Refer to the License for specific language governing permissions and limitations.