IoT Device Connectivity on the AWS Cloud

Quick Start Reference Deployment

March 2021
Geoffroy Rollat, Partner Solutions Architect, Travel & Hospitality
Dylan Owen, AWS Quick Start team

Visit our GitHub repository for source files and to post feedback, report bugs, or submit feature ideas for this Quick Start.

This Quick Start was created by Rigado in collaboration with Amazon Web Services (AWS). Quick Starts are automated reference deployments that use AWS CloudFormation templates to deploy key technologies on AWS, following AWS best practices.

Overview

This Quick Start helps AWS Internet of Things (IoT) customers get started with an IoT landing zone on the AWS Cloud. It sets up the required resources and services for device onboarding along with example dashboards for visualization. Typical use cases for this Quick Start include management of IoT devices for smart kitchens and retail stores.

This architecture is built in coorperation with Rigado, a travel and hospitality AWS Partner. The Quick Start is designed to provide an optimized onboarding experience for users of the Rigado Allegro Kit, although it can be adapted for any device that supports standard MQ Telemetry Transport (MQTT) connectivity.

The Quick Start sets up:

A REST serverless microservice to onboard devices and gateways by serial number. The service creates the AWS IoT Core resources to connect to the MQTT broker.
An IoT data lake for ingesting data from the long-term storage and analytics.
An example Amazon QuickSight dashboard to display data from the data lake (compatible devices only).
An example IoT device real-time monitoring dashboard using AWS IoT SiteWise (compatible devices only).

Amazon may share user-deployment information with the AWS Partner that collaborated with AWS on the Quick Start.

IoT Device Connectivity on AWS

The Rigado Allegro Kit for AWS

The kit helps accelerate IoT proof-of-concept and pilot projects. You can set it up in a few minutes to demonstrate real-world scenarios with live data for a variety of commercial spaces, including retail, travel and hospitality, smart buildings, and more. It uses production sensors and cloud components based on Rigado and AWS IoT reference architectures.

The kit includes:

The Rigado plug-and-play sensor network, including cascade gateways, temperature and humidity sensors, alert buttons, door sensors, asset trackers, and more.
An online Rigado configuration wizard that requires output data from the Quick Start.

AWS costs

You are responsible for the cost of the AWS services and any third-party licenses used while running this Quick Start. There is no additional cost for using the Quick Start.

The AWS CloudFormation templates for Quick Starts include configuration parameters that you can customize. Some of the settings, such as the instance type, affect the cost of deployment. For cost estimates, see the pricing pages for each AWS service you use. Prices are subject to change.

After you deploy the Quick Start, create AWS Cost and Usage Reports to deliver billing metrics to an Amazon Simple Storage Service (Amazon S3) bucket in your account. These reports provide cost estimates based on usage throughout each month and aggregate the data at the end of the month. For more information, see What are AWS Cost and Usage Reports?

Software licenses

No licenses are required to deploy this Quick Start. All AWS service resources that are consumed during the launch of the Quick Start incur AWS service usage costs.

Architecture

Deploying this Quick Start into an AWS Region with default parameters builds the following IoT Device Connectivity environment in the AWS Cloud.

Figure 1. Quick Start architecture for IoT Device Connectivity on AWS

As shown in Figure 1, the Quick Start sets up two planes—a control plane to onboard devices and a data plane to ingest sensor data.

For the control plane:

A serverless microservice for onboarding IoT devices through a REST API call.
IoT security certificates and MQTT endpoints for the onboarding microservice to use when sending device traffic.
Amazon API Gateway to expose three REST endpoints for creating, retrieving, and deleting onboarded devices.
Amazon Cognito to secure the onboarding microservice. The Quick Start generates a refresh token so users can obtain a session token from Amazon Cognito.
A Lambda function to host a service for creating, retrieving, and deleting onboarded devices.
An Amazon S3 bucket to store IoT certificates.
Amazon DynamoDB to store metadata from onboarded devices.

For the data plane:

A device gateway to allow the AWS Cloud to ingest traffic from IoT devices.
An IoT MQTT protocol so data can be sent to the MQTT endpoint from the onboarding microservice.
AWS IoT Core, including an MQTT broker that securely receives traffic from authorized devices that are filtered by the IoT topic. AWS IoT Core uses the following resources:
- An IoT rule to forward traffic to both the AWS IoT SiteWise Monitor and the IoT data lake. You can add more rules to extend the Quick Start for other use cases.
- An IoT topic to filter messages. The topic is configured within the AWS CloudFormation template.
AWS IoT SiteWise to monitor data in real time.
An IoT data lake to ingest cold data for posteriori monitoring. By default, the data lake is configured to display data from the last 24 hours.
- Amazon Kinesis Data Firehose to move the data into Amazon S3.
- Amazon S3 to store raw data ingested from the sensors along with refined data processed by the extract, transform, load (ETL) script.
- AWS Glue to host the data catalog, crawlers, and serverless ETL jobs for the IoT data lake.
- Amazon Athena to query the sensor data for display in dashboards.
- Amazon QuickSight to display the sensor data in a preconfigured dashboard. You can configure additional dashboards if desired.

Planning the deployment

Specialized knowledge

This deployment requires a moderate level of familiarity with AWS services. If you’re new to AWS, see Getting Started Resource Center and AWS Training and Certification. These sites provide materials for learning how to design, deploy, and operate your infrastructure and applications on the AWS Cloud.

This Quick Start also assumes familiarity with AWS IoT, AWS Sitewise, AWS Glue, AWS QuickSight.

AWS account

If you don’t already have an AWS account, create one at https://aws.amazon.com by following the on-screen instructions. Part of the sign-up process involves receiving a phone call and entering a PIN using the phone keypad.

Your AWS account is automatically signed up for all AWS services. You are charged only for the services you use.

Technical requirements

Before you launch the Quick Start, review the following information and ensure that your account is properly configured. Otherwise, deployment might fail.

Resource quotas

If necessary, request service quota increases for the following resources. You might request quota increases to avoid exceeding the default limits for any resources that are shared across multiple deployments. The Service Quotas console displays your usage and quotas for some aspects of some services. For more information, see What is Service Quotas? and AWS service quotas.

Resource	This deployment uses
CodeBuild concurrent builds	2

Resource

This deployment uses

CodeBuild concurrent builds

Supported Regions

This Quick Start supports all Regions supported by the followings services:

AWS IoT Sitewise
AWS IoT Core
Amazon QuickSight
Amazon Simple Email Service (SES)

Please refer to the dedicated service regional availability to validate that a specific region is included.

Certain Regions are available on an opt-in basis. For more information, see Managing AWS Regions.

IAM permissions

Before launching the Quick Start, you must sign in to the AWS Management Console with IAM permissions for the resources that the templates deploy. The AdministratorAccess managed policy within IAM provides sufficient permissions, although your organization may choose to use a custom policy with more restrictions. For more information, see AWS managed policies for job functions.

Prepare your AWS account

Activate AWS SSO (optional)

AWS Single Sign-On (AWS SSO) is optional if you don’t want to use an AWS IoT SiteWise dashboard.

AWS SSO provides identity federation for AWS IoT SiteWise Monitor so that you can control access to your portals. With AWS SSO, users sign in with their corporate email and password instead of an AWS account. For more information, see Getting started with AWS IoT SiteWise Monitor and follow the steps in the Enabling AWS SSO section.

Create a QuickSight account (optional)

Creating a QuickSite account is optional if you already have an account or you don’t want to use a QuickSight dashboard.

To sign up for QuickSight, see Signing Up for an Amazon QuickSight Subscription. If you plan to deploy the default dashboard, you must have a QuickSight Enterprise account.

Validate your email address with Amazon SES

This Quick Start uses the email address that you provide in the AWS CloudFormation template for sending and receiving email notifications. These notifications provide the key credentials to use the device onboarding microservices. For users of the Rigado Allegro Kit, the email provides the data required to use the Rigado wizard to automatically onboard the Rigado Gateway. To use the email address, Amazon Simple Email Service (Amazon SES) requires you to verify the email address. See Verifying email addresses in Amazon SES for more information.

Enable logging for AWS IoT Core

Enabling logging for AWS IoT Core helps you troubleshoot device connectivity. Troubleshooting is especially useful if you are not using a Rigado device. For more information, see Configure AWS IoT logging.

GitHub fork and access token

This QuickStart deploys a CICD Pipeline using AWS Code Pipeline from source code hosted on GitHub. As AWS CodePipeline requires private access to GitHub Repositories, you must have a GitHub account and have completed the following steps:

Log in to GitHub and fork the quickstart-iot-device-connectivity repository.
Create a personal access token for your GitHub Account.
Save the token and user name for use in the deployment parameters.

Deployment options

This Quick Start deploys a continuous delivery and/or continuous deployment (CICD) pipeline using AWS CodePipeline.

If you prefer not to sign up with Amazon QuickSight, you can exclude the QuickSight dashboard from the deployment by omitting the QuickSight ADMIN user name on the AWS CloudFormation template.

Deployment steps

Sign in to your AWS account

Sign in to your AWS account at https://aws.amazon.com with an IAM user role that has the necessary permissions. For details, see Planning the deployment earlier in this guide.
Make sure that your AWS account is configured correctly, as discussed in the Technical requirements section.

Launch the Quick Start

This Quick Start was built using AWS Cloud Development Kit (AWS CDK) and is deployed via a synthesized template.

The deployment takes about 15 minutes to complete.

Deploy IoT Device Connectivity on AWS

View template

Check the AWS Region that’s displayed in the upper-right corner of the navigation bar, and change it if necessary. This Region is where the network infrastructure for IoT Device Connectivity is built. The template is launched in the us-east-1 Region by default.

On the Create stack page, keep the default setting for the template URL, and then choose Next.
On the Specify stack details page, change the stack name if needed. Review the parameters for the template. Provide values for the parameters that require input. For all other parameters, review the default settings and customize them if needed. For details on each parameter, see the section below. When you finish reviewing and customizing the parameters, choose Next.
On the Configure stack options page, you can specify tags (key-value pairs) for resources in your stack and set advanced options. When you’re finished, choose Next.
On the Review page, review and confirm the template settings. Under Capabilities, select the two check boxes to acknowledge that the template creates IAM resources and might require the ability to automatically expand macros.
Choose Create stack to deploy the stack.
Monitor the status of the stack. When the status is CREATE_COMPLETE, the IoT Device Connectivity deployment is ready.

View the CDK deployment using AWS CodePipeline

Once the AWS CloudFormation template has deployed successfully, open AWS CodePipeline to see the pipeline running.

Figure 2. AWS CodePipeline console

If you choose the pipeline name, you can see the steps of the pipeline running:

Figure 3. AWS CodePipeline console

After the pipeline has deployed the CDK code, open the CloudFormation console, choose the template (it will be named something similar to IOTOnboardingInfraStackint), and then choose Outputs to view the created resources.

Figure 4. Outputs after pipeline CDK deployment

Connecting devices

Rigado devices

If you are using Rigado devices using the Rigado Allegro Kit, go to the Rigado wizard and enter the data that you received from the Quick Start script. The email that you received after the devices were activated provides further instructions.

AWS IOT Connectivity QuickStart Output Values


--------------------------------------------------------------------------------------------
| Cognito URL | https://iot-onboarding-quickstart-<env>.auth.us-east-1.amazoncognito.com/oauth2/token
--------------------------------------------------------------------------------------------
| API Gateway URL | https://<api_id>.execute-api.us-east-1.amazonaws.com/
--------------------------------------------------------------------------------------------
| Client ID | 228v...t9c3
--------------------------------------------------------------------------------------------
| Refresh Token | eyJjdHkiOiJKV1QiLCJl...slN29FrDNqHWo_0e5U85ow

After the wizard finishes, the Rigado gateway is configured to send traffic automatically and securely to AWS IoT Core using MQTT.

The build script creates a new Amazon Cognito user each time it runs. If you are using the pipeline to manage your own CICD process, you should clean up the list of users in your Amazon Cognito user pool to avoid unnecessary charges. Only one active user is required to access the API Gateway to onboard devices.

Non-Rigado devices

For non-Rigado devices (or devices not supported by the Rigado Allegro Kit), manually configure your device to allow it to communicate with AWS IoT Core. The Lambda function deployed by the Quick Start lets you generate the device certificate and key pair that’s used for the help secure communication between the device and AWS IoT Core. The function also creates the IoT Thing and appropriate policy, and it returns:

The MQTT endpoint of the AWS IoT Core broker.
The certificate generated during the service call.
The key pair (public and private key) generated during the service call.
Other information about the device.

The following example is an exchange with the onboarding service. First, the session token is obtained using the client ID and the refresh token.

--header 'Content-Type: application/x-www-form-urlencoded'
{
    "id_token": "<id_token>",
    "access_token": "<access_token>",
    "expires_in": 3600,
    "token_type": "Bearer"
}

Next is a call to the onboarding service to create the device.

curl --location --request POST '<api_gateway_url>/api/onboard/<device_serial_number> \
--header 'Authorization: Bearer <access_token>'
{
    "serialNumber": "<device_serial_number> ",
    "deviceName": "<device_serial_number> ",
    "thingId": "<iot_thing_id>",
    "credential": {
        "certificateId": "<certificate_id>",
        "certificatePem": "<certificate data>",
        "privateKey": "<private key data>",
        "publicKey": "<public key data>"
    },
    "mqttEndpoint": "https://data.iot.<region>.amazonaws.com",
    "error": {
        "code": "",
        "msg": "",
        "type": ""
    }
}

The response contains the required certificates and keys for configuring the device. For information about configuring IoT devices to communicate with AWS, see Connecting to AWS IoT Core.

Checking connectivity

To validate that traffic flows correctly, go to the AWS IoT Core console and subscribe to the MQTT topic that’s provided in the CloudFormation template (by default, data/# for Rigado). As long as at least one sensor is turned on, you should see messages flowing, as shown below.

Figure 5. Testing device connectivity on AWS IoT Core

After you have validated that traffic is flowing, now you are ready to visualize the data (Rigado Allegro Kit users only).

Visualizing the data (Rigado Allegro Kit users only)

As described in the Architecture section above, the data sent to the AWS IoT Core MQTT broker is ingested by an IoT data lake for a posterior visualization and by AWS IoT SiteWise for live monitoring.

Visualizing the data using Amazon QuickSight

To speed up your IoT project, this Quick Start deploys a predefined dashboard within Amazon QuickSight. To benefit from this feature, you need the following:

An Amazon QuickSight Enterprise customer.
Use of the Rigado Allegro Kit for your devices.

While the AWS Glue ETL service that ingests and processes the data is not device-agnostic, the dashboard makes assumptions about the names of the fields received and therefore works only for Rigado Allegro Kit users. By default, the data is processed by a scheduled glue crawler and ETL service every 24 hours. This default value helps minimize the cost of running the IoT data lake and can be updated easily by changing a cron expression in the AWS CDK script or directly from the AWS Glue console. The Quick Start also creates a glue trigger that can start the data refresh on demand directly from the AWS Glue console.

When accessing the Amazon QuickSight dashboard for the first time, you must provide access to the Amazon S3 bucket that contains the refined data (the data processed by the ETL script). For instructions on giving Amazon QuickSight access to the Amazon S3 bucket, see I Can’t Connect to Amazon Athena. After deploying the Quick Start, you should see an analysis called Rigado Quick Start Dashboard in your Amazon QuickSight account, as shown below.

Figure 6. Newly created Analysis in Amazon QuickSight

This dashboard is configured to query 48 hours of data in the past. This setting limits cost and improves dashboard load time as the quantity of data increases in the future. To learn how to change this setting while scaling with large amounts of data, you can use Amazon QuickSight Super-fast, Parallel, In-memory Calculation Engine (SPICE). For more information, see Importing data into SPICE. Note that using SPICE incurs additional cost.

Using pushdown predicates, the AWS Glue ETL service that processes data into a flat structure also queries only 48 hours of data in the past. You can change this setting with a minor update to the Python script that is available from the AWS Glue console. For more information, see Managing Partitions for ETL Output in AWS Glue.

If you are not a Rigado Allegro Kit user, you must create your own analysis and data source that targets the Athena Table for refined data. The Glue job that refines the data is device-agnostic, as it just flattens the JSON-nested fields. However, it might not lead to practical results for deeply nested data.

Visualizing the data using AWS IoT SiteWise

This Quick Start creates an AWS IoT SiteWise assets model hierarchy composed of one root asset model and four children assets models. It also creates a portal. To start visualizing the data in the portal, perform the following steps:

Go to AWS IoT SiteWise and select Build > Models.
Choose the asset model that corresponds to your Rigado device. If your device does not correspond to an existing asset model, see the AWS IoT SiteWise documentation to create a dedicated asset model and route the traffic of your device through the appropriate alias using AWS IoT Core.
Create an asset under this asset model using the device ID in the device name.
Once created, choose Edit and provide a property alias for each of the model measurements. For consistency with the AWS IoT Core broker rule, the alias value must be as follows:

<deviceId><MeasurementNameWithoutDoubleQuotes>

See the following example for a device with ID ffcfed4dd3ab:

Figure 7. Setting up AWS IoT SiteWise property alias

Repeat these steps for all devices that send traffic behind the Rigado gateway. Using the Amazon QuickSight dashboard, you can view a list of all devices that send traffic though the gateway and use it to configure live monitoring with AWS IoT SiteWise.

After the asset is created, you can access the portal created by the Quick Start or create a portal from scratch following the AWS IoT SiteWise documentation. Then you can add your assets to the dedicated dashboards.

Now, you can use the portal to design dashboards for your devices.

If you are not a Rigado Allegro Kit user, you must create your own AWS IoT Core broker rule to ingest the properly formatted data into AWS IoT SiteWise. You can follow the same model as the one created in the Quick Start. You must also manually create the assets models and assets following the AWS IoT SiteWise documentation.

Cleaning up

This Quick Start uses a combination of command line interface (CLI) and CDK for deploying AWS resources. This is because some services such as Amazon QuickSight and AWS IoT SiteWise are not yet supported by AWS CloudFormation. Consequently, the following steps are required to clean up the deployed resources in the user account:

Identify the buckets created by the stack, which are prefixed by iotonboardinginfrastack, and delete their contents before deleting the stack.
Go to the AWS CloudFormation console and delete the infrastructure stack, starting with IoTOnboardingInfraStack.
Delete the AWS CodePipeline stack that you created.
Clean up the Amazon QuickSight dashboard by manually deleting resources. If you created an Amazon QuickSight account just for this Quick Start, you can unsubscribe to the service.
Clean up the AWS IoT SiteWise dashboard by deleting the following resources:
- AWS IoT SiteWise assets
- AWS IoT SiteWise assets models (the Quick Start creates one root asset model and four child asset models)
- AWS IoT SiteWise projects and dashboards

FAQ

Q. I encountered a CREATE_FAILED error when I launched the Quick Start.

A. If AWS CloudFormation fails to create the stack, relaunch the template with Rollback on failure set to Disabled. This setting is under Advanced in the AWS CloudFormation console on the Configure stack options page. With this setting, the stack’s state is retained, and the instance keeps running so that you can troubleshoot the issue. (For Windows, look at the log files in %ProgramFiles%\Amazon\EC2ConfigService and C:\cfn\log.)

When you set Rollback on failure to Disabled, you continue to incur AWS charges for this stack. Delete the stack when you finish troubleshooting.

For more information, see Troubleshooting AWS CloudFormation.

Q. I encountered a size-limitation error when I deployed the AWS CloudFormation templates.

A. Launch the Quick Start template from the link in this guide or from another S3 bucket. If you deploy the templates from a local copy on your computer or from a location other than an S3 bucket, you might encounter template-size limitations. For more information, see AWS CloudFormation quotas.

Q. My Amazon QuickSight dashboard displays errors in the widgets.

A. Errors displayed in the widgets of the QuickSight dashboard typically result from one of the following causes:

You do not have access to the S3 bucket created by the CDK script to store the data solution. To provide access to the bucket, see the Amazon QuickSight product documentation.
No data is available in the data lake. The data is either not yet crawled or received. To resolve this issue, if the data has already been received, you can manually invoke the on-demand trigger from the AWS Glue console.

Q. My QuickSight deployment script fails with an error.

A. This error can occur when the deployment of a previous version has been only partially completed. As a result, some resources already exist and others not yet. To resolve this issue, go to QuickSight and delete the existing dashboard and dataset and then retry the AWS CodePipeline steps.

Q. My CodePipeline action fails with errors related to the concurrent build.

A. Some AWS accounts are by default configured to allow only one concurrent build using CodePipeline. To fix the issue, you can retry the CodePipeline steps for the failed build. As a long-term solution, request an AWS CodeBuild service limit increase from AWS Support.

Q. My AWS IoT SiteWise script fails with an ResourceAlreadyExistsException error.

A. To resolve the following error:

An error occurred (ResourceAlreadyExistsException) when calling the CreateAssetModel operation: Another resource is already using the name `RigadoHoboMX100QsTestint`.

Delete the AWS IoT SiteWise resources and retry the CodePipeline steps.

Q. I am using the Rigado Allegro Kit and I don’t see a column in my Amazon QuickSight dataset.

A. Data takes up to one day to propagate to the IoT data lake due to the crawler schedule. You can confirm that data is flowing by using AWS IoT Core Monitor and manually triggering the crawlers and jobs using the ON_DEMAND trigger from the AWS Glue console.

Q. Deleting the infrastructure stack fails.

A. The Amazon S3 buckets may still contain data. If so, they are not removed to avoid unintended data loss. Manually empty and delete S3 buckets before retrying the stack deletion.

Q. Deleting AWS IoT SiteWise portal fails.

A. A project associated with the portal may still exist. Ff you can log in as administrator, you can delete the project from the created portal. If you are unable to log in as an administrator, you can delete the underlying project using the AWS IoT SiteWise CLI.

Parameter reference

Unless you are customizing the Quick Start templates for your own deployment projects, keep the default settings for the parameters labeled Quick Start S3 bucket name, Quick Start S3 bucket Region, and Quick Start S3 key prefix. Changing these parameter settings automatically updates code references to point to a new Quick Start location. For more information, see the AWS Quick Start Contributor’s Guide.

Parameters for launching the deployment pipeline

Table 1. IoT Device Connectivity
Parameter label (name)	Default value	Description
Email address (`contactEmail`)	`Requires input`	(Optional) Email address for the administrator. This is also used for the AWS IoT SiteWise portal creation.
QuickSight user name (`quickSightAdminUserName`)	`Requires input`	(Optional) User name of an Amazon QuickSight user with an administrator role. If left blank, the QuickSight dashboard will not be included.
QuickSight user Region (`quickSightAdminUserRegion`)	`Requires input`	The Region of the above QuickSight user (for example, us-east-1).
QuickSight dashboard ARN (`sourceTemplateArn`)	`Requires input`	(Optional) Amazon Resource Name (ARN) of a public QuickSight dashboard. If using Rigado Allegro Kit, use ‘arn:aws:quicksight:eu-central-1:660526416360:template/iotOnboardingRigadoQuicksightPublicTemplatedev’ for an example dashboard.
MQTT topic (`rootMqttTopic`)	`data/#`	The root MQTT topic to which devices publish data. Leave the default (data/#) if using the Rigado Allegro Kit. If using your own devices, create your own dataset, analysis, and dashboard based on your devices.
Environment name (`environment`)	`int`	Your environment name. Change to a unique name only if deploying the stack multiple times in the same Region and account.
GitHub user name (`gitHubUserName`)	`Requires input`	GitHub user name.
GitHub token (`githubtoken`)	`Requires input`	GitHub personal access token allowing access to the forked repository.

Send us feedback

To post feedback, submit feature ideas, or report bugs, use the Issues section of the GitHub repository for this Quick Start. To submit code, see the Quick Start Contributor’s Guide.

Quick Start reference deployments

See the AWS Quick Start home page.

GitHub repository

Visit our GitHub repository to download the templates and scripts for this Quick Start, to post your comments, and to share your customizations with others.

Notices

This document is provided for informational purposes only. It represents AWS’s current product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS’s products or services, each of which is provided “as is” without warranty of any kind, whether expressed or implied. This document does not create any warranties, representations, contractual commitments, conditions, or assurances from AWS, its affiliates, suppliers, or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.

The software included with this paper is licensed under the Apache License, version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the accompanying "license" file. This code is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either expressed or implied. See the License for specific language governing permissions and limitations.