Headend System Simulator on AWS

Partner Solution Deployment Guide

QS

May 2023
Michael Litz, Tony Bulding, Sascha Janssen, and John Mousa, Amazon Web Services

Refer to the GitHub repository to view source files, report bugs, submit feature ideas, and post feedback about this Partner Solution. To comment on the documentation, refer to Feedback.

This Partner Solution was created by Amazon Web Services, Inc. in collaboration with Amazon Web Services (AWS). Partner Solutions are automated reference deployments that help people deploy popular technologies on AWS according to AWS best practices. If you’re unfamiliar with AWS Partner Solutions, refer to the AWS Partner Solution General Information Guide.

Overview

This guide covers the information you need to deploy the Headend System Simulator Partner Solution in the AWS Cloud.

Costs and licenses

There is no cost to use this Partner Solution, but you will be billed for any AWS services or resources that this Partner Solution deploys. For more information, refer to the AWS Partner Solution General Information Guide.

Architecture

Deploying this Partner Solution with default parameters builds the following Headend System Simulator environment in the AWS Cloud.

Architecture
Figure 1. Partner Solution architecture for Headend System Simulator on AWS

As shown in Figure 1, this Partner Solution sets up the following:

  • An Amazon API Gateway readings resource with GET method that returns paginated smart-meter readings.

  • Four AWS Lambda functions to do the following:

    • Query Amazon Timestream and return paginated results to utility supplier.

    • Insert asynchronous file request messages using a POST method into an Amazon Simple Queue Service (Amazon SQS) queue and update the status in an Amazon DynamoDB table.

    • Return the details of file requests to utility supplier using a GET method.

    • Query Timestream for number of records requested to pass right-sized parameters to an AWS Glue job.

  • An Amazon SQS queue that invokes a worker Lambda function.

  • An AWS Glue job that does the following:

    • Uses a Java Database Connectivity (JDBC) driver to connect to Timestream.

    • Combines query results into a single file and uploads to the meter readings Amazon Simple Storage Service (Amazon S3) bucket.

    • In the DynamoDB table, marks the file request status as complete and record the location of the results file in the meter readings S3 bucket.

  • A Timestream database and table that contains utility smart-meter readings.

  • An S3 bucket that contains the readings files exported by AWS Glue.

  • AWS Transfer for SFTP (AWS SFTP) to transfer meter readings files.

  • AWS Secrets Manager to store the private certificate used by AWS SFTP.

  • (Optional) The Device Data Generator Partner Solution to generate and insert simulated smart-meter readings into Amazon Timestream at regular intervals. (not shown)

Deployment options

This Partner Solution provides the following deployment options:

Predeployment steps

This Partner Solution uses Docker to build the AWS Lambda functions. Install the latest version of Docker Desktop.

Deployment steps

  1. Sign in to your AWS account, and launch this Partner Solution, as described under Deployment options. The AWS CloudFormation console opens with a prepopulated template.

  2. Choose the correct AWS Region, and then choose Next.

  3. On the Create stack page, keep the default setting for the template URL, and then choose Next.

  4. On the Specify stack details page, change the stack name if needed. Review the parameters for the template. Provide values for the parameters that require input. For all other parameters, review the default settings and customize them as necessary. When you finish reviewing and customizing the parameters, choose Next.

    Unless you’re customizing the Partner Solution templates or are instructed otherwise in this guide’s Predeployment section, don’t change the default settings for the following parameters: QSS3BucketName, QSS3BucketRegion, and QSS3KeyPrefix. Changing the values of these parameters will modify code references that point to the Amazon Simple Storage Service (Amazon S3) bucket name and key prefix. For more information, refer to the AWS Partner Solutions Contributor’s Guide.

  5. On the Configure stack options page, you can specify tags (key-value pairs) for resources in your stack and set advanced options. When you finish, choose Next.

  6. On the Review page, review and confirm the template settings. Under Capabilities, select all of the check boxes to acknowledge that the template creates AWS Identity and Access Management (IAM) resources that might require the ability to automatically expand macros.

  7. Choose Create stack. The stack takes about 5 minutes to deploy.

  8. Monitor the stack’s status, and when the status is CREATE_COMPLETE, the Headend System Simulator deployment is ready.

  9. To view the created resources, choose the Outputs tab.

Postdeployment steps

Output values

After you launch Headend System Simulator, refer to the following output values located on the Outputs tab of the stack, as shown in Figure 2 below.

Outputs
Figure 2. Outputs for Headend System Simulator on AWS

  • apiGatewayInvokeURL: https://<id>.execute-api.<region>.amazonaws.com/prod

  • SecretARN: arn:aws:secretsmanager:<region>:<account>:secret:<stackname>/sftp/private_key-<id>

  • SecretName: <stackname>/sftp/private_key

  • SftpServerAddress: <id>.server.transfer.<region>.amazonaws.com

  • SftpUserName: sftp-user

Verifying meter data generation

If you launched the optional Device Data Generator, records will be added to Amazon Timestream at the interval you defined at stack creation. To check this: . Sign in to the AWS Management Console and open the Amazon Timestream console. . From the left navigation pane, choose Query editor. . Run the below query. If the devices database and readings table values were changed during launch of Headend System Simulator, substitute your custom values.

Amazon Timestream query to get min and max dates for records
select
	min(time) time_min,
    max(time) time_max
from devices.readings

The results of the query should look similar to Figure 3, below.

Outputs
Figure 3. Min/max SQL query results for Headend System Simulator on AWS

Interacting with REST API endpoints

When interacting with the Headend System Simulator REST API, use the output value from apiGatewayInvokeURL as the base URL, as shown in Figure 2 .
Example: {apiGatewayInvokeURL}

Paginated smart meter readings

  • readings_endpoint: readings

  • start_date: 2023-02-27 16:00:00.000000

  • end_date: 2023-02-27 16:40:00.000000

  • page_size: 100

To fetch paginated readings from the REST API, /readings must be appended to the apiGatewayInvokeURL. In addition, the following parameters are required:

Readings GET parameters

  • start_date - Required date representing the start time from which to return smart meter readings.

  • page_size - Optional number of records to return in each response. Default is 10.

  • offset - Optional value used to return subsequent records after initial results.

The initial readings GET calls to the Headend System Simulator REST API should look like the following example:

{apiGatewayInvokeURL}/{readings_endpoint}?start_date={start_date}&page_size={page_size}

See an example response below with the first and last record in the response.

Example response of initial readings GET calls
{
  "filter_params": {
    "start_date": "2023-02-27 16:00:00.000000",
    "offset": 0
  },
  "results": {
    "records_total": 100,
    "last_offset": 100,
    "last_date": "2023-02-27 16:00:00.000000"
  },
  "records": [
    {
        "offset": 1,
        "time": "2023-02-27 16:00:00.000000",
        "arrival_time": "2023-02-27 16:00:00.000000",
        "device_id": "0001b33e-0898-33a6-ac1d-82b59b904034",
        "measure_name": "readings",
        "load": 18,
        "crrnt": 88.926,
        "pf": 0.918,
        "kva": 19.257,
        "kw": 17.674,
        "vltg": 0.217
    },
    ...
    {
        "offset": 100,
        "time": "2023-02-27 16:00:00.000000",
        "arrival_time": "2023-02-27 16:00:00.000000",
        "device_id": "00040f4a-f316-3114-bef0-504a37fbfbc4",
        "measure_name": "readings",
        "load": 12,
        "crrnt": 44.314,
        "pf": 0.913,
        "kva": 9.273,
        "kw": 8.464,
        "vltg": 0.209
    }
  ]
}

For subsequent calls for additional pages, supply the last offset value in the request, as shown below.

{apiGatewayInvokeURL}/{readings_endpoint}?start_date={start_date}&page_size={page_size}&offset=100

See an example response below. Continue repeating the pattern of passing in the last_ofsset value for subsequent pages of results. When the records_total is less than the request offset value, or the number of records is 0, there are no more results to process.

Example response of subsequent readings GET call
{
  "filter_params": {
    "start_date": "2023-02-27 16:00:00.000000",
    "offset": 100
  },
  "results": {
    "records_total": 100,
    "last_offset": 200,
    "last_date": "2023-02-27 16:00:00.000000"
  },
  "records": [
    {
        "offset": 101,
        "time": "2023-02-27 16:00:00.000000",
        "arrival_time": "2023-02-27 16:00:00.000000",
        "device_id": "00a0f4b9-c962-3c30-ae64-1a7e9313da75",
        "measure_name": "readings",
        "load": 15,
        "crrnt": 9.556,
        "pf": 0.958,
        "kva": 2.021,
        "kw": 1.936,
        "vltg": 0.212
    },
    ...
    {
        "offset": 200,
        "time": "2023-02-27 16:00:00.000000",
        "arrival_time": "2023-02-27 16:00:00.000000",
        "device_id": "011e2268-8ec6-3e78-b84e-96425c1b5d50",
        "measure_name": "readings",
        "load": 16,
        "crrnt": 67.688,
        "pf": 0.951,
        "kva": 15.083,
        "kw": 14.35,
        "vltg": 0.223
    }
  ]
}

Bulk meter readings files

  • readingsfile_endpoint: readings/file

To request a bulk readings file, make a POST request by appending /readings/file to the apiGatewayInvokeURL URL, as shown below.

{apiGatewayInvokeURL}/{readingsfile_endpoint}

The body of the post should include start_date and end_date, as shown below:

Readings/file POST body
{
    "start_date": "2023-02-27 16:00:00.000000",
    "end_date": "2023-02-27 16:40:00.000000"
}

The REST API returns a request_id and status for the request. The lifecycle statuses of a file request are "queued" > "submitted" > "running" > "completed" or "failed".

Example response from POST
{
    "request_id": "55e3059d-6274-4d97-89a8-40ca93bcd084",
    "status": "queued"
}

Using the request_id from the previous call, send a GET request to the /readings/file endpoint with a request_id parameter, as shown below.

{apiGatewayInvokeURL}/{readingsfile_endpoint}?request_id=55e3059d-6274-4d97-89a8-40ca93bcd084

The response returns the current status, as well as a record_count to signify the number of records that are contained in the file. When the file request is complete, the status is completed and the sftp_location is present, as shown in the example below.

Example response from GET to readings/file resource
{
    "request_id": "55e3059d-6274-4d97-89a8-40ca93bcd084",
    "status": "completed",
    "sftp_location": "55e3059d-6274-4d97-89a8-40ca93bcd084/run-1664371535713-part-r-00000.gz",
    "record_count": 2920000
}

Downloading readings files

Download readings files from the SFTP file transfer service (provided by the AWS Transfer Family) by authenticating with the private certificate stored in AWS Secrets Manager. For demonstration purposes, we’ll use the AWS Command Line Interface (AWS CLI) to retrieve the secret to a local file, and then use the certificate to download the file to our local machine.

Retrieving the secret to a file

Substitute <SecretValue> for the value displayed in Figure 2.

AWS CLI command to export secret value to a file
$ aws secretsmanager get-secret-value --secret-id <SecretValue> --output text --query SecretString > private.pem
Change file permissions to 400 to make it usable
$ chmod 400 private.pem

Authenticating with the secret

Authenticate to the SFTP service using the SftpUserName and SftpServerAddress, as shown in Figure 2, as well as with the certificate. See the example command below:

$ sftp -i private.pem <SftpUserName>@<SftpSereverAddress>

Downloading file

When authenticated, type get followed by the sftp_location, as demonstrated below.

$ get 55e3059d-6274-4d97-89a8-40ca93bcd084/run-1664371535713-part-r-00000.gz

Troubleshooting

For troubleshooting common Partner Solution issues, refer to the AWS Partner Solution General Information Guide and Troubleshooting CloudFormation.

Customer responsibility

After you deploy a Partner Solution, confirm that your resources and services are updated and configured—including any required patches—to meet your security and other needs. For more information, refer to the Shared Responsibility Model.

Feedback

To submit feature ideas and report bugs, use the Issues section of the GitHub repository for this Partner Solution. To submit code, refer to the Partner Solution Contributor’s Guide. To submit feedback on this deployment guide, use the following GitHub links:

Notices

This document is provided for informational purposes only. It represents current AWS product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS products or services, each of which is provided "as is" without warranty of any kind, whether expressed or implied. This document does not create any warranties, representations, contractual commitments, conditions, or assurances from AWS, its affiliates, suppliers, or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.

The software included with this paper is licensed under the Apache License, version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at https://aws.amazon.com/apache2.0/ or in the accompanying "license" file. This code is distributed on an "as is" basis, without warranties or conditions of any kind, either expressed or implied. Refer to the License for specific language governing permissions and limitations.