This guide provides instructions for deploying the Utility Meter Data Analytics Quick Start reference architecture on the AWS Cloud.

This Quick Start deploys a platform that uses machine learning (ML) to help you analyze data from smart utility meters. It’s for utility companies or other organizations that are looking to gain insights from smart-meter data. This data comes from meter data management (MDM) or similar systems. Insights include unusual energy usage, energy-usage forecasts, and meter-outage details.

Except for Amazon Redshift, this is a serverless architecture.

The Utility Meter Data Analytics Quick Start deploys a serverless architecture to ingest, store, and analyze utility-meter data. It creates an AWS Glue workflow, which consists of AWS Glue triggers, crawlers, and jobs as well as the AWS Glue Data Catalog. This workflow converts raw meter data into clean data and partitioned business data.

Next, this partitioned data enters an ML pipeline, which uses AWS Step Functions and Amazon SageMaker. First, an AWS Step Functions model-training workflow creates an ML model. Then another AWS Step Functions workflow uses the ML model to batch process the meter data. The processed data becomes the basis for forecasting energy usage, detecting usage anomalies, and reporting on meter outages. The Quick Start provides a Jupyter notebook that you can use to do data science and data visualizations on a provided sample dataset or run queries on your own meter data.

This Quick Start also creates application programming interface (API) endpoints using Amazon API Gateway. API Gateway provides REST APIs that deliver the query results.

You are responsible for the cost of the AWS services and any third-party licenses used while running this Quick Start. There is no additional cost for using the Quick Start.

The AWS CloudFormation templates for Quick Starts include configuration parameters that you can customize. Some of the settings, such as the instance type, affect the cost of deployment. For cost estimates, see the pricing pages for each AWS service you use. Prices are subject to change.

This Quick Start doesn’t require any software license or AWS Marketplace subscription.

Deploying this Quick Start for a new virtual private cloud (VPC) with default parameters builds the following Utility Meter Data Analytics environment in the AWS Cloud.

As shown in Figure 1, the Quick Start sets up the following:

*The template that deploys the Quick Start into an existing VPC skips the components marked by asterisks and prompts you for your existing VPC configuration.

This Quick Start features a MRASCo data adapter option to transform MRASCo data flow files to MDA landing zone format. To deploy this feature, choose mrasco for the DeploySpecialAdapters parameter during deployment. The MRASCo data adapter option deploys the following resources:

To see the capabilities of the QuickStart, follow the steps in this section to use the dataset of meter reads from the City of London between 2011 and 2014. The Customize this Quick Start section explains how to edit the raw-data ETL script to work with your own meter data.

You can customize this Quick Start for use with your own data format. To do that, adjust the first AWS Glue job to map the incoming data to the internal meter-data-lake schema. The following steps describe how.

The first AWS Glue job in the ETL workflow transforms the raw data in the landing-zone S3 bucket to clean data in the clean-zone S3 bucket. This step also maps the inbound data to the internal data schema, which is used by the rest of the steps in the AWS ETL workflow and the ML state machines.

To update the data mapping, you can edit the AWS Glue job directly in the web editor.

This Quick Start deploys an ML pipeline, including a model-training workflow and a batch-processing workflow. This pipeline generates data that becomes the basis for forecasts and anomaly results. The initial deployment runs both workflows after ETL jobs are done.

After new data is added, you can rerun the whole AWS CloudFormation to reprocess the entire data and train a new model. Alternatively, you might rerun each part at a different schedule, as in these examples:

You can also use a custom model for batch pipeline and forecast API.

Use Amazon S3 Glacier to archive meter data from raw, clean, and business S3 buckets for long-term storage and cost savings.

Q. I encountered a CREATE_FAILED error when I launched the Quick Start.

A. If AWS CloudFormation fails to create the stack, relaunch the template with Rollback on failure set to Disabled. This setting is under Advanced in the AWS CloudFormation console on the Configure stack options page. With this setting, the stack’s state is retained and the instance is left running so that you can troubleshoot the issue. (For Windows, look at the log files in %ProgramFiles%\Amazon\EC2ConfigService and C:\cfn\log.)

For additional information, see Troubleshooting AWS CloudFormation on the AWS website.

Q. I encountered a size-limitation error when I deployed the AWS CloudFormation templates.

A. Launch the Quick Start templates from the links in this guide or from another S3 bucket. If you deploy the templates from a local copy on your computer or from a location other than an S3 bucket, you might encounter template-size limitations. For more information, see AWS CloudFormation quotas.

Q. Does the Quick Start replace the existing MDMS from ISVs that utility companies have today?

A. Utility customers have expressed a need to break down data silos and bring greater value out of their data, specifically case meter data. They have asked for AWS best practices for storing meter data in a data lake, and for help getting started running analytics and ML against that data to derive value for their own customers. This Quick Start directly answers these needs. Utility commpanies can hydrate the AWS data lake with data from their existing systems and continue to evolve the Quick Start architecture to meet their business needs. Customers, solution providers, and system integrators can take advantage of this Quick Start to start applying ML against meter data in a matter of days.

Q. Does the Quick Start include a connectivity (topology) model?

A. No. However, S3 is an object store and can hold any type of data, so you can modify it to hold a copy of the topology model from a utilities Geographic Information System (GIS).

Q. What meter dataset is included with the Quick Start?

A. The Quick Start does not include meter data, but instead includes instructions on how to download a public dataset from the city of London, UK, including 5,567 households from November 2011 to February 2014.

Q. What weather data is included with the Quick Start?

A. The Quick Start does not include weather data, but instead includes instructions on how to download a public dataset. This weather data aligns with the geographical area and time period provided for London.

Q. Are any public datasets with large volumes of sample or anonymized smart meter data available?

A. Not at this time, although Amazon, via the Registry of OpenData on AWS (RODA) project, is currently researching options. Utility of Meter manufacturers can donate datasets for public use.

Q. Will the register and interval reads be taken into the data lake?

A. Yes. The data lake can take and process multiple reading types. Register and interval reads are stored in the data lake. However, only the interval data is used for the model training.

Q. Does the Quick Start include publicly available data about homes (for example, sqft) based on address?

A. No. Currently the Quick Start doesn’t include this information.

Q. How is the meter data structured in the data lake, and why is the data model standardized this way?

A. See information about the internal data structure in the Table schemas section in this deployment guide. The internal data schema is the result of working backwards from the requirements and talking to Partners with a long history of industry experience. Future use cases are possible for implementation. For information about separating the data lake into three layers, see AWS releases smart meter data analytics.

Q. Are there any other similar solutions that you are aware of, either from other cloud providers or on-premises solutions?

A. Some AWS Partners are working to adopt the Quick Start as a starting point and then build on top of it. When it comes to on-premises solutions, most utility companies have meter data and an MDM, and do some degree of analytics against the data. Very few do any form of ML against the data. Most want to do more, and do it in the cloud, to free themselves from the costs and constraints of on-premises analytics and ML.

Q. Are the ML models pre-trained?

A. No. The Quick Start contains data pipelines and notebooks for loading and training the models using your own data.

Q. What type of ML model is used for energy forcasting? Do I have to train a separate model for each meter?

A. The Quick Start uses the Amazon SageMaker DeepAR forecasting algorithm, which is a supervised learning algorithm for forecasting scalar (one-dimensional) time series using recurrent neural networks (RNN). This approach automatically produces a global model from historical data of all time series data (for example, data from all your meters) in the dataset. This allows you to have one model consumption forecasting, rather than a model for each meter. Note that this doesn’t mean one model can be used forever; you should retrain the model periodically to learn new patterns.

For more information, see DeepAR Forecasting Algorithm and DeepAR: Probabilistic forecasting with autoregressive recurrent networks.

Q. For energy forecasting, does the ML model take weather into account? If so, how do I include weather data with meter data for training?

A. Weather data is optional. See the (Optional) Prepare weather data for use in training and forecasting section in this guide for instructions on how to use weather information as part of the training pipeline, including the weather data format. The data structure that is expected for weather information is also explained.

Q. For energy forecasting, can you provide an example of how I would query the Forecast API for next week’s forecast energy usage with and without a weather forecast?

A. See the Data schema and API I/O format section in this guide for API descriptions with example requests that demonstrate usage. Also, the provided Jupyter Notebook provides example implementation usage of the API.

Q. What model is used for anomaly detection?

A. The Quick Start uses the Prophet library for anomaly detection. Prophet is a procedure for forecasting time-series data based on an additive model where nonlinear trends are fit with yearly, weekly, and daily seasonality plus holiday effects. It doesn’t support a feedback loop. You can choose to use other algorithms.

Q. Which use cases are available in the Quick Start for anomaly detection?

A. From a anomaly-detection perspective, the Quick Start supports the detection of abnormal usage (higher or lower than expected range) in a certain time period based on historic data.

Q. Has any financial information been included for any of the forecasts, for example, to forecast the most cost-effective generation source to cover peak loads and prevent brownouts or blackouts?

A. No. Currently the only forecast use case that supported is meter usage.

Q. Since meter data is time-series data, why is the Quick Start not using Amazon Timestream as a data store?

A. Timestream was not available at the time this Quick Start was published, although it will be considered for future iterations. Timestream is a good option for certain use cases such as real-time dashboards.

After you successfully deploy this Quick Start, confirm that your resources and services are updated and configured — including any required patches — to meet your security and other needs. For more information, see the AWS Shared Responsibility Model.

To post feedback, submit feature ideas, or report bugs, use the Issues section of the GitHub repository for this Quick Start. To submit code, see the Quick Start Contributor’s Guide.

See the AWS Quick Start home page.

Visit our GitHub repository to download the templates and scripts for this Quick Start, to post your comments, and to share your customizations with others.

This document is provided for informational purposes only. It represents AWS’s current product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS’s products or services, each of which is provided “as is” without warranty of any kind, whether expressed or implied. This document does not create any warranties, representations, contractual commitments, conditions, or assurances from AWS, its affiliates, suppliers, or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.

The software included with this paper is licensed under the Apache License, version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the accompanying "license" file. This code is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either expressed or implied. See the License for specific language governing permissions and limitations.