Apache RocketMQ on the AWS Cloud
Quick Start Reference Deployment

April 2021
George Gu, Fan Su, Chen Chuan, Apache
Troy Ameigh, Mingwei Leong, AWS Quick Start team
Visit our GitHub repository for source files and to post feedback, report bugs, or submit feature ideas for this Quick Start. |
This Quick Start was created by AWS Great China Region Solution Team in collaboration with Amazon Web Services (AWS). Quick Starts are automated reference deployments that use AWS CloudFormation templates to deploy key technologies on AWS, following AWS best practices.
Overview
This guide provides instructions for deploying the RocketMQ Quick Start reference architecture on the AWS Cloud.
Apache RocketMQ is a unified messaging engine and lightweight data processing platform. The Apache RocketMQ Quick Start helps you deploy a configurable and highly available Apache RocketMQ cluster in the AWS Cloud.
Amazon may share user-deployment information with the AWS Partner that collaborated with AWS on the Quick Start. |
Apache RocketMQ on AWS
This Quick Start sets up an Amazon Web Services (AWS) architecture that deploys an Apache RocketMQ cluster on AWS. Apache RocketMQ is a unified messaging engine and lightweight data processing platform. This Quick Start creates and configures Amazon Elastic Compute Cloud (Amazon EC2) instances to host Apache name server and broker nodes. You can configure basic cluster settings such as EC2 instance types during deployment.
AWS costs
You are responsible for the cost of the AWS services and any third-party licenses used while running this Quick Start. There is no additional cost for using the Quick Start.
The AWS CloudFormation templates for Quick Starts include configuration parameters that you can customize. Some of the settings, such as the instance type, affect the cost of deployment. For cost estimates, see the pricing pages for each AWS service you use. Prices are subject to change.
After you deploy the Quick Start, create AWS Cost and Usage Reports to deliver billing metrics to an Amazon Simple Storage Service (Amazon S3) bucket in your account. These reports provide cost estimates based on usage throughout each month and aggregate the data at the end of the month. For more information, see What are AWS Cost and Usage Reports? |
Software licenses
No licenses are required to deploy this Quick Start. You are responsible for the cost of the AWS services and any third-party licenses used while running this Quick Start reference deployment. There is no additional cost for using this Quick Start.
Architecture
Deploying this Quick Start for a new virtual private cloud (VPC) with default parameters builds the following RocketMQ environment in the AWS Cloud.

As shown in Figure 1, the Quick Start sets up the following:
-
A highly available architecture that spans up to three Availability Zones.*
-
A VPC configured with public and private subnets, according to AWS best practices, to provide you with your own virtual network on AWS.*
-
In the public subnets:
-
Managed network address translation (NAT) gateways to allow outbound internet access for resources in the private subnets.*
-
Linux bastion hosts in an Auto Scaling group to allow inbound Secure Shell (SSH) access to EC2 instances in public and private subnets.*
-
-
In the private subnets:
-
Amazon Elastic Compute Cloud (Amazon EC2) instances for Apache RocketMQ NameServers. You can deploy 1–3 NameServer nodes. The default is three (shown).
-
Apache RocketMQ web console running on each Apache NameServer instance.
-
EC2 instances for Apache RocketMQ Broker servers. You can deploy either one or three Broker server nodes. The default is three (shown).
-
*The template that deploys the Quick Start into an existing VPC skips the components marked by asterisks and prompts you for your existing VPC configuration.
Planning the deployment
Specialized knowledge
This deployment requires a moderate level of familiarity with AWS services. If you’re new to AWS, see Getting Started Resource Center and AWS Training and Certification. These sites provide materials for learning how to design, deploy, and operate your infrastructure and applications on the AWS Cloud.
This Quick Start assumes familiarity with Amazon EC2, Amazon VPC, and Apache RocketMQ.
AWS account
If you don’t already have an AWS account, create one at https://aws.amazon.com by following the on-screen instructions. Part of the sign-up process involves receiving a phone call and entering a PIN using the phone keypad.
Your AWS account is automatically signed up for all AWS services. You are charged only for the services you use.
Technical requirements
Before you launch the Quick Start, review the following information and ensure that your account is properly configured. Otherwise, deployment might fail.
Resource quotas
If necessary, request service quota increases for the following resources. You might need to request increases if your existing deployment currently uses these resources and if this Quick Start deployment could result in exceeding the default quotas. The Service Quotas console displays your usage and quotas for some aspects of some services. For more information, see What is Service Quotas? and AWS service quotas.
Resource | This deployment uses |
---|---|
VPCs |
1 |
Elastic IP addresses |
3 |
Security groups |
4 |
AWS Identity and Access Management (IAM) roles |
2 |
Auto Scaling groups |
1 |
m5.large instances (Apache name server hosts) |
2 |
m5.xlarge instances (Apache broker hosts) |
3 |
t2.micro instances (bastion hosts) |
1 |
Supported Regions
This Quick Start is available in all AWS Regions. For more information, see AWS service endpoints.
Certain Regions are available on an opt-in basis. For more information, see Managing AWS Regions. |
EC2 key pairs
Make sure that at least one Amazon EC2 key pair exists in your AWS account in the Region where you plan to deploy the Quick Start. Make note of the key pair name. You need it during deployment. To create a key pair, see Amazon EC2 key pairs and Linux instances.
For testing or proof-of-concept purposes, we recommend creating a new key pair instead of using one that’s already being used by a production instance.
IAM permissions
Before launching the Quick Start, you must sign in to the AWS Management Console with IAM permissions for the resources that the templates deploy. The AdministratorAccess managed policy within IAM provides sufficient permissions, although your organization may choose to use a custom policy with more restrictions. For more information, see AWS managed policies for job functions.
Deployment options
This Quick Start provides two deployment options:
-
Deploy RocketMQ into a new VPC. This option builds a new AWS environment consisting of the VPC, subnets, NAT gateways, security groups, bastion hosts, and other infrastructure components. It then deploys RocketMQ into this new VPC.
-
Deploy RocketMQ into an existing VPC. This option provisions RocketMQ in your existing AWS infrastructure.
The Quick Start provides separate templates for these options. It also lets you configure Classless Inter-Domain Routing (CIDR) blocks, instance types, and RocketMQ settings, as discussed later in this guide.
Deployment steps
Sign in to your AWS account
-
Sign in to your AWS account at https://aws.amazon.com with an IAM user role that has the necessary permissions. For details, see Planning the deployment earlier in this guide.
-
Make sure that your AWS account is configured correctly, as discussed in the Technical requirements section.
Launch the Quick Start
If you’re deploying RocketMQ into an existing VPC, make sure that your VPC has two private subnets in different Availability Zones for the workload instances and that the subnets aren’t shared. This Quick Start doesn’t support shared subnets. These subnets require NAT gateways in their route tables to allow the instances to download packages and software without exposing them to the internet. Also make sure that the domain name option in the DHCP options is configured as explained in DHCP options sets. You provide your VPC settings when you launch the Quick Start. |
Each deployment takes about 15-20 minutes to complete.
-
Sign in to your AWS account, and choose one of the following options to launch the AWS CloudFormation template. For help with choosing an option, see Deployment options earlier in this guide.
-
Check the AWS Region that’s displayed in the upper-right corner of the navigation bar, and change it if necessary. This Region is where the network infrastructure for RocketMQ is built. The template is launched in the us-east-1 Region by default. For other choices, see Supported Regions earlier in this guide.
-
On the Create stack page, keep the default setting for the template URL, and then choose Next.
-
On the Specify stack details page, change the stack name if needed. Review the parameters for the template. Provide values for the parameters that require input. For all other parameters, review the default settings and customize them as necessary. For details on each parameter, see the Parameter reference section of this guide. When you finish reviewing and customizing the parameters, choose Next.
-
On the Configure stack options page, you can specify tags (key-value pairs) for resources in your stack and set advanced options. When you’re finished, choose Next.
-
On the Review page, review and confirm the template settings. Under Capabilities, select the two check boxes to acknowledge that the template creates IAM resources and might require the ability to automatically expand macros.
-
Choose Create stack to deploy the stack.
-
Monitor the status of the stack. When the status is CREATE_COMPLETE, the RocketMQ deployment is ready.
-
To view the created resources, see the values displayed in the Outputs tab for the stack.
Logging into an Apache RocketMQ NameServer or Broker server node
-
Add a PEM key pair. For example:
ssh-add -k /Users/username/tmp/rocketmqXXX.pem
-
Sign into the bastion host.
ssh -A ec2-user@ecXXXXXXXX.cn-north-1.compute.amazonaws.com.cn
-
Get the IP address from the Amazon EC2 console.
-
Open the Amazon EC2 console.
-
On the Instances screen, select the checkbox for the instance. Note the Private IPs in the Description tab, as shown in Figure 2.
-
-
From the bastion host, log into the node with the IP address. For example:
(ec2-user@ip-10-0-136-135) $ ssh 10.0.6.235 Last login: Wed Jan 6 04:55:01 2021 from ip-10-0-136-135.cn-north-1.compute.internal __| __|_ ) _| ( / Amazon Linux 2 AMI ___|\___|___| https://aws.amazon.com/amazon-linux-2/ [ec2-user@ip-10-0-6-235 ~]$
-
Once you are signed into the node, you can use Apache RocketMQ command-line tools to query, manage, and diagnose issues.
Viewing the Apache RocketMQ web console in a browser
The default deployment installs the Apache RocketMQ web console on the name servers. However, as Apache name servers run in the private subnets, they cannot be accessed directly through a browser. To access port 8080 of the private IP address of the Apache name server, configure an SSH connection with the bastion host SSH tunnel. Then use the SSH tunnel to access the web console.
-
Establish a connection with the bastion host using SSH with the following command. Replace
port number
,key pair.pem
,user name
, andhost name
with your parameters.ssh -qTfnN -D port number -i "key pair.pem" user name@host name
For example:
ssh -qTfnN -D 40011 -i "rocketMQ.pem" ec2-user@ec2-54-223-36-247.cn-north-1.compute.amazonaws.com.cn
-
Set up a proxy manager in your browser. There are many proxy manager plug-ins available. These instructions use SwitchyOmega.
-
Install the SwitchyOmega plug-in.
-
Open the SwitchyOmega options page and select New Profile in the left sidebar.
-
Enter a name and select Create.
-
Enter the Protocol, Server, and Port for proxy server. The port should be the local port where you set up the SSH tunnel in step 1.
-
Select Apply Changes.
-
Access SwitchyOmega through the extension icon of your browser. Select your created profile in the proxy list. The browser will send all traffic through the local port 40011 proxy to the bastion host.
-
-
To view the Apache web console on a name server in your browser, navigate to
http://10.0.xx.xx:8080
. Replacexx.xx
with the private IP address of the name server. You can find the private IP address of a name server in the EC2 console.Figure 8 shows an example of an Apache RocketMQ cluster in the Apache web console.
Apache RocketMQ resources
NameServer nodes
-
Apache RocketMQ installation guide:
/home/ec2-user/rocketmq-deploy
-
Apache RocketMQ installation directory:
/home/ec2-user/rocketmq-deploy/rocketmq-all-4.7.1-bin-release
or/home/ec2-user/rocketmq-deploy/rocketmq-all-4.8.0-bin-release
-
Deployment script installation log used to troubleshoot error messages:
/home/ec2-user/rocketmq-deploy/install.log
-
Apache RocketMQ web console:
/home/ec2-user/rocketmq-deploy/rocketmq-console-ng-1.0.0.jar
Broker server nodes
-
Apache RocketMQ installation guide:
/home/ec2-user/rocketmq-deploy
-
Apache RocketMQ installation directory:
/home/ec2-user/rocketmq-deploy/rocketmq-all-4.7.1-bin-release
or/home/ec2-user/rocketmq-deploy/rocketmq-all-4.8.0-bin-release
-
Deployment script installation logs:
/home/ec2-user/rocketmq-deploy/install.log
-
Configuration file:
/home/ec2-user/rocketmq-deploy/rocketMQ-config/
-
Apache RocketMQ local file storage:
/home/ec2-user/rocketmq-deploy/rmqstore
Best practices for using RocketMQ on AWS
FAQ
Q. I encountered a CREATE_FAILED error when I launched the Quick Start.
A. If AWS CloudFormation fails to create the stack, relaunch the template with Rollback on failure set to Disabled. This setting is under Advanced in the AWS CloudFormation console on the Configure stack options page. With this setting, the stack’s state is retained, and the instance keeps running so that you can troubleshoot the issue. (For Windows, look at the log files in %ProgramFiles%\Amazon\EC2ConfigService
and C:\cfn\log
.)
When you set Rollback on failure to Disabled, you continue to incur AWS charges for this stack. Delete the stack when you finish troubleshooting. |
For more information, see Troubleshooting AWS CloudFormation.
Q. I encountered a size-limitation error when I deployed the AWS CloudFormation templates.
A. Launch the Quick Start templates from the links in this guide or from another S3 bucket. If you deploy the templates from a local copy on your computer or from a location other than an S3 bucket, you might encounter template-size limitations. For more information, see AWS CloudFormation quotas.
Troubleshooting
-
I cannot access the Apache RocketMQ web console.
-
The Apache RocketMQ web Console runs on an Apache name server in a private subnet. Verify that the SSH tunnel from your desktop to the bastion host is set up to connect to the Apache name server.
-
If the SSH tunnel is setup correctly, but you still cannot open the web console, it is possible that the web console did not start properly. Log in to the name server and check the install log file at
/home/ec2-user/rocketmq-deploy/install.log
.
-
-
The Apache RocketMQ broker server cluster does not show up in the Apache RocketMQ web console.
-
Log in to the broker node and check the install log in
/home/ec2-user/rocketmq-deploy/install.log
. You can submit an issue to the Quick Start GitHub repository for assistance. Include the error log with your submission.
-
-
How can I access the Apache RocketMQ internal logs?
-
On the name server and brokers, internal Apache RocketMQ logs are found in
/home/ec2-user/rocketmq-deploy/
.
-
Customer responsibility
After you successfully deploy this Quick Start, confirm that your resources and services are updated and configured — including any required patches — to meet your security and other needs. For more information, see the AWS Shared Responsibility Model.
Parameter reference
Unless you are customizing the Quick Start templates for your own deployment projects, keep the default settings for the parameters labeled Quick Start S3 bucket name, Quick Start S3 bucket Region, and Quick Start S3 key prefix. Changing these parameter settings automatically updates code references to point to a new Quick Start location. For more information, see the AWS Quick Start Contributor’s Guide. |
Launch into an new VPC
Parameter label (name) | Default value | Description |
---|---|---|
Availability Zones
( |
|
Choose two or three Availability Zones to use for the VPC subnets. |
Number of Availability Zones
( |
|
Choose the number of Availability Zones to use in the VPC. This must match the number of AZs selected in the Availability Zones parameter. |
VPC CIDR
( |
|
CIDR Block for the VPC |
Private subnet 1 CIDR
( |
|
CIDR block for private subnet 1, located in Availability Zone 1. |
Private subnet 2 CIDR
( |
|
CIDR block for private subnet 2, located in Availability Zone 2. |
Private subnet 3 CIDR
( |
|
CIDR block for private subnet 3, located in Availability Zone 2. |
Public subnet 1 CIDR
( |
|
CIDR Block for public subnet 1, located in Availability Zone 1. |
Public subnet 2 CIDR
( |
|
CIDR block for public subnet 2, located in Availability Zone 2. |
Public subnet 3 CIDR
( |
|
CIDR Block for public subnet 3, located in Availability Zone 3. |
Allowed bastion external access CIDR
( |
|
Allowed CIDR block for external SSH access to the bastion hosts. |
Parameter label (name) | Default value | Description |
---|---|---|
Key pair name
( |
|
Public/private key pairs to connect to your instance after deployment. |
Parameter label (name) | Default value | Description |
---|---|---|
Bastion host AMI operating system
( |
|
The Linux distribution for the AMI to be used for the bastion host instances. |
Bastion host instance type
( |
|
Amazon EC2 instance type for the second bastion host instance. |
Number of bastion hosts
( |
|
Enter the number of bastion hosts to create. |
Parameter label (name) | Default value | Description |
---|---|---|
Number of Apache RocketMQ NameServer cluster nodes
( |
|
Number of Apache RocketMQ NameServer nodes (1–3). |
Number of Apache RocketMQ Broker cluster nodes
( |
|
Number of replica set members (1 or 3). |
IOPS
( |
|
IOPS of io1 EBS volume. Not applicable for EBS volume types gp2 and gp3. |
RocketMQ version
( |
|
Apache RocketMQ version. |
NameServer node instance type
( |
|
Amazon EC2 instance type for Apache RocketMQ NameServer nodes. |
Broker node instance type
( |
|
Amazon EC2 instance type for Apache RocketMQ Broker nodes. |
Apache RocketMQ flush disk type
( |
|
Apache RocketMQ flush disk type (ASYNC_FLUSH or SYNC_FLUSH). |
Volume size
( |
|
Size of attached EBS volume, in GBs. |
Volume type
( |
|
Type of attached EBS volume (io1, gp2, or gp3). |
Parameter label (name) | Default value | Description |
---|---|---|
Quick Start S3 bucket name
( |
|
Name of the S3 bucket for your copy of the Quick Start assets. Keep the default name unless you are customizing the template. Changing the name updates code references to point to a new Quick Start location. This name can include numbers, lowercase letters, uppercase letters, and hyphens, but do not start or end with a hyphen (-). See https://aws-quickstart.github.io/option1.html. |
Quick Start S3 key prefix
( |
|
S3 key prefix that is used to simulate a directory for your copy of the Quick Start assets. Keep the default prefix unless you are customizing the template. Changing this prefix updates code references to point to a new Quick Start location. This prefix can include numbers, lowercase letters, uppercase letters, hyphens (-), and forward slashes (/). End with a forward slash. See https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html and https://aws-quickstart.github.io/option1.html. |
Quick Start S3 bucket Region
( |
|
AWS Region where the Quick Start S3 bucket (QSS3BucketName) is hosted. Keep the default Region unless you are customizing the template. Changing this Region updates code references to point to a new Quick Start location. When using your own bucket, specify the Region. See https://aws-quickstart.github.io/option1.html. |
Launch into an existing VPC
Parameter label (name) | Default value | Description |
---|---|---|
Number of Availability Zones
( |
|
Choose the number of Availability Zones to use in the VPC. This must match the number of AZs selected in the Availability Zones parameter. |
VPC ID
( |
|
ID of your existing VPC (e.g., vpc-0343606e). |
Private subnet 1 ID
( |
|
ID of the private subnet in Availability Zone 1 of your existing VPC (e.g., subnet-fe9a8b32). |
Private subnet 2 ID
( |
|
ID of the private subnet in Availability Zone 2 of your existing VPC (e.g., subnet-be8b01ea). |
Private subnet 3 ID
( |
|
ID of the private subnet in Availability Zone 3 of your existing VPC (e.g., subnet-abd39039). |
Public subnet 1 ID
( |
|
ID of the public subnet in Availability Zone 1 of your existing VPC (e.g., subnet-a0246dcd). |
Public subnet 2 ID
( |
|
ID of the public subnet in Availability Zone 2 of your existing VPC (e.g., subnet-b1236eea). |
Public subnet 3 ID
( |
|
ID of the public subnet in Availability Zone 3 of your existing VPC (e.g., subnet-c3456aba). |
Allowed bastion external access CIDR
( |
|
Allowed CIDR block for external SSH access to the bastions |
Parameter label (name) | Default value | Description |
---|---|---|
Key pair name
( |
|
Public/private key pairs to connect to your instance after deployment. |
Parameter label (name) | Default value | Description |
---|---|---|
Bastion host AMI operating system
( |
|
The Linux distribution for the AMI to be used for the bastion instances. |
Bastion host instance type
( |
|
Amazon EC2 instance type for the second bastion host instance. |
Number of bastion hosts
( |
|
Enter the number of bastion hosts to create. |
Parameter label (name) | Default value | Description |
---|---|---|
Number of Apache RocketMQ NameServer cluster nodes
( |
|
Number of Apache RocketMQ NameServer nodes (1–3). |
Number of Apache RocketMQ Broker cluster nodes
( |
|
Number of replica set members (1 or 3). |
IOPS
( |
|
IOPS of io1 EBS volume. Not applicable for EBS volume types gp2 and gp3. |
Apache RocketMQ Version
( |
|
Apache RocketMQ version. |
NameServer instance type
( |
|
Amazon EC2 instance type for Apache RocketMQ NameServer nodes. |
Broker node instance type
( |
|
Amazon EC2 instance type for Apache RocketMQ Broker nodes. |
Apache RocketMQ flush disk type
( |
|
Apache RocketMQ flush disk type (ASYNC_FLUSH or SYNC_FLUSH). |
Volume size
( |
|
Size of attached EBS volume, in GBs. |
Volume type
( |
|
Type of attached EBS volume (io1, gp2, or gp3). |
Parameter label (name) | Default value | Description |
---|---|---|
Quick Start S3 bucket name
( |
|
Name of the S3 bucket for your copy of the Quick Start assets. Keep the default name unless you are customizing the template. Changing the name updates code references to point to a new Quick Start location. This name can include numbers, lowercase letters, uppercase letters, and hyphens, but do not start or end with a hyphen (-). See https://aws-quickstart.github.io/option1.html. |
Quick Start S3 key prefix
( |
|
S3 key prefix that is used to simulate a directory for your copy of the Quick Start assets. Keep the default prefix unless you are customizing the template. Changing this prefix updates code references to point to a new Quick Start location. This prefix can include numbers, lowercase letters, uppercase letters, hyphens (-), and forward slashes (/). End with a forward slash. See https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html and https://aws-quickstart.github.io/option1.html. |
Quick Start S3 bucket Region
( |
|
AWS Region where the Quick Start S3 bucket (QSS3BucketName) is hosted. Keep the default Region unless you are customizing the template. Changing this Region updates code references to point to a new Quick Start location. When using your own bucket, specify the Region. See https://aws-quickstart.github.io/option1.html. |
Send us feedback
To post feedback, submit feature ideas, or report bugs, use the Issues section of the GitHub repository for this Quick Start. To submit code, see the Quick Start Contributor’s Guide.
Quick Start reference deployments
See the AWS Quick Start home page.
GitHub repository
Visit our GitHub repository to download the templates and scripts for this Quick Start, to post your comments, and to share your customizations with others.
Notices
This document is provided for informational purposes only. It represents AWS’s current product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS’s products or services, each of which is provided “as is” without warranty of any kind, whether expressed or implied. This document does not create any warranties, representations, contractual commitments, conditions, or assurances from AWS, its affiliates, suppliers, or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.
The software included with this paper is licensed under the Apache License, version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the accompanying "license" file. This code is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either expressed or implied. See the License for specific language governing permissions and limitations.