Accessing EC2 with AWS SSM Session Manager

AWS offers a way to access EC2 in private subnet without needing outbound connection to the internet with AWS Systems Manager Session Manager. This is useful for situation where a bastion server is needed for accessing databases / cache.

Some of the benefits in using Session Manager:

1. Access using IAM policies
2. No opening of inbound ports to EC2 needed
3. No management of SSH keys needed
4. Easy access through AWS Console / AWS CLI
5. Logging and auditing sessions through AWS CloudTrail / S3 / CloudWatch Logs

Below is an example of an architecture showing how user access the instance which is in the intra subnet (without internet access).

Architecture

Figure 1. EC2 private access with SSM Session Manager

We will assume us-east-1 as the preferred region in this article.

Requirements

• For EC2 instance in a private subnet without internet access, these 3 VPC endpoints are needed:

  1. ssm
  2. ec2messages
  3. ssmmessages

• SSM agent must be present in the EC2 instance (this is already bundled in Amazon Linux AMIs)

• EC2 IAM role needs to have the IAM policy AmazonSSMManagedInstanceCore

Things to take note

By default, sessions are launched using user with administrative access; ssm-user. There are a couple of ways to prevent launching users with administrative roles:

1. Tagging IAM user or role by specifying an OS user name in Session Manager preference.
   Use the tag key SSMSessionRunAS in the IAM user or role.
2. Configuring the Session Manager preference operating system user name.
   Create the AWS SSM Document with the following name SSM-SessionManagerRunShell and the values accordingly:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

{
  "schemaVersion": "1.0",
  "description": "Document to hold regional settings for Session Manager",
  "sessionType": "Standard_Stream",
  "inputs": {
    "s3BucketName": "",
    "s3KeyPrefix": "",
    "s3EncryptionEnabled": true,
    "cloudWatchLogGroupName": "",
    "cloudWatchEncryptionEnabled": true,
    "cloudWatchStreamingEnabled": true,
    "idleSessionTimeout": "20",
    "maxSessionDuration": "",
    "kmsKeyId": "",
    "runAsEnabled": false,
    "runAsDefaultUser": "",
    "shellProfile": {
      "windows": "",
      "linux": ""
    }
  }
}

3. Disable the ssm-user account sudo permission by adding the following user-data to the EC2 instance.

1
2

#!/bin/bash
echo "#User rules for ssm-user" > /etc/sudoers.d/ssm-agent-users

Hands on

Let’s try to create the following architecture in AWS and accessing the EC2 via Sessions Manager. You can clone the terraform code from sparrow-bork/aws-lambda-post to follow along.

Note that you will incur some cost for running of the resources. The bulk of the cost is mostly from the VPC endpoint which is billed per hour.

*Figure 2. Our Imaginary Infrastructure

The terraform code creates a simple architecture consisting of an EC2 instance, lambda, and 3 VPC endpoint in the intra subnet with the ElastiCache Redis in its own dedicated subnet. Note that this is not meant to be production ready, just an example to show how a user would access the EC2 instance that is on a subnet without internet access.

EC2 AMI

Before spinning up the infra, we will need to create the AMI needed for the EC2 instance with the redis-cli and session manager plugin binary. The base image is based on Amazon Linux 2.

This is necessary as the bastion host will not have internet access and thus, will be unable to install those packages with user-data input when we launch the EC2 instance.

1
2
3
4
5
6

# this assumes that you're in the aws-lambda-post repository directory
# ensure aws cli is properly set up before starting
cd packer
packer init .
packer validate .
packer build .

Output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

==> aml2-redis-cli.amazon-ebs.this: Stopping the source instance...
    aml2-redis-cli.amazon-ebs.this: Stopping instance
==> aml2-redis-cli.amazon-ebs.this: Waiting for the instance to stop...
==> aml2-redis-cli.amazon-ebs.this: Creating AMI hazmei-aml2 from instance i-0faacf9f7384863d6
    aml2-redis-cli.amazon-ebs.this: AMI: ami-00529c6d5f33e25b3
==> aml2-redis-cli.amazon-ebs.this: Waiting for AMI to become ready...
==> aml2-redis-cli.amazon-ebs.this: Skipping Enable AMI deprecation...
==> aml2-redis-cli.amazon-ebs.this: Terminating the source AWS instance...
==> aml2-redis-cli.amazon-ebs.this: Cleaning up any extra volumes...
==> aml2-redis-cli.amazon-ebs.this: No volumes to clean up, skipping
==> aml2-redis-cli.amazon-ebs.this: Deleting temporary security group...
==> aml2-redis-cli.amazon-ebs.this: Deleting temporary keypair...
Build 'aml2-redis-cli.amazon-ebs.this' finished after 5 minutes 44 seconds.

==> Wait completed after 5 minutes 44 seconds

==> Builds finished. The artifacts of successful builds are:
--> aml2-redis-cli.amazon-ebs.this: AMIs were created:
us-east-1: ami-00529c6d5f33e25b3

Take note of the ami id in the output. We will be using it in the terraform values file.

Spinning up the Infra

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# aws-lambda-post/terraform/terraform.tfvars
#
# These are the minimum config required in the terraform directory

name = "tftest-lambda"

vpc_cidr            = "10.0.0.0/16"
azs                 = ["us-east-1a"]
intra_subnets       = ["10.0.1.0/24"]
private_subnets     = ["10.0.11.0/24"]
elasticache_subnets = ["10.0.21.0/24"]

enable_elasticache = true

asg_min_size          = 0
asg_max_size          = 1
asg_desired_cacpacity = 1
# replace this value with the output from the packer build
asg_ec2_ami = "ami-xxxxxxxxxxxxxxxxx"

vpce_services = [
  "ssm",
  "ec2messages",
  "ssmmessages",
]

1
2
3

terraform init
terraform validate
terraform apply

Once the terraform applied successfully, you should be able to see the resources in the us-east-1 region.

Validate default Session Manager Preferences

Figure 3. Custom Session Manager default preferences

The custom values should be applied to the default session manager preferences as shown in figure 3. This allows us to enforce the default preferences to the users.

Access EC2 with Session Manager

Go to the ElastiCache Redis page in the AWS Console and take note of the host endpoint.

To connect to the EC2 instance, go to AWS Systems Manager page in the AWS console followed by the Session Manager. Click on Start a session button and the EC2 instance should show up in the Target instances.

Figure 4. Connecting to EC2 instance via Session Manager

Figure 5. EC2 access via Session Manager and Connected to Redis

Figure 6. Session Manager Session History

We have successfully connected to the EC2 instance and ran a couple of redis commands on the ElastiCache Redis as shown in Figure 5. The history of the past sessions will be shown in the Session history tab and session activity logging will be in CloudWatch / S3 (if this is set) respectively.

Before you go

Don’t forget to clean the resources so you don’t incur additional billing. 😉 The AMI resource created by Packer needs to be manually deleted after terraform destroy command successfully completed.