Troubleshoot A VPC

Step 1: Creating VPC Flow Logs

I started by creating an S3 bucket to publish data from VPC Flow Logs. Then I needed to create VPC Flow Logs on VPC1 to capture information about IP traffic between network interfaces in the VPC and publish the flow logs to the S3 bucket.

First, I created an S3 bucket with a unique name:

I received the following JSON output confirming the bucket creation:

Next, I needed to get the VPC ID for VPC1 to create the Flow Logs:

This command returned the VPC ID I needed:

With both the bucket name and VPC ID, I created the VPC Flow Logs:

The command was successful, showing me the Flow Log ID that was created:

I verified that the flow log was properly created by running:

The output confirmed that my flow log was active and properly configured:

With the flow logs now set up, I was ready to move on to the troubleshooting phase.

Step 2: Troubleshooting Web Server Access

Next, I needed to analyze access to the web server instance and troubleshoot some networking issues. I knew that the cafe web server instance was supposed to be running in the public subnet in VPC1.

I tried accessing the web server at 54.190.138.58 in my browser, but after waiting a few moments, the page failed to load with a "connection timed out" message. This was expected as part of the troubleshooting exercise.

I began investigating by gathering information about the web server instance:

This returned a large JSON document, so I filtered the results to get just the essential information:

The output showed me that the instance was indeed running:

I also tried to establish an SSH connection to the web server using EC2 Instance Connect, but it failed with an error saying "Failed to connect to your instance." This was also expected as part of the troubleshooting challenge.

Troubleshooting Challenge #1: Web Server Connectivity

I needed to figure out why the web server instance was running but the webpage wasn't loading. I decided to take a systematic approach using only the AWS CLI.

First, I used nmap to check for open ports on the web server:

Since nmap couldn't find any open ports, I suspected there might be something at the network level blocking access to the instance. I checked the security group settings:

The security group settings looked correct - they were allowing inbound traffic on ports 22 and 80. So I moved on to checking the route table settings:

Here I found the issue! The route table was named "VPC1-Public-RT" indicating it should be for a public subnet, but it only contained a local route for 10.0.0.0/16. It was missing a route to the internet gateway (0.0.0.0/0). Without this route, traffic couldn't reach the internet.

I created the missing route:

I refreshed the browser page and saw that the web server was now accessible! The page displayed the message "Hello From Your Web Server!" This confirmed that I had resolved the first issue.

Troubleshooting Challenge #2: SSH Connection Issue

Even though I could now access the web server, I still couldn't connect to it using SSH via EC2 Instance Connect. I had already verified that the web server was running, created the route table entry for internet connectivity, and confirmed that the security group was allowing connections on port 22.

I decided to check the network access control list (NACL) settings:

I found another issue! The network ACL had a rule (number 40) explicitly denying inbound SSH traffic (port 22) from any source (0.0.0.0/0). This would block SSH connections regardless of security group settings, as NACLs are evaluated before security groups.

I deleted the problematic NACL entry:

After removing the restrictive NACL rule, I tried connecting via EC2 Instance Connect again and successfully connected to the web server. I ran the hostname command to confirm I was connected to the correct instance:

Success! I had now resolved both connectivity issues: web access and SSH access.

Step 3: Analyzing Flow Logs

Now that I had resolved the network issues, I wanted to analyze the flow logs to see the record of what had happened. This would help me understand how the flow logs captured networking activities.

Downloading and Extracting Flow Logs

First, I needed to download the flow logs from the S3 bucket:

I listed the S3 buckets to recall the bucket name:

I downloaded the flow logs from my S3 bucket:

I navigated to the directory with the log files, pressing tab several times unveiled the whole dir path:

I saw the compressed log files:

I extracted the compressed files:

Analyzing the Log Content

I looked at the structure of one of the log files:

I could see that each log entry contained the source IP, destination IP, ports, and whether the traffic was accepted or rejected. I wanted to focus on the REJECT entries, so I searched for them:

This returned many entries, so I counted them:

I refined my search to focus just on SSH traffic (port 22) that was rejected:

To isolate just my own connection attempts, I determined my public IP address (198.51.100.73) using the EC2 console's security group editor, then filtered the results:

I confirmed that the network interface ID in the logs matched the web server's interface:

Finally, I translated one of the Unix timestamps to a human-readable format:

I compared this to the current time:

This confirmed that the logs were capturing my recent connection attempts that were rejected by the network ACL rule that I identified and removed earlier.