CloudTrail and VPC Endpoints Logging¶
Today I learnt that AWS CloudTrail does not log requests that are denied by VPC endpoint policy. The reason for this is that it would allow an attacker to exfiltrate data via CloudTrail.
Consider the following scenario:
Account A is a corporate account that belongs to The ColaCo company with a treasure trove of data, and lots of controls to prevent data exfiltration, policy on the S3 VPC endpoint to ensure only buckets local to the org can be accessed. And of course CloudTrail data event logging.
Account B is some random attackers personal account, with CloudTrail logging of data events enabled.
A person with nefarious intent in Account A wants to exfiltrate the secret recipe for ColaCo, they find the VPC endpoint restricts access to their public S3 Bucket in Account B and they are unable to copy the recipe.
The attacker gets a sneaky idea, what if they write a some python to base64 encode the recipe and then cut it into tiny chunks and then send S3 requests to their bucket with the tiny chunks of the recipe in the request data. Although they will get an access denied, the request data will appear in the attackers Cloudtrail! They can then write another script to scrape the denied requests from the Account B CloudTrail and re-assemble the recipe.
Luckily for the ColaCo, AWS has thought of this, and all access denied due to to S3 VPC endpoint policy are not logged anywhere . No event will appear in the CloudTrail data event logs of either Account A or Account B.
Cross account S3 CloudTrail data event logging¶
This document describes what happens regarding S3 data event logging in the above scenario, where the API calls are blocked by VPC Endpoint Policy.
I set up a test to validate the confusing documentation, and in short validated CloudTrail event data for authorized access is sent to both Account A and Account B Cloudtrail.
Here is a sample of the CloudTrail data event (with all null fields stripped) as extracted from both accounts.
Account A Event¶
ColaCo (Account A) from where a PutObject
was made against a Bucket in Account
B.
You are able to see the owner of the bucket in the resources[0].accountId
field.
eventVersion: 1.09
eventTime: 2024-08-12T05:26:38Z
eventSource: s3.amazonaws.com
eventName: PutObject
awsRegion: ap-southeast-2
sourceIPAddress: 10.8.0.4
userAgent: [aws-cli/2.15.30 Python/3.9.16 Linux/6.1.102-108.177.amzn2023.x86_64 source/x86_64.amzn.2023 prompt/off command/s3.cp]
requestID: EA7B4DQGKYZJYDV3
eventID: 1d3f51b4-8458-4789-befb-5229247a31e8
readOnly: False
resources: [{'type': 'AWS::S3::Object', 'ARN': 'arn:aws:s3:::public-bucketa802822/testreal'}, {'accountId': '003422198502', 'type': 'AWS::S3::Bucket', 'ARN': 'arn:aws:s3:::public-bucketa802822'}]
eventType: AwsApiCall
managementEvent: False
recipientAccountId: 750263812530
sharedEventID: 99bd211a-cc79-4024-84a6-dceb8910ec46
vpcEndpointId: vpce-0d3d7c2c2b14e6316
eventCategory: Data
userIdentity.type: IAMUser
userIdentity.principalId: AIDA25LZVIGZO2HVIV5L3
userIdentity.accountId: 750263812530
requestParameters.bucketName: public-bucketa802822
requestParameters.Host: public-bucketa802822.s3.ap-southeast-2.amazonaws.com
additionalEventData.SignatureVersion: SigV4
additionalEventData.CipherSuite: ECDHE-RSA-AES128-GCM-SHA256
additionalEventData.bytesTransferredIn: 0.0
additionalEventData.AuthenticationMethod: AuthHeader
additionalEventData.x-amz-id-2: hEw66KOOCR/o41mTL4NMoADLPRZoirzBnoysj8XEfGVzw7Gj4tyaWUaSiBSCpN23MUEjqpswmsHGKKxnfMNV715SOgau3aVn
additionalEventData.bytesTransferredOut: 0.0
tlsDetails.tlsVersion: TLSv1.2
tlsDetails.cipherSuite: ECDHE-RSA-AES128-GCM-SHA256
tlsDetails.clientProvidedHostHeader: public-bucketa802822.s3.ap-southeast-2.amazonaws.com
filename: /home/brent/s3test/750263812530_CloudTrail_ap-southeast-2_20240812T0530Z_jqPiH6xPtjO8MDex.json.gz
userIdentity.arn: arn:aws:iam::750263812530:user/admin
userIdentity.accessKeyId: AKIA25LZVIGZM4JSIDL2
userIdentity.userName: admin
requestParameters.key: testreal
responseElements.x-amz-server-side-encryption: AES256
additionalEventData.SSEApplied: Default_SSE_S3
Account B Event¶
Account B also gets the CloudTrail S3 Data event:
eventVersion: 1.09
eventTime: 2024-08-12T05:26:38Z
eventSource: s3.amazonaws.com
eventName: PutObject
awsRegion: ap-southeast-2
sourceIPAddress: 10.8.0.4
userAgent: [aws-cli/2.15.30 Python/3.9.16 Linux/6.1.102-108.177.amzn2023.x86_64 source/x86_64.amzn.2023 prompt/off command/s3.cp]
requestID: EA7B4DQGKYZJYDV3
eventID: 30b6a538-8fcb-45f1-ab72-a3037ca5b335
readOnly: False
resources: [{'type': 'AWS::S3::Object', 'ARN': 'arn:aws:s3:::public-bucketa802822/testreal'}, {'accountId': '003422198502', 'type': 'AWS::S3::Bucket', 'ARN': 'arn:aws:s3:::public-bucketa802822'}]
eventType: AwsApiCall
managementEvent: False
recipientAccountId: 003422198502
sharedEventID: 99bd211a-cc79-4024-84a6-dceb8910ec46
vpcEndpointId: vpce-0d3d7c2c2b14e6316
eventCategory: Data
userIdentity.type: AWSAccount
userIdentity.principalId: AIDA25LZVIGZO2HVIV5L3
userIdentity.accountId: 750263812530
requestParameters.bucketName: public-bucketa802822
requestParameters.Host: public-bucketa802822.s3.ap-southeast-2.amazonaws.com
additionalEventData.SignatureVersion: SigV4
additionalEventData.CipherSuite: ECDHE-RSA-AES128-GCM-SHA256
additionalEventData.bytesTransferredIn: 0.0
additionalEventData.AuthenticationMethod: AuthHeader
additionalEventData.x-amz-id-2: hEw66KOOCR/o41mTL4NMoADLPRZoirzBnoysj8XEfGVzw7Gj4tyaWUaSiBSCpN23MUEjqpswmsHGKKxnfMNV715SOgau3aVn
additionalEventData.bytesTransferredOut: 0.0
tlsDetails.tlsVersion: TLSv1.2
tlsDetails.cipherSuite: ECDHE-RSA-AES128-GCM-SHA256
tlsDetails.clientProvidedHostHeader: public-bucketa802822.s3.ap-southeast-2.amazonaws.com
filename: /home/brent/s3test/003422198502_CloudTrail_ap-southeast-2_20240812T0535Z_z0bV9nRfhQ736Zxb.json.gz
requestParameters.key: testreal
responseElements.x-amz-server-side-encryption: AES256
additionalEventData.SSEApplied: Default_SSE_S3
The events are almost identical except for:
eventID
recipientAccountId
userIdentity.type:
Account A: IAMUser
Account B: AWSAccount
Other Events that are not logged in CloudTrail¶
iam:PassRole
Has no event in CloudTrailData events do not appear in the CloudTrail Console and can only be observed from the Bucket or Loggroup
Cloudawtch:PutMetricData
s3:GeneratePresignedUrl
(Only logged when the URL is used to get/put)
MSK Events not logged in cloudtrail¶
kafka-cluster:Connect
kafka-cluster:ReadData
kafka-cluster:WriteData
For these you need to enable broker logs and send them to CloudWatch
Additional Reading¶
Some extracts from the KMS docs:
Logging your VPC endpoint AWS CloudTrail logs all operations that use the VPC endpoint. When a request to AWS KMS uses a VPC endpoint, the VPC endpoint ID appears in the AWS CloudTrail log entry that records the request. You can use the endpoint ID to audit the use of your AWS KMS VPC endpoint.
However, your CloudTrail logs don’t include operations requested by principals in other accounts or requests for AWS KMS operations on KMS keys and aliases in other accounts. Also, to protect your VPC, requests that are denied by a VPC endpoint policy, but otherwise would have been allowed, are not recorded in AWS CloudTrail.
Comments
comments powered by Disqus