Deployment Specifications

s3lim is deployed using AWS Serverless Application Model (SAM) templates. All data plane capabilities are consolidated into a single unified template: data-plane-template.yaml (with multi-region orchestration supported via data-plane-stackset.yaml).

Unified Data Plane Spec

Template: data-plane-template.yaml

A single template supporting all deployment methods via parameters:

• Automated Setup Method: Set SourceBucketName to automatically configure S3 Inventory reporting on your source bucket(s) and auto-generate an inventory destination bucket if needed.
• Existing Inventory Method: Set InventoryDestination to point to an existing S3 Inventory destination bucket with managed least-privilege IAM roles.
• BYO-IAM Method: Set LambdaRoleArn to use a pre-created, manually audited IAM role without creating IAM roles in the stack.

Parameters

Resources Created

• Lambda Function (CoreFunction): Core s3lim analysis and workflow coordinator.
• Step Functions State Machine (DistributedStateMachine): Distributed Map workflow for concurrent shard processing and map-reduce aggregation.
• SQS Queue (ProcessingDLQ): Dead Letter Queue for analysis failure handling.
• CloudWatch LogGroup (CoreFunctionLogGroup): Retains execution logs with configurable retention.
• CloudWatch Alarms (ErrorAlarm, DLQDepthAlarm): Built-in operational monitoring for analysis failures and DLQ depth.
• IAM Roles & Policies (managed mode only): Least-privilege execution roles and policies for S3 read, Step Functions orchestration, CloudWatch metrics, SQS, and marketplace metering.
• Inventory Bucket & Custom Resource (automated setup mode only): Dedicated S3 destination bucket and custom resource Lambda to configure S3 inventory on source buckets.
• MCP Gateway Stack (optional): Bedrock AgentCore MCP Gateway integration when enabled.

Outputs

• InventoryDestinationURI: S3 URI where S3 Inventory reports are stored.
• CoreFunctionArn: ARN of the processor Lambda function.
• StateMachineArn: ARN of the Step Functions Distributed Map state machine.

IAM Requirements (BYO-IAM Method)

When using an existing IAM role via LambdaRoleArn, your pre-created IAM role must possess the following minimum permissions:

S3 Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::your-inventory-bucket",
                "arn:aws:s3:::your-inventory-bucket/*"
            ]
        }
    ]
}

CloudWatch, SQS, Step Functions & Marketplace Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudWatchMetrics",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData"
            ],
            "Resource": "*"
        },
        {
            "Sid": "CloudWatchLogs",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:FilterLogEvents",
                "logs:StartQuery",
                "logs:GetQueryResults",
                "logs:DescribeLogStreams",
                "logs:GetLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:log-group:/aws/lambda/s3lim-*"
        },
        {
            "Sid": "SQSDLQAccess",
            "Effect": "Allow",
            "Action": [
                "sqs:SendMessage",
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage",
                "sqs:GetQueueAttributes"
            ],
            "Resource": "arn:aws:sqs:*:*:s3lim-*"
        },
        {
            "Sid": "StepFunctionsExecution",
            "Effect": "Allow",
            "Action": [
                "states:StartExecution",
                "states:DescribeExecution",
                "states:GetExecutionHistory"
            ],
            "Resource": "arn:aws:states:*:*:stateMachine:s3lim-*"
        },
        {
            "Sid": "MarketplaceMetering",
            "Effect": "Allow",
            "Action": [
                "aws-marketplace:BatchMeterUsage",
                "aws-marketplace:GetEntitlements"
            ],
            "Resource": "*"
        }
    ]
}

VPC Execution Policy (Optional - VPC Deployments Only)

When deploying inside a VPC with a BYO-IAM role (LambdaRoleArn), attach the AWS-managed policy AWSLambdaVPCAccessExecutionRole or grant the following EC2 permissions to manage elastic network interfaces (ENIs):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LambdaVPCAccess",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateNetworkInterface",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DeleteNetworkInterface",
                "ec2:AssignPrivateIpAddresses",
                "ec2:UnassignPrivateIpAddresses"
            ],
            "Resource": "*"
        }
    ]
}

(Note: When deploying with managed IAM roles, SAM automatically attaches AWSLambdaVPCAccessExecutionRole to the Lambda execution role whenever SubnetIds is specified).

Wildcard (*) Rationale in IAM Policies

• arn:aws:s3:::<bucket>/*: Required by AWS S3 for object-level actions like s3:GetObject across inventory folders and keys, as well as temporary intermediate state management (s3:PutObject, s3:DeleteObject, s3:DeleteObjectVersion under .s3lim/).
• arn:aws:states:*:*:stateMachine:s3lim-*: Scopes Step Functions state machine execution and status monitoring strictly to s3lim workflows.
• **cloudwatch:PutMetricData (Resource: "*"): Required because the CloudWatch PutMetricData API does not support resource-level ARNs in AWS IAM.
• **aws-marketplace:BatchMeterUsage / GetEntitlements (Resource: "*"): Required because AWS Marketplace Metering APIs do not support resource-level ARNs in AWS IAM.
• **ec2:CreateNetworkInterface / ec2:DescribeNetworkInterfaces / etc. (Resource: "*"): Required because EC2 ENI creation and attachment APIs do not support resource-level ARNs for Lambda VPC integration.
• arn:aws:logs:*:*:log-group:/aws/lambda/s3lim-* & arn:aws:sqs:*:*:s3lim-*: Scopes actions strictly to s3lim Lambda log groups and DLQ queues across deployment regions.

Distributed Execution & Concurrency Tuning (MaxConcurrency)

s3lim executes inventory analyses via an AWS Step Functions Distributed Map workflow, fanning out Worker Lambdas concurrently across inventory data shards.

Parameters

Concurrency Tuning & Quotas

• Quota Protection: Setting MaxConcurrency prevents s3lim from exhausting your regional AWS Lambda concurrent execution quota (default 1,000 unreserved concurrency per account/region).
• Throughput Optimization: For inventories with hundreds or thousands of shards (e.g. multi-terabyte data lakes), increasing MaxConcurrency proportionally speeds up total analysis runtime.
• No Arbitrary Limits: There is no hard-coded cap on MaxConcurrency. You can tune it to match your account’s regional capacity or requested quota increases.
• AWS Quota Reference: For complete information on managing unreserved concurrency, reserved concurrency, and requesting increases, see the official AWS Lambda Concurrency Documentation.

For complete architectural diagrams, worker lifecycles, and performance benchmarks, see the Execution Architecture Guide.

VPC Networking & Egress Configurations

When deploying s3lim inside an Amazon VPC (SubnetIds and SecurityGroupIds), the CoreFunction Lambda (which runs both the Distributed Map Worker and Reducer workloads) attaches elastic network interfaces (ENIs) to the specified private subnets. Depending on your organization’s security posture and network architecture, s3lim supports three deployment tiers:

VPC Deployment Tiers

Tier 3 (Isolated VPC) Endpoint Prerequisites

In private subnets with zero internet egress, Lambda functions cannot reach public AWS service endpoints. The following VPC Endpoints must be provisioned and associated with your VPC route tables and subnets:

1. Critical Execution Endpoints (Mandatory)

• S3 Gateway Endpoint (com.amazonaws.<region>.s3):
  • Endpoint Type: Gateway Endpoint (attached directly to subnet route tables; no hourly or data processing charges).
  • Purpose: Required by Worker and Reducer Lambdas to read inventory manifests, scan inventory Parquet/CSV data shards, and write intermediate state (.s3lim/).
  • Impact: Without this endpoint, S3 API calls hang and fail with network timeouts.
• EventBridge Interface Endpoint (com.amazonaws.<region>.events):
  • Endpoint Type: Interface Endpoint (AWS PrivateLink; provisioned into your target subnets).
  • Purpose: Enables the Reducer Lambda to emit post-scan telemetry and commercial metering events (events:PutEvents) to the Seller EventBridge bus (ControlPlaneEventBusArn).
  • Impact: Telemetry emission operates under a strict fail-closed contract. In isolated subnets without this endpoint, PutEvents calls hang until the Lambda 15-minute (900-second) timeout terminates the task, failing the Step Functions execution.
  • Private DNS: Ensure Enable DNS hostnames and Enable Private DNS are enabled on the interface endpoint so that the AWS SDK resolves events.<region>.amazonaws.com to private ENI IP addresses.

2. Observability & Error Handling Endpoints (Recommended)

• CloudWatch Logs Interface Endpoint (com.amazonaws.<region>.logs):
  • Enables Lambda functions to write execution logs to CloudWatch Log Groups (/aws/lambda/s3lim-*). Without this endpoint, Lambda executes but log events may be dropped or delayed.
• CloudWatch Monitoring Interface Endpoint (com.amazonaws.<region>.monitoring):
  • Enables cloudwatch:PutMetricData for publishing storage analytics metrics and top contributor stats.
• SQS Interface Endpoint (com.amazonaws.<region>.sqs):
  • Enables DLQ routing to ProcessingDLQ for failed message redrive and analysis error tracking.
• Marketplace Metering Interface Endpoint (com.amazonaws.<region>.meteringmarketplace):
  • Required if deploying via AWS Marketplace paid listings to verify entitlements and register hourly metering.

Security Group Configuration

The security groups specified in SecurityGroupIds must allow outbound HTTPS (TCP port 443) traffic to:

• The S3 Gateway Endpoint prefix list (or all traffic within the VPC CIDR).
• The Security Group attached to the Interface VPC Endpoints (events, logs, monitoring, sqs).

Troubleshooting: Reducer Lambda Timeouts on EventBridge PutEvents

Symptoms

1. The Step Functions Distributed Map workflow finishes all Worker map tasks, but the Reducer task hangs for 15 minutes before terminating with a States.Timeout or Lambda.Unknown error.
2. In the CoreFunction CloudWatch logs, the execution prints successful shard aggregation:

   INFO: Successfully aggregated X shard summaries. Emitting telemetry to control plane...
   

   Followed by complete silence until the 900-second Lambda execution timeout triggers.

Root Cause

The stack is deployed with SubnetIds into an isolated private VPC that has no NAT Gateway, and either:

1. The com.amazonaws.<region>.events Interface VPC Endpoint is missing.
2. The com.amazonaws.<region>.events Interface VPC Endpoint has Private DNS disabled.
3. The security group attached to the Interface VPC Endpoint blocks inbound TCP port 443 from the SecurityGroupIds assigned to s3lim.

Because s3lim enforces commercial metering integrity, the Reducer attempts to emit scan telemetry with exponential backoff retries. Without network connectivity to EventBridge, the AWS Go SDK hangs until the Lambda execution timeout expires.

Resolution Steps

1. Verify Outbound Internet Egress:
   • If using a NAT Gateway (Tier 2), verify that the subnet route table has a 0.0.0.0/0 route pointing to an active NAT Gateway in a public subnet.
   • If running in an isolated VPC (Tier 3), confirm an Interface Endpoint for com.amazonaws.<region>.events exists in your VPC.
2. Verify Interface Endpoint Configuration:
   • Navigate to VPC > Endpoints > com.amazonaws..events.
   • Verify that Private DNS names enabled is set to true.
   • Check the Security Groups tab on the endpoint: verify that inbound HTTPS (port 443) is permitted from the s3lim Lambda security group (or the VPC CIDR).
3. Verify S3 Gateway Endpoint:
   • Navigate to VPC > Endpoints > com.amazonaws..s3.
   • Verify that all route tables associated with the subnets in SubnetIds are checked under the Route Tables tab.

Cross-Region Considerations

When deploying s3lim, keep the following in mind:

• Data Transfer Costs: s3lim operates in-place, but if the Lambda function and the S3 Inventory destination bucket reside in different regions, downloading the inventory reports will incur standard AWS Data Transfer Out (egress) costs.
• Execution Timeout: Cross-region S3 downloads suffer from higher latency. For large buckets with billions of keys, cross-region transfers may cause the Lambda execution to exceed its 15-minute (900-second) timeout limit.
• KMS Encryption: If your S3 Inventory destination bucket is encrypted using a Customer Managed Key (CMK) via KMS, the Lambda execution role must be granted cross-region decrypt permissions (kms:Decrypt) on the KMS key.
• CloudWatch Namespace: s3lim publishes metrics to the region in which the Lambda is running.

[!IMPORTANT] Best Practice: Always deploy the s3lim stack in the same region as the S3 Inventory destination bucket to eliminate data transfer fees, minimize latency, and prevent execution timeouts.

Multi-Region StackSet Spec

Template: data-plane-stackset.yaml

Deploys s3lim data planes across multiple AWS regions and accounts using CloudFormation StackSets. S3 Inventory reports are region-locked. Deploying a StackSet ensures each target region runs a local s3lim Lambda instance, avoiding cross-region egress charges and timeouts. StackSets also support multi-account environments and integration with AWS Organizations for centralized governance.

Prerequisites

To deploy StackSets using the SELF_MANAGED permission model, the following IAM roles must exist in the deploying account:

• AWSCloudFormationStackSetAdministrationRole
• AWSCloudFormationStackSetExecutionRole (must trust the administration role)

For detailed instructions on setting up these roles, refer to the official AWS CloudFormation StackSets Prerequisites Guide.

Parameters

Deploying Stack Instances and Parameter Overrides

Deploying the coordinator template creates the StackSet container. You can override parameters per target region:

aws cloudformation create-stack-instances \
  --stack-set-name s3lim-processor-multiregion \
  --accounts 111122223333 \
  --regions us-west-2 \
  --parameter-overrides ParameterKey=InventoryDestination,ParameterValue=s3://my-inventory-bucket-us-west-2/