Standard Deployment Walkthrough

This guide is for users who already have S3 Inventory reports configured and delivered to a destination bucket, and want s3lim to analyze them.

Use Case

Ideal for analyzing pre-existing inventory pipelines. s3lim supports two IAM permission options:

• Managed IAM (Default): Automatically provisions least-privilege IAM execution roles so you don’t have to configure policies manually.
• Bring-Your-Own-IAM (BYO-IAM): For strict enterprise compliance environments where IAM policies must be pre-audited, you can supply your own execution role via LambdaRoleArn.

Prerequisites

• An AWS account with permissions to deploy Lambda and CloudFormation (and IAM roles if using Managed IAM).
• A pre-existing S3 Inventory report pipeline delivering reports to a destination S3 bucket.
• The S3 URI where your existing inventory manifests are delivered (e.g., s3://my-inventory-bucket/inventory/).
• (BYO-IAM only): A pre-created IAM role ARN (LambdaRoleArn) configured with the permissions outlined in BYO-IAM Permissions below.

Deployment Steps

1. Locate Inventory Destination: Note the S3 bucket and prefix path where inventory manifests are delivered (e.g. s3://my-inventory-bucket/inventory/).
2. Launch Stack: Navigate to the s3lim application on the AWS Serverless Application Repository (or deploy aws/customer/data-plane-template.yaml).
3. Configure Stack Parameters:
   • InventoryDestination: Enter your inventory reports S3 URI.
   • LambdaRoleArn (optional): For BYO-IAM deployment, provide the ARN of your pre-audited IAM role. Leave blank for Managed IAM.
   • SourceBucketName (optional): Required only if you want to run previews using S3 ListObjectsV2 API.
   • For other options, see the Deployment Specifications.
4. Deploy: Click Deploy to start provisioning.

BYO-IAM Role Permissions (LambdaRoleArn)

If your enterprise security policy requires using a pre-audited IAM role, create an IAM role using the policies below and supply its ARN to LambdaRoleArn.

1. Trust Relationship (Assume Role Policy)

The IAM execution role must trust the AWS Lambda service.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

2. IAM Permissions Policy

Attach this policy to the role. Resources are scoped strictly to the inventory bucket, s3lim-* log groups, and s3lim-* DLQs.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3InventoryAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:DeleteObjectVersion"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR_INVENTORY_BUCKET",
        "arn:aws:s3:::YOUR_INVENTORY_BUCKET/*"
      ]
    },
    {
      "Sid": "CloudWatchMetrics",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:PutMetricData"
      ],
      "Resource": "*"
    },
    {
      "Sid": "CloudWatchLogs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:FilterLogEvents",
        "logs:StartQuery",
        "logs:GetQueryResults",
        "logs:DescribeLogStreams",
        "logs:GetLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/aws/lambda/s3lim-*"
    },
    {
      "Sid": "SQSDLQAccess",
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "arn:aws:sqs:*:*:s3lim-*"
    },
    {
      "Sid": "StepFunctionsExecution",
      "Effect": "Allow",
      "Action": [
        "states:StartExecution",
        "states:DescribeExecution",
        "states:GetExecutionHistory"
      ],
      "Resource": "arn:aws:states:*:*:stateMachine:s3lim-*"
    },
    {
      "Sid": "MarketplaceMetering",
      "Effect": "Allow",
      "Action": [
        "aws-marketplace:BatchMeterUsage",
        "aws-marketplace:GetEntitlements"
      ],
      "Resource": "*"
    }
  ]
}

3. Optional KMS Key Policy

If your inventory destination bucket or manifests are encrypted with SSE-KMS using a Customer Managed Key (CMK), grant decryption permissions on that specific key ARN:

{
  "Sid": "KmsDecryptAccess",
  "Effect": "Allow",
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey"
  ],
  "Resource": "arn:aws:kms:REGION:ACCOUNT_ID:key/KEY_ID"
}

Least-Privilege Scoping & Wildcard Rationale

All permissions follow strict least-privilege scoping with wildcards minimized. For enterprise security audits, each wildcard (*) is used only where mandated by AWS IAM APIs:

• arn:aws:s3:::<bucket>/* (S3 Object Keys):
  • Why * is required: Bucket ARNs (arn:aws:s3:::<bucket>) only apply to bucket actions like s3:ListBucket. S3 requires object ARNs (arn:aws:s3:::<bucket>/*) for object-level read actions (s3:GetObject) and intermediate state storage (s3:PutObject, s3:DeleteObject, s3:DeleteObjectVersion under .s3lim/).
• arn:aws:states:*:*:stateMachine:s3lim-* (Step Functions Execution):
  • Why prefix is used: Scopes state machine invocation and status inspection strictly to s3lim Step Functions workflows.
• cloudwatch:PutMetricData (Resource: "*"):
  • Why * is required: AWS CloudWatch does not support resource-level permissions (ARNs) for the PutMetricData API. Per AWS IAM authorization specifications, CloudWatch metric publishing requires "Resource": "*".
• aws-marketplace:BatchMeterUsage / GetEntitlements (Resource: "*"):
  • Why * is required: AWS Marketplace Metering APIs do not support resource-level ARNs. AWS requires "Resource": "*" to report usage dimensions.
• arn:aws:logs:*:*:log-group:/aws/lambda/s3lim-* & arn:aws:sqs:*:*:s3lim-*:
  • Why region/account * is used: Scopes resource access strictly to s3lim prefixes while allowing multi-region or StackSet operation. You can optionally replace *:* with your explicit AWS Region and Account ID (e.g. arn:aws:logs:us-east-1:123456789012:log-group:/aws/lambda/s3lim-*).

Configure Analysis Trigger

By default, s3lim sets up a daily scheduled trigger (cron) to scan the destination bucket for new manifests.

If you want real-time analysis immediately upon report delivery:

1. Copy the CoreFunctionArn from the CloudFormation stack outputs.
2. Navigate to your S3 Inventory destination bucket in the AWS Console.
3. Go to the Properties tab and scroll to Event notifications.
4. Click Create event notification and configure:
   • Prefix: The folder path of your inventory reports (e.g., inventory/).
   • Suffix: manifest.json.
   • Event types: All object create events (s3:ObjectCreated:*).
   • Destination: Lambda Function (select your s3lim function).

Verification

Once a new S3 Inventory report is delivered, the Lambda will automatically execute. You can view the output analysis and metrics in your AWS CloudWatch dashboard.