aws-wtf
from opsyhq/opsy
AI DevOps Agent that won't take down your production
npx skills add https://github.com/opsyhq/opsy --skill aws-wtfSKILL.md
AWS WTF Skill for Opsy
Explains every charge on your AWS bill — what it is, why you're paying, and what resource caused it.
⚠️ Mandatory Output
You MUST automatically generate and save a CSV file at the end of every analysis. Do not wait for the user to ask. The analysis is incomplete until the CSV exists.
Step 1: Cost Explorer (Two Queries Required)
Run BOTH queries to detect credit coverage:
# Query 1: Normal (shows $0 if credits cover costs)
aws ce get-cost-and-usage \
--time-period Start=$(date -v-30d +%Y-%m-%d),End=$(date +%Y-%m-%d) \
--granularity MONTHLY \
--metrics "UnblendedCost" "UsageQuantity" \
--group-by Type=DIMENSION,Key=SERVICE
# Query 2: Exclude credits (shows ACTUAL usage cost)
aws ce get-cost-and-usage \
--time-period Start=$(date -v-30d +%Y-%m-%d),End=$(date +%Y-%m-%d) \
--granularity MONTHLY \
--metrics "UnblendedCost" "UsageQuantity" \
--group-by Type=DIMENSION,Key=SERVICE \
--filter '{"Not": {"Dimensions": {"Key": "RECORD_TYPE", "Values": ["Credit", "Refund"]}}}'
Interpretation:
- Query 1 = $0, Query 2 = $X → Credits covering $X actual usage
- Query 1 = Query 2 = $0 → Real free tier
- Query 1 = Query 2 = $X → Normal billing
If credits detected, warn user: "Your bill shows $0 but actual usage is $X/month. When credits run out, you WILL be charged."
Step 2: Identify ALL Regions
aws ce get-cost-and-usage \
--time-period Start=$(date -v-30d +%Y-%m-%d),End=$(date +%Y-%m-%d) \
--granularity MONTHLY \
--metrics "UnblendedCost" \
--group-by Type=DIMENSION,Key=REGION \
--filter '{"Not": {"Dimensions": {"Key": "RECORD_TYPE", "Values": ["Credit", "Refund"]}}}'
You MUST enumerate resources in EVERY region showing charges > $0.01.
Step 3: Enumerate ALL Resources
For each region with charges, query ALL applicable services using --region $REGION:
aws ec2 describe-instancesaws ec2 describe-volumesaws ec2 describe-snapshots --owner-ids selfaws ec2 describe-addressesaws ec2 describe-nat-gatewaysaws rds describe-db-instancesaws elbv2 describe-load-balancersaws ecs list-clusters→describe-clusters→list-services→list-tasksaws eks list-clusters→describe-clusteraws lambda list-functionsaws s3api list-buckets(global) →get-bucket-locationper bucketaws ecr describe-repositoriesaws secretsmanager list-secretsaws logs describe-log-groupsaws kms list-keysaws route53 list-hosted-zones(global)
ARN Construction
For resources without ARN in response, construct: arn:aws:{service}:{region}:{account}:{resource-type}/{id}
Examples:
- EC2:
arn:aws:ec2:us-east-1:123456789012:instance/i-abc123 - EBS:
arn:aws:ec2:us-east-1:123456789012:volume/vol-abc123 - S3:
arn:aws:s3:::bucket-name
⚠️ CRITICAL: Every Charge Must Have Identification
Every row in the CSV MUST have a resource identifier (ARN + resource_id) UNLESS it is truly untraceable.
What CAN Be Traced (MUST have ARN)
ANY charge from these services MUST be traced to a specific resource:
| Service | Has ARN | Example |
|---|---|---|
| EC2 | ✅ Always | Instance, Volume, Snapshot, EIP, NAT Gateway |
| S3 | ✅ Always | Bucket |
| RDS | ✅ Always | Instance |
| ECS/EKS | ✅ Always | Cluster, Service, Task |
| Lambda | ✅ Always | Function |
| ALB/NLB | ✅ Always | Load Balancer |
| CloudWatch Logs | ✅ Always | Log Group |
| Secrets Manager | ✅ Always | Secret |
| KMS | ✅ Always | Key |
| ECR | ✅ Always | Repository |
| Route 53 | ✅ Always | Hosted Zone |
If Cost Explorer shows charges for these but you can't find the resource → it was deleted mid-period. Put ARN as DELETED - {service} and note in description.
What CANNOT Be Traced (N/A allowed)
Only these charges are truly untraceable to a single resource:
| Charge Type | Why Untraceable |
|---|---|
| Data Transfer Out | Aggregated from multiple sources |
| Data Transfer Inter-Region | No single source |
| Data Transfer Inter-AZ | No single source |
| Support Plan | Account-level |
| Tax | Account-level |
| CloudWatch Custom Metrics (aggregated) | No single dimension |
For these only: use arn: N/A - Service-level charge or N/A - Account-level charge
Verification Rule
Before marking ANY charge as N/A, ask: "Is there a specific AWS resource that caused this?"
- If YES → find it, get its ARN
- If NO (only data transfer, support, tax) → N/A is acceptable
Elastic IP Verification
Check AssociationId before calling an IP "unattached":
AssociationIdpresent → attached (even ifInstanceIdis empty)NetworkInterfaceOwnerId = "amazon-..."→ service-managed (ALB, RDS, NAT)
Public IPv4 Charges
AWS charges $0.005/hr ($3.60/mo) per public IPv4. Find all sources:
- EC2 public IPs, Elastic IPs, internet-facing ALBs, NAT Gateways
CSV Output (Mandatory)
Filename: aws-wtf-{account-id}-{date}.csv
account_id,resourc
...