In the dynamic world of web development, efficient deployment workflows are essential. When pushing code changes, invalidating outdated content within CloudFront’s edge caches guarantees users experience the latest application version. However, traditional invalidation methods often trigger unnecessary cache refreshes, compromising deployment speed and resource utilization.

Efficient cloudfront cache invalidation is crucial for maintaining up-to-date content delivery while minimizing unnecessary invalidations. This whitepaper provides a step-by-step guide to create a Bitbucket Pipeline that selectively invalidates CloudFront cache for files changed in pull requests, addressing the common issue of over-invalidation.

Addressing CloudFront Cache Invalidation Challenges:

Current CloudFront invalidation methods can lead to inefficiencies:

Increased latency for users
Higher AWS costs due to unnecessary invalidations
Potential performance impacts on CloudFront distribution
All-or-Nothing Invalidation: Invalidating the entire cache directory with each deployment refreshes everything, even unchanged content. This is resource-intensive and time-consuming.
Manual Path Selection: Manually specifying paths for invalidation is error-prone and tedious, especially for large deployments.

Crafting a Refined Invalidation Strategy with Bitbucket Pipelines:

Bitbucket Pipelines, a built-in continuous integration/continuous delivery (CI/CD) tool for Bitbucket, empowers you to automate tasks within your development workflow. By integrating the aws CLI with your Bitbucket pipeline, you can create a script that meticulously identifies only the files modified in the pull request and triggers their invalidation within the CloudFront cache.

Core Components of the Pipeline Script:

AWS CLI Installation: Utilize a pipeline step to install the AWS CLI within your build environment.
Identifying Changed Files: Employ the git diff command to generate a list of files modified in the pull request.
Constructing Invalidation Paths: Loop through the list of changed files and build a comma-separated string containing their relative paths within the CloudFront distribution.
AWS CLI Integration: Integrate the aws cloudfront create-invalidation command with the constructed paths to trigger the targeted cache invalidation on your CloudFront distribution.

Benefits of Targeted Invalidation

Improved content delivery efficiency
Reduced AWS costs
Faster updates for changed content

Step-by-Step Implementation Guide for CloudFront Cache Invalidation

Step 1: Configure Bitbucket Pipeline

Create a `bitbucket-pipelines.yml` file in your repository root
Define the pipeline structure:

				
					```yaml
pipelines:
  pull-requests:
    '**':
      - step:
          name: Invalidate CloudFront Cache
          script:
            - pip install awscli

Step 2: Create Invalidation Script

1. Create `invalidate-cloudfront.sh` in the repository root
2. Implement the script logic:

				
					```bash
#!/bin/bash

# Get list of changed files
CHANGED_FILES=$(git diff --name-only origin/${BITBUCKET_PR_DESTINATION_BRANCH}...${BITBUCKET_BRANCH})

# Filter for relevant file types (e.g., html, css, js)
RELEVANT_FILES=$(echo "$CHANGED_FILES" | grep -E '\.(html|css|js)$')

# Prepare invalidation paths
INVALIDATION_PATHS=""
for file in $RELEVANT_FILES; do
  INVALIDATION_PATHS="$INVALIDATION_PATHS /$file"
done

# Perform invalidation if there are paths to invalidate
if [ ! -z "$INVALIDATION_PATHS" ]; then
  aws cloudfront create-invalidation --distribution-id $CLOUDFRONT_DISTRIBUTION_ID --paths $INVALIDATION_PATHS
else
  echo "No relevant files changed. Skipping invalidation."
fi```

Step 3: Configure AWS Credentials

Add AWS credentials as Bitbucket repository variables:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_DEFAULT_REGION
Add CloudFront distribution ID as a variable:
- CLOUDFRONT_DISTRIBUTION_ID

Step 4: Optimize for Large Pull Requests

Implement batching for large numbers of changed files:

				
					Bash
# ... (previous script content)

# Batch invalidations (max 3000 paths per invalidation)
BATCH_SIZE=3000
TOTAL_PATHS=$(echo "$INVALIDATION_PATHS" | wc -w)
BATCHES=$(( ($TOTAL_PATHS + $BATCH_SIZE - 1) / $BATCH_SIZE ))

for ((i=0; i<$BATCHES; i++)); do
  START=$(($i * $BATCH_SIZE + 1))
  END=$(((($i + 1) * $BATCH_SIZE) > $TOTAL_PATHS ? $TOTAL_PATHS : (($i + 1) * $BATCH_SIZE)))
  BATCH_PATHS=$(echo $INVALIDATION_PATHS | cut -d' ' -f$START-$END)
  
  aws cloudfront create-invalidation --distribution-id $CLOUDFRONT_DISTRIBUTION_ID --paths $BATCH_PATHS
done

Step 5: Implement Error Handling and Logging

Add error checking and logging to the script:

				
					```bash
# ... (previous script content)

# Function for logging
log_message() {
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"
}

# Error handling
set -e
trap 'log_message "Error occurred. Exiting."; exit 1' ERR

# ... (rest of the script with log_message calls added)

Best Practices and Optimizations

Use file type filtering to focus on cacheable content
Implement rate limiting to avoid API throttling
Consider using AWS SDK instead of AWS CLI for more control
Regularly review and update the invalidation strategy

Frequently Asked Questions

1. Q: How does this solution improve over invalidating everything?
A: It reduces unnecessary invalidations, lowering costs and improving overall CDN performance.
1. Q: Can this approach handle large-scale deployments?
A: Yes, the batching mechanism allows it to handle large numbers of file changes efficiently.
1. Q: How do we ensure critical updates are not missed?
A: The script can be configured to always invalidate critical paths in addition to changed files.
1. Q: Is this method compatible with other CI/CD tools besides Bitbucket Pipelines?
A: While optimized for Bitbucket, the core logic can be adapted to other CI/CD platforms.
1. Q: How can we monitor the effectiveness of this invalidation strategy?
A: Implement CloudWatch metrics to track invalidation requests and cache hit rates.

Using Cloud Workflow Automation in Businesses can take operations to the Next Level

Ready to explore further?

Set Up Advanced Bank Reconciliation Import For Bai2

05 Oct

All Blogs

CloudFront Cache Invalidation Optimization: Bitbucket Pipelines

Posted By

Syed Muhammad Kashif