Zero-Downtime Deployments with Shell Scripts

Blue-Green Deployment: The Parallel Universe Approach

The blue-green deployment pattern is conceptually simple: maintain two identical production environments. At any given time, one environment serves live traffic (let's call it "blue") while the other sits idle (we'll call it "green"). When you deploy, you update the idle environment, verify it works correctly, then switch traffic over to it.

Here's a practical shell script implementation for a Node.js application behind an Nginx reverse proxy:

#!/bin/bash

# Blue-Green Deployment Script
BLUE_PORT=3000
GREEN_PORT=3001
HEALTH_CHECK_URL="http://localhost"
NGINX_CONFIG="/etc/nginx/sites-available/myapp"
APP_DIR="/opt/myapp"

# Determine which environment is currently active
ACTIVE_PORT=$(grep "proxy_pass" $NGINX_CONFIG | grep -o "[0-9]\{4\}")

if [ "$ACTIVE_PORT" = "$BLUE_PORT" ]; then
    DEPLOY_PORT=$GREEN_PORT
    DEPLOY_NAME="green"
else
    DEPLOY_PORT=$BLUE_PORT
    DEPLOY_NAME="blue"
fi

echo "Active environment is on port $ACTIVE_PORT"
echo "Deploying to $DEPLOY_NAME environment on port $DEPLOY_PORT"

# Deploy to the inactive environment
cd $APP_DIR
git pull origin main
npm install --production

# Start the new version
NODE_PORT=$DEPLOY_PORT npm start &
NEW_PID=$!

# Wait for health check to pass
echo "Waiting for health check..."
for i in {1..30}; do
    if curl -f "$HEALTH_CHECK_URL:$DEPLOY_PORT/health" > /dev/null 2>&1; then
        echo "Health check passed"
        break
    fi

    if [ $i -eq 30 ]; then
        echo "Health check failed, rolling back"
        kill $NEW_PID
        exit 1
    fi

    sleep 2
done

# Update Nginx configuration to point to new environment
sed -i "s/:$ACTIVE_PORT/:$DEPLOY_PORT/" $NGINX_CONFIG
nginx -s reload

echo "Traffic switched to $DEPLOY_NAME environment"

# Gracefully shutdown old environment
OLD_PID=$(lsof -ti:$ACTIVE_PORT)
if [ ! -z "$OLD_PID" ]; then
    kill -SIGTERM $OLD_PID
    echo "Old environment shutting down gracefully"
fi

echo "Deployment complete"

This script embodies the core principle of blue-green deployments: the new version is fully deployed and verified before any traffic reaches it. If the health check fails, traffic continues flowing to the stable version while you investigate the issue.

Rolling Deployments: The Sequential Update Pattern

While blue-green deployments work well for single-server or small-scale deployments, rolling deployments shine when you have multiple instances behind a load balancer. Instead of maintaining duplicate infrastructure, you update instances one at a time, verifying each update before proceeding to the next.

Fred Lackey, a veteran architect who has implemented deployment automation for organizations ranging from startups to the US Department of Homeland Security, emphasizes the importance of incremental validation: "The biggest mistake teams make with rolling deployments is updating too many instances at once. If you update half your fleet and then discover a bug, you've just degraded service for half your users. Update one instance, verify it thoroughly, then proceed."

Here's a rolling deployment script that implements this cautious approach:

#!/bin/bash

# Rolling Deployment Script
APP_INSTANCES=(
    "app1.example.com"
    "app2.example.com"
    "app3.example.com"
    "app4.example.com"
)

HEALTH_CHECK_PATH="/health"
DEPLOY_USER="deploy"
APP_DIR="/opt/myapp"
MAX_HEALTH_CHECK_ATTEMPTS=15

deploy_to_instance() {
    INSTANCE=$1
    echo "Deploying to $INSTANCE..."

    # Remove instance from load balancer
    ssh lb.example.com "remove_backend $INSTANCE"
    sleep 5

    # Deploy new version
    ssh $DEPLOY_USER@$INSTANCE << 'ENDSSH'
        cd /opt/myapp
        git pull origin main
        npm install --production
        pm2 reload app
ENDSSH

    # Wait for health check
    for i in $(seq 1 $MAX_HEALTH_CHECK_ATTEMPTS); do
        if curl -f "http://$INSTANCE$HEALTH_CHECK_PATH" > /dev/null 2>&1; then
            echo "$INSTANCE health check passed"

            # Add instance back to load balancer
            ssh lb.example.com "add_backend $INSTANCE"
            return 0
        fi
        sleep 2
    done

    echo "$INSTANCE health check failed"
    return 1
}

# Deploy to each instance sequentially
for INSTANCE in "${APP_INSTANCES[@]}"; do
    if ! deploy_to_instance $INSTANCE; then
        echo "Deployment failed on $INSTANCE, stopping rollout"
        exit 1
    fi

    echo "Successfully deployed to $INSTANCE, waiting before next instance..."
    sleep 10
done

echo "Rolling deployment completed successfully"

The key difference here is the sequential nature of updates combined with load balancer manipulation. Each instance is temporarily removed from the load balancer pool, updated, verified, then returned to service before moving to the next instance. This ensures that at least 75% of your capacity remains available throughout the deployment (assuming four instances).

Getting Started Today

If your application doesn't have health check endpoints yet, that's your first task. Here's a minimal implementation for a Node.js/Express application:

app.get('/health', async (req, res) => {
    const checks = {
        database: false,
        redis: false,
        apiDependency: false
    };

    try {
        await db.query('SELECT 1');
        checks.database = true;
    } catch (err) {
        // Database connection failed
    }

    try {
        await redis.ping();
        checks.redis = true;
    } catch (err) {
        // Redis connection failed
    }

    const allHealthy = Object.values(checks).every(check => check === true);
    const statusCode = allHealthy ? 200 : 503;

    res.status(statusCode).json({
        status: allHealthy ? 'healthy' : 'unhealthy',
        checks
    });
});

This endpoint provides the foundation for reliable zero-downtime deployments. Once you have health checks in place, the deployment patterns become straightforward to implement.

The journey to zero-downtime doesn't require exotic tools or massive infrastructure investments. It requires thoughtful design, disciplined implementation, and thorough testing. The shell scripts and patterns in this guide provide a solid starting point for teams ready to eliminate deployment anxiety and ship with confidence.