Complete Guide: Production-Ready SFTP Automation

Automating file transfers between servers is a common task, but doing it securely and reliably requires more than a simple script. This guide provides a copy-paste production-ready solution that features retry logic, logging, error alerting, and high-security user restrictions.

Architecture Overview

• Source Server: Initiates the connection using an automated script.
• Destination Server: Receives files into a restricted “Jail” (Chroot) environment for maximum security.
• Authentication: SSH Keys (Ed25519) without passphrases.

Phase 1: Secure Server Configuration

First, we configure the destination server to accept files but reject shell commands.

1. Set Up the Destination User

Run these commands on the Destination Server:

# Create a restricted user with no shell access
sudo useradd -s /bin/false -m -d /home/sftpin sftpin
sudo passwd sftpin

# CRITICAL: Prepare the Jail Directory
# The root of the chroot directory (/data) MUST be owned by root
sudo mkdir -p /data/incoming
sudo chown root:root /data
sudo chmod 755 /data

# Give the user ownership of the incoming folder only
sudo chown sftpin:sftpin /data/incoming

2. Configure SSHD

Edit the SSH config file: sudo nano /etc/ssh/sshd_config. Add the following block to the very bottom:

Match User sftpin
    ChrootDirectory /data
    ForceCommand internal-sftp
    PermitTunnel no
    X11Forwarding no
    AllowTcpForwarding no
    PasswordAuthentication no
    PubkeyAuthentication yes

3. Set Up SSH Keys

Run this on the Source Server to generate keys:

# Create a dedicated automation user
sudo useradd -m -s /bin/bash sftpauto
sudo su - sftpauto

# Generate key (Press Enter for all prompts)
ssh-keygen -t ed25519 -f ~/.ssh/sftp_auto_key -N ""

Copy the Key: Copy the content of ~/.ssh/sftp_auto_key.pub from the Source Server and paste it into /home/sftpin/.ssh/authorized_keys on the Destination Server. Ensure the permissions on the authorized_keys file are set to 600.

Phase 2: The Enterprise Script

Save this script as /opt/sftp-auto/sftp_transfer.sh on the Source Server. This version solves the “Shell vs SFTP” conflict by using an SFTP-safe verification method.

#!/bin/bash
# -------------------------------------------------------------------
# ENTERPRISE SFTP TRANSFER SCRIPT
# -------------------------------------------------------------------

CONFIG_FILE="/opt/sftp-auto/config.conf"
LOCK_FILE="/tmp/sftp_transfer.lock"
MAX_LOCK_AGE=3600

# Load Config
if [ -r "$CONFIG_FILE" ]; then
    source "$CONFIG_FILE"
else
    echo "Config file missing!" && exit 1
fi

log_message() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$1] $2" | tee -a "$LOG_FILE"
}

# Acquire Lock to prevent overlapping runs
if [ -f "$LOCK_FILE" ]; then
    # Check if lock is stale (older than 1 hour)
    if [ $(($(date +%s) - $(stat -c %Y "$LOCK_FILE"))) -gt $MAX_LOCK_AGE ]; then
        rm -f "$LOCK_FILE"
    else
        echo "Script is already running." && exit 1
    fi
fi
touch "$LOCK_FILE"
trap "rm -f '$LOCK_FILE'" EXIT

# Transfer Logic
transfer_file() {
    local file=$1
    local base_name=$(basename "$file")
    
    log_message "INFO" "Starting transfer: $base_name"

    # Create temporary batch file for SFTP
    local batch=$(mktemp)
    echo "cd $DEST_DIR" > "$batch"
    echo "put $file" >> "$batch"
    echo "ls -l $DEST_DIR/$base_name" >> "$batch" # For Verification
    echo "bye" >> "$batch"

    # Execute SFTP
    OUTPUT=$(sftp -o BatchMode=yes -o IdentityFile="$PRIVATE_KEY" -b "$batch" "$DEST_USER@$DEST_HOST" 2>&1)
    EXIT_CODE=$?
    
    rm -f "$batch"

    # Verify Success
    if [ $EXIT_CODE -eq 0 ] && echo "$OUTPUT" | grep -q "$base_name"; then
        log_message "SUCCESS" "Transferred: $base_name"
        
        # Archive the local file
        mkdir -p "$ARCHIVE_DIR/$(date +%Y/%m)"
        mv "$file" "$ARCHIVE_DIR/$(date +%Y/%m)/"
        return 0
    else
        log_message "ERROR" "Failed to transfer $base_name. Output: $OUTPUT"
        return 1
    fi
}

# Main Loop
find "$SOURCE_DIR" -maxdepth 1 -type f -name "$FILE_PATTERN" | while read file; do
    transfer_file "$file"
done

Phase 3: Configuration

Create the config file at /opt/sftp-auto/config.conf:

DEST_HOST="192.168.x.x"
DEST_USER="sftpin"
DEST_DIR="/incoming" # This is relative to the Chroot (/data)
PRIVATE_KEY="/home/sftpauto/.ssh/sftp_auto_key"

SOURCE_DIR="/data/outgoing"
ARCHIVE_DIR="/data/archive"
FILE_PATTERN="*.csv" # Or "*" for all files
LOG_FILE="/var/log/sftp_transfer.log"

Important Notes & Troubleshooting