hirejared-kw-db/docs/project-docs/05-operations/configuration.md

System Configuration Documentation

Overview

This document details the configuration settings and files required for the keyword management system. The primary configuration is managed through the .database file, with additional settings managed through script-specific configuration files.

Database Configuration

.database File

Located in project root directory: /.database

# Database Connection Settings
DB_USER=postgres
DB_PASSWORD=your_password
DB_HOST=localhost
DB_PORT=5432
DB_NAME=keyword_management

# Backup Configuration
BACKUP_DIR=backups
BACKUP_RETENTION_DAYS=30

# Logging Configuration
LOG_DIR=logs
LOG_LEVEL=INFO  # DEBUG, INFO, WARN, ERROR

# Import Settings
IMPORT_BATCH_SIZE=1000
MAX_IMPORT_ERRORS=50

Security Requirements

• File permissions: 600 (-rw-------)
• Owner: Script execution user
• Group: Script execution group
• Location: Project root only
• No version control tracking

Environment Variables

Alternative to .database file for CI/CD environments:

export KW_DB_USER=postgres
export KW_DB_PASSWORD=your_password
export KW_DB_HOST=localhost
export KW_DB_PORT=5432
export KW_DB_NAME=keyword_management

Configuration Management

Loading Configuration

Scripts should load configuration in this order:

1. Environment variables (if present)
2. .database file
3. Default values

Example configuration loading:

#!/bin/bash

# Load configuration with fallback to defaults
load_config() {
    # Try environment variables first
    DB_USER=${KW_DB_USER:-}
    DB_PASSWORD=${KW_DB_PASSWORD:-}
    DB_HOST=${KW_DB_HOST:-}
    DB_PORT=${KW_DB_PORT:-}
    DB_NAME=${KW_DB_NAME:-}

    # If any required env vars are missing, try .database file
    if [[ -z "$DB_USER" || -z "$DB_PASSWORD" ]]; then
        if [[ -f "/.database" ]]; then
            source "/.database"
        else
            echo "Error: No configuration found"
            exit 1
        fi
    fi

    # Apply defaults if still not set
    DB_HOST=${DB_HOST:-localhost}
    DB_PORT=${DB_PORT:-5432}
    BACKUP_RETENTION_DAYS=${BACKUP_RETENTION_DAYS:-30}
    LOG_LEVEL=${LOG_LEVEL:-INFO}
}

Directory Structure Configuration

Required Directories

# Create required directories
mkdir -p "${BACKUP_DIR}"
mkdir -p "${LOG_DIR}"
chmod 750 "${BACKUP_DIR}" "${LOG_DIR}"

Directory Permissions

backups/    drwxr-x---  Script user:script group
logs/       drwxr-x---  Script user:script group

Logging Configuration

Log Files

• Format: YYYY-MM-DD-script-name.log
• Location: ${LOG_DIR}
• Rotation: Daily
• Retention: 30 days

Log Levels

LOG_LEVEL_DEBUG=0
LOG_LEVEL_INFO=1
LOG_LEVEL_WARN=2
LOG_LEVEL_ERROR=3

# Logging function
log_message() {
    local level=$1
    local message=$2
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    
    if [[ $level -ge ${LOG_LEVEL:-1} ]]; then
        echo "${timestamp} [${level}] ${message}" >> "${LOG_FILE}"
    fi
}

Import Configuration

GKWP Import Settings

# GKWP specific settings
GKWP_BATCH_SIZE=1000
GKWP_MAX_ERRORS=50
GKWP_TIMEOUT=300

SEMRush Import Settings

# SEMRush specific settings
SEMRUSH_BATCH_SIZE=1000
SEMRUSH_MAX_ERRORS=50
SEMRUSH_TIMEOUT=300

Plain List Import Settings

# Plain list import settings
PLAIN_BATCH_SIZE=5000
PLAIN_MAX_ERRORS=100

Error Handling Configuration

Error Levels

# Error level definitions
ERR_NONE=0
ERR_WARN=1
ERR_ERROR=2
ERR_FATAL=3

# Error handling configuration
MAX_RETRIES=3
RETRY_DELAY=5

Error Handling Function

handle_error() {
    local error_level=$1
    local message=$2
    local script_name=$(basename "$0")
    
    case $error_level in
        $ERR_WARN)
            log_message "WARN" "${script_name}: ${message}"
            ;;
        $ERR_ERROR)
            log_message "ERROR" "${script_name}: ${message}"
            return 1
            ;;
        $ERR_FATAL)
            log_message "FATAL" "${script_name}: ${message}"
            exit 1
            ;;
    esac
}

Testing Configuration

Test Database Settings

# Test database configuration
TEST_DB_NAME=keyword_management_test
TEST_DATA_DIR=tests/data

Test Framework Configuration

# Test framework settings
TEST_TIMEOUT=30
TEST_PARALLEL=false
TEST_VERBOSE=true

Backup Configuration

Backup Settings

# Backup configuration
BACKUP_COMPRESSION=gzip
BACKUP_PREFIX=keyword_management
BACKUP_SUFFIX=.sql.gz

Backup Rotation

# Backup rotation settings
DAILY_RETENTION=7
WEEKLY_RETENTION=4
MONTHLY_RETENTION=12

Configuration Validation

Validation Script

#!/bin/bash

validate_config() {
    # Required variables
    local required_vars=(
        "DB_USER"
        "DB_PASSWORD"
        "DB_HOST"
        "DB_PORT"
        "DB_NAME"
    )
    
    # Check required variables
    for var in "${required_vars[@]}"; do
        if [[ -z "${!var}" ]]; then
            echo "Error: Required variable $var is not set"
            exit 1
        fi
    done
    
    # Validate directories
    if [[ ! -d "${BACKUP_DIR}" ]]; then
        echo "Error: Backup directory does not exist"
        exit 1
    fi
    
    if [[ ! -d "${LOG_DIR}" ]]; then
        echo "Error: Log directory does not exist"
        exit 1
    fi
}

Configuration Updates

Update Process

1. Stop running scripts
2. Backup current configuration
3. Apply changes
4. Validate new configuration
5. Restart scripts

Version Control

• Keep configuration templates in version control
• Document changes in changelog
• Include update instructions

Would you like me to expand on any of these configuration aspects or provide additional examples?