c-relay/docs/unified_startup_implementation_plan.md

Unified Startup Implementation Plan

Overview

This document provides a detailed implementation plan for refactoring the startup sequence to use atomic config creation followed by CLI overrides. This plan breaks down the work into discrete, testable steps.

---

Phase 1: Create New Functions in config.c

Step 1.1: Implement populate_all_config_values_atomic()

Location: src/config.c

Purpose: Create complete config table in single transaction for first-time startup

Function Signature:

int populate_all_config_values_atomic(const cli_options_t* cli_options);

Implementation Details:

int populate_all_config_values_atomic(const cli_options_t* cli_options) {
    if (!g_database) {
        DEBUG_ERROR("Database not initialized");
        return -1;
    }

    // Begin transaction
    char* err_msg = NULL;
    int rc = sqlite3_exec(g_database, "BEGIN TRANSACTION;", NULL, NULL, &err_msg);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to begin transaction: %s", err_msg);
        sqlite3_free(err_msg);
        return -1;
    }

    // Prepare INSERT statement
    sqlite3_stmt* stmt = NULL;
    const char* sql = "INSERT INTO config (key, value) VALUES (?, ?)";
    rc = sqlite3_prepare_v2(g_database, sql, -1, &stmt, NULL);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to prepare statement: %s", sqlite3_errmsg(g_database));
        sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
        return -1;
    }

    // Insert all default config values
    for (size_t i = 0; i < sizeof(DEFAULT_CONFIG_VALUES) / sizeof(DEFAULT_CONFIG_VALUES[0]); i++) {
        sqlite3_reset(stmt);
        sqlite3_bind_text(stmt, 1, DEFAULT_CONFIG_VALUES[i].key, -1, SQLITE_STATIC);
        sqlite3_bind_text(stmt, 2, DEFAULT_CONFIG_VALUES[i].value, -1, SQLITE_STATIC);
        
        rc = sqlite3_step(stmt);
        if (rc != SQLITE_DONE) {
            DEBUG_ERROR("Failed to insert config key '%s': %s", 
                       DEFAULT_CONFIG_VALUES[i].key, sqlite3_errmsg(g_database));
            sqlite3_finalize(stmt);
            sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
            return -1;
        }
    }

    // Insert admin_pubkey from cache
    sqlite3_reset(stmt);
    sqlite3_bind_text(stmt, 1, "admin_pubkey", -1, SQLITE_STATIC);
    sqlite3_bind_text(stmt, 2, g_unified_cache.admin_pubkey, -1, SQLITE_STATIC);
    rc = sqlite3_step(stmt);
    if (rc != SQLITE_DONE) {
        DEBUG_ERROR("Failed to insert admin_pubkey: %s", sqlite3_errmsg(g_database));
        sqlite3_finalize(stmt);
        sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
        return -1;
    }

    // Insert relay_pubkey from cache
    sqlite3_reset(stmt);
    sqlite3_bind_text(stmt, 1, "relay_pubkey", -1, SQLITE_STATIC);
    sqlite3_bind_text(stmt, 2, g_unified_cache.relay_pubkey, -1, SQLITE_STATIC);
    rc = sqlite3_step(stmt);
    if (rc != SQLITE_DONE) {
        DEBUG_ERROR("Failed to insert relay_pubkey: %s", sqlite3_errmsg(g_database));
        sqlite3_finalize(stmt);
        sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
        return -1;
    }

    sqlite3_finalize(stmt);

    // Commit transaction
    rc = sqlite3_exec(g_database, "COMMIT;", NULL, NULL, &err_msg);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to commit transaction: %s", err_msg);
        sqlite3_free(err_msg);
        sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
        return -1;
    }

    DEBUG_INFO("Successfully populated all config values atomically");
    return 0;
}

Testing:

• Verify transaction atomicity (all or nothing)
• Verify all DEFAULT_CONFIG_VALUES inserted
• Verify admin_pubkey and relay_pubkey inserted
• Verify error handling on failure

---

Step 1.2: Implement apply_cli_overrides_atomic()

Location: src/config.c

Purpose: Apply CLI overrides to existing config table in single transaction

Function Signature:

int apply_cli_overrides_atomic(const cli_options_t* cli_options);

Implementation Details:

int apply_cli_overrides_atomic(const cli_options_t* cli_options) {
    if (!g_database) {
        DEBUG_ERROR("Database not initialized");
        return -1;
    }

    if (!cli_options) {
        DEBUG_ERROR("CLI options is NULL");
        return -1;
    }

    // Check if any overrides exist
    bool has_overrides = false;
    if (cli_options->port_override > 0) has_overrides = true;
    if (cli_options->admin_pubkey_override[0] != '\0') has_overrides = true;
    if (cli_options->relay_privkey_override[0] != '\0') has_overrides = true;

    if (!has_overrides) {
        DEBUG_INFO("No CLI overrides to apply");
        return 0;
    }

    // Begin transaction
    char* err_msg = NULL;
    int rc = sqlite3_exec(g_database, "BEGIN TRANSACTION;", NULL, NULL, &err_msg);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to begin transaction: %s", err_msg);
        sqlite3_free(err_msg);
        return -1;
    }

    // Prepare UPDATE statement
    sqlite3_stmt* stmt = NULL;
    const char* sql = "UPDATE config SET value = ? WHERE key = ?";
    rc = sqlite3_prepare_v2(g_database, sql, -1, &stmt, NULL);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to prepare statement: %s", sqlite3_errmsg(g_database));
        sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
        return -1;
    }

    // Apply port override
    if (cli_options->port_override > 0) {
        char port_str[16];
        snprintf(port_str, sizeof(port_str), "%d", cli_options->port_override);
        
        sqlite3_reset(stmt);
        sqlite3_bind_text(stmt, 1, port_str, -1, SQLITE_TRANSIENT);
        sqlite3_bind_text(stmt, 2, "relay_port", -1, SQLITE_STATIC);
        
        rc = sqlite3_step(stmt);
        if (rc != SQLITE_DONE) {
            DEBUG_ERROR("Failed to update relay_port: %s", sqlite3_errmsg(g_database));
            sqlite3_finalize(stmt);
            sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
            return -1;
        }
        DEBUG_INFO("Applied CLI override: relay_port = %s", port_str);
    }

    // Apply admin_pubkey override
    if (cli_options->admin_pubkey_override[0] != '\0') {
        sqlite3_reset(stmt);
        sqlite3_bind_text(stmt, 1, cli_options->admin_pubkey_override, -1, SQLITE_STATIC);
        sqlite3_bind_text(stmt, 2, "admin_pubkey", -1, SQLITE_STATIC);
        
        rc = sqlite3_step(stmt);
        if (rc != SQLITE_DONE) {
            DEBUG_ERROR("Failed to update admin_pubkey: %s", sqlite3_errmsg(g_database));
            sqlite3_finalize(stmt);
            sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
            return -1;
        }
        DEBUG_INFO("Applied CLI override: admin_pubkey");
    }

    // Apply relay_privkey override
    if (cli_options->relay_privkey_override[0] != '\0') {
        sqlite3_reset(stmt);
        sqlite3_bind_text(stmt, 1, cli_options->relay_privkey_override, -1, SQLITE_STATIC);
        sqlite3_bind_text(stmt, 2, "relay_privkey", -1, SQLITE_STATIC);
        
        rc = sqlite3_step(stmt);
        if (rc != SQLITE_DONE) {
            DEBUG_ERROR("Failed to update relay_privkey: %s", sqlite3_errmsg(g_database));
            sqlite3_finalize(stmt);
            sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
            return -1;
        }
        DEBUG_INFO("Applied CLI override: relay_privkey");
    }

    sqlite3_finalize(stmt);

    // Commit transaction
    rc = sqlite3_exec(g_database, "COMMIT;", NULL, NULL, &err_msg);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to commit transaction: %s", err_msg);
        sqlite3_free(err_msg);
        sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
        return -1;
    }

    // Invalidate cache to force refresh
    invalidate_config_cache();

    DEBUG_INFO("Successfully applied CLI overrides atomically");
    return 0;
}

Testing:

• Verify transaction atomicity
• Verify each override type (port, admin_pubkey, relay_privkey)
• Verify cache invalidation after overrides
• Verify no-op when no overrides present

---

Step 1.3: Implement validate_config_table_completeness()

Location: src/config.c

Purpose: Validate config table has all required keys, populate missing ones

Function Signature:

int validate_config_table_completeness(void);

Implementation Details:

int validate_config_table_completeness(void) {
    if (!g_database) {
        DEBUG_ERROR("Database not initialized");
        return -1;
    }

    DEBUG_INFO("Validating config table completeness");

    // Check each default config key
    for (size_t i = 0; i < sizeof(DEFAULT_CONFIG_VALUES) / sizeof(DEFAULT_CONFIG_VALUES[0]); i++) {
        const char* key = DEFAULT_CONFIG_VALUES[i].key;
        
        // Check if key exists
        sqlite3_stmt* stmt = NULL;
        const char* sql = "SELECT COUNT(*) FROM config WHERE key = ?";
        int rc = sqlite3_prepare_v2(g_database, sql, -1, &stmt, NULL);
        if (rc != SQLITE_OK) {
            DEBUG_ERROR("Failed to prepare statement: %s", sqlite3_errmsg(g_database));
            return -1;
        }

        sqlite3_bind_text(stmt, 1, key, -1, SQLITE_STATIC);
        rc = sqlite3_step(stmt);
        
        int count = 0;
        if (rc == SQLITE_ROW) {
            count = sqlite3_column_int(stmt, 0);
        }
        sqlite3_finalize(stmt);

        // If key missing, populate it
        if (count == 0) {
            DEBUG_WARN("Config key '%s' missing, populating with default", key);
            rc = populate_missing_config_key(key, DEFAULT_CONFIG_VALUES[i].value);
            if (rc != 0) {
                DEBUG_ERROR("Failed to populate missing key '%s'", key);
                return -1;
            }
        }
    }

    DEBUG_INFO("Config table validation complete");
    return 0;
}

Helper Function:

static int populate_missing_config_key(const char* key, const char* value) {
    sqlite3_stmt* stmt = NULL;
    const char* sql = "INSERT INTO config (key, value) VALUES (?, ?)";
    
    int rc = sqlite3_prepare_v2(g_database, sql, -1, &stmt, NULL);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to prepare statement: %s", sqlite3_errmsg(g_database));
        return -1;
    }

    sqlite3_bind_text(stmt, 1, key, -1, SQLITE_STATIC);
    sqlite3_bind_text(stmt, 2, value, -1, SQLITE_STATIC);
    
    rc = sqlite3_step(stmt);
    sqlite3_finalize(stmt);

    if (rc != SQLITE_DONE) {
        DEBUG_ERROR("Failed to insert config key '%s': %s", key, sqlite3_errmsg(g_database));
        return -1;
    }

    return 0;
}

Testing:

• Verify detection of missing keys
• Verify population of missing keys with defaults
• Verify no changes when all keys present
• Verify error handling

---

Step 1.4: Implement has_cli_overrides()

Location: src/config.c

Purpose: Check if any CLI overrides are present

Function Signature:

bool has_cli_overrides(const cli_options_t* cli_options);

Implementation Details:

bool has_cli_overrides(const cli_options_t* cli_options) {
    if (!cli_options) {
        return false;
    }

    return (cli_options->port_override > 0 ||
            cli_options->admin_pubkey_override[0] != '\0' ||
            cli_options->relay_privkey_override[0] != '\0');
}

Testing:

• Verify returns true when any override present
• Verify returns false when no overrides
• Verify NULL safety

---

Phase 2: Update Function Declarations in config.h

Step 2.1: Add New Function Declarations

Location: src/config.h

Changes:

// Add after existing function declarations

// Atomic config population for first-time startup
int populate_all_config_values_atomic(const cli_options_t* cli_options);

// Atomic CLI override application
int apply_cli_overrides_atomic(const cli_options_t* cli_options);

// Config validation for existing databases
int validate_config_table_completeness(void);

// Helper function to check for CLI overrides
bool has_cli_overrides(const cli_options_t* cli_options);

---

Phase 3: Refactor Startup Flow in main.c

Step 3.1: Update First-Time Startup Branch

Location: src/main.c (around lines 1624-1740)

Current Code:

if (is_first_time_startup()) {
    first_time_startup_sequence(&cli_options);
    init_database(g_database_path);
    
    // Current incremental approach
    populate_default_config_values();
    if (cli_options.port_override > 0) {
        char port_str[16];
        snprintf(port_str, sizeof(port_str), "%d", cli_options.port_override);
        update_config_in_table("relay_port", port_str);
    }
    add_pubkeys_to_config_table();
    
    store_relay_private_key(relay_privkey);
    refresh_unified_cache_from_table();
}

New Code:

if (is_first_time_startup()) {
    // 1. Generate keys and set database path
    first_time_startup_sequence(&cli_options);
    
    // 2. Create database with schema
    init_database(g_database_path);
    
    // 3. Populate ALL config values atomically (defaults + pubkeys)
    if (populate_all_config_values_atomic(&cli_options) != 0) {
        DEBUG_ERROR("Failed to populate config values");
        return EXIT_FAILURE;
    }
    
    // 4. Apply CLI overrides atomically (separate transaction)
    if (apply_cli_overrides_atomic(&cli_options) != 0) {
        DEBUG_ERROR("Failed to apply CLI overrides");
        return EXIT_FAILURE;
    }
    
    // 5. Store relay private key securely
    store_relay_private_key(relay_privkey);
    
    // 6. Load complete config into cache
    refresh_unified_cache_from_table();
}

Testing:

• Verify first-time startup creates complete config
• Verify CLI overrides applied correctly
• Verify cache loads complete config
• Verify error handling at each step

---

Step 3.2: Update Existing Relay Startup Branch

Location: src/main.c (around lines 1741-1928)

Current Code:

else {
    char** existing_files = find_existing_db_files();
    char* relay_pubkey = extract_pubkey_from_filename(existing_files[0]);
    startup_existing_relay(relay_pubkey);
    
    init_database(g_database_path);
    
    // Current approach - unclear when overrides applied
    populate_default_config_values();
    if (cli_options.port_override > 0) {
        // ... override logic ...
    }
    
    refresh_unified_cache_from_table();
}

New Code:

else {
    // 1. Discover existing database
    char** existing_files = find_existing_db_files();
    if (!existing_files || !existing_files[0]) {
        DEBUG_ERROR("No existing database files found");
        return EXIT_FAILURE;
    }
    
    char* relay_pubkey = extract_pubkey_from_filename(existing_files[0]);
    startup_existing_relay(relay_pubkey);
    
    // 2. Open existing database
    init_database(g_database_path);
    
    // 3. Validate config table completeness (populate missing keys)
    if (validate_config_table_completeness() != 0) {
        DEBUG_ERROR("Failed to validate config table");
        return EXIT_FAILURE;
    }
    
    // 4. Apply CLI overrides if present (separate transaction)
    if (has_cli_overrides(&cli_options)) {
        if (apply_cli_overrides_atomic(&cli_options) != 0) {
            DEBUG_ERROR("Failed to apply CLI overrides");
            return EXIT_FAILURE;
        }
    }
    
    // 5. Load complete config into cache
    refresh_unified_cache_from_table();
}

Testing:

• Verify existing relay startup with complete config
• Verify missing keys populated
• Verify CLI overrides applied when present
• Verify no changes when no overrides
• Verify cache loads correctly

---

Phase 4: Deprecate Old Functions

Step 4.1: Mark Functions as Deprecated

Location: src/config.c

Functions to Deprecate:

1. populate_default_config_values() - replaced by populate_all_config_values_atomic()
2. add_pubkeys_to_config_table() - logic moved to populate_all_config_values_atomic()

Changes:

// Mark as deprecated in comments
// DEPRECATED: Use populate_all_config_values_atomic() instead
// This function will be removed in a future version
int populate_default_config_values(void) {
    // ... existing implementation ...
}

// DEPRECATED: Use populate_all_config_values_atomic() instead
// This function will be removed in a future version
int add_pubkeys_to_config_table(void) {
    // ... existing implementation ...
}

---

Phase 5: Testing Strategy

Unit Tests

1. Test populate_all_config_values_atomic()

   • Test with valid cli_options
   • Test transaction rollback on error
   • Test all config keys inserted
   • Test pubkeys inserted correctly

2. Test apply_cli_overrides_atomic()

   • Test port override
   • Test admin_pubkey override
   • Test relay_privkey override
   • Test multiple overrides
   • Test no overrides
   • Test transaction rollback on error

3. Test validate_config_table_completeness()

   • Test with complete config
   • Test with missing keys
   • Test population of missing keys

4. Test has_cli_overrides()

   • Test with each override type
   • Test with no overrides
   • Test with NULL cli_options

Integration Tests

1. First-Time Startup

   # Clean environment
   rm -f *.db
   
   # Start relay with defaults
   ./build/c_relay_x86
   
   # Verify config table complete
   sqlite3 <relay_pubkey>.db "SELECT COUNT(*) FROM config;"
   # Expected: 20+ rows (all defaults + pubkeys)
   
   # Verify cache loaded
   # Check relay.log for cache refresh message
   

2. First-Time Startup with CLI Overrides

   # Clean environment
   rm -f *.db
   
   # Start relay with port override
   ./build/c_relay_x86 --port 9999
   
   # Verify port override applied
   sqlite3 <relay_pubkey>.db "SELECT value FROM config WHERE key='relay_port';"
   # Expected: 9999
   

3. Restart with Existing Database

   # Start relay (creates database)
   ./build/c_relay_x86
   
   # Stop relay
   pkill -f c_relay_
   
   # Restart relay
   ./build/c_relay_x86
   
   # Verify config unchanged
   # Check relay.log for validation message
   

4. Restart with CLI Overrides

   # Start relay (creates database)
   ./build/c_relay_x86
   
   # Stop relay
   pkill -f c_relay_
   
   # Restart with port override
   ./build/c_relay_x86 --port 9999
   
   # Verify port override applied
   sqlite3 <relay_pubkey>.db "SELECT value FROM config WHERE key='relay_port';"
   # Expected: 9999
   

Regression Tests

Run existing test suite to ensure no breakage:

./tests/run_all_tests.sh

---

Phase 6: Documentation Updates

Files to Update

1. docs/configuration_guide.md

   • Update startup sequence description
   • Document new atomic config creation
   • Document CLI override behavior

2. docs/startup_flows_complete.md

   • Update with new flow diagrams
   • Document new function calls

3. README.md

   • Update CLI options documentation
   • Document override behavior

---

Implementation Timeline

Week 1: Core Functions

• Day 1-2: Implement populate_all_config_values_atomic()
• Day 3-4: Implement apply_cli_overrides_atomic()
• Day 5: Implement validate_config_table_completeness() and has_cli_overrides()

Week 2: Integration

• Day 1-2: Update main.c startup flow
• Day 3-4: Testing and bug fixes
• Day 5: Documentation updates

Week 3: Cleanup

• Day 1-2: Deprecate old functions
• Day 3-4: Final testing and validation
• Day 5: Code review and merge

---

Risk Mitigation

Potential Issues

1. Database Lock Contention

   • Risk: Multiple transactions could cause locks
   • Mitigation: Use BEGIN IMMEDIATE for write transactions

2. Cache Invalidation Timing

   • Risk: Cache could be read before overrides applied
   • Mitigation: Invalidate cache immediately after overrides

3. Backward Compatibility

   • Risk: Existing databases might have incomplete config
   • Mitigation: validate_config_table_completeness() handles this

4. Transaction Rollback

   • Risk: Partial config on error
   • Mitigation: All operations in transactions with proper rollback

---

Success Criteria

1. ✅ All config values created atomically in first-time startup
2. ✅ CLI overrides applied in separate atomic transaction
3. ✅ Existing databases validated and missing keys populated
4. ✅ Cache only loaded after complete config exists
5. ✅ All existing tests pass
6. ✅ No race conditions in config creation
7. ✅ Clear separation between config creation and override phases

---

Rollback Plan

If issues arise during implementation:

1. Revert main.c changes - restore original startup flow
2. Keep new functions - they can coexist with old code
3. Add feature flag - allow toggling between old and new behavior
4. Gradual migration - enable new behavior per scenario

// Feature flag approach
#define USE_ATOMIC_CONFIG_CREATION 1

#if USE_ATOMIC_CONFIG_CREATION
    // New atomic approach
    populate_all_config_values_atomic(&cli_options);
    apply_cli_overrides_atomic(&cli_options);
#else
    // Old incremental approach
    populate_default_config_values();
    // ... existing code ...
#endif