# Unified Startup Implementation Plan

## Overview

This document provides a detailed implementation plan for refactoring the startup sequence to use atomic config creation followed by CLI overrides. This plan breaks down the work into discrete, testable steps.

---

## Phase 1: Create New Functions in config.c

### Step 1.1: Implement `populate_all_config_values_atomic()`

**Location**: `src/config.c`

**Purpose**: Create complete config table in single transaction for first-time startup

**Function Signature**:
```c
int populate_all_config_values_atomic(const cli_options_t* cli_options);
```

**Implementation Details**:
```c
int populate_all_config_values_atomic(const cli_options_t* cli_options) {
    if (!g_database) {
        DEBUG_ERROR("Database not initialized");
        return -1;
    }

    // Begin transaction
    char* err_msg = NULL;
    int rc = sqlite3_exec(g_database, "BEGIN TRANSACTION;", NULL, NULL, &err_msg);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to begin transaction: %s", err_msg);
        sqlite3_free(err_msg);
        return -1;
    }

    // Prepare INSERT statement
    sqlite3_stmt* stmt = NULL;
    const char* sql = "INSERT INTO config (key, value) VALUES (?, ?)";
    rc = sqlite3_prepare_v2(g_database, sql, -1, &stmt, NULL);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to prepare statement: %s", sqlite3_errmsg(g_database));
        sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
        return -1;
    }

    // Insert all default config values
    for (size_t i = 0; i < sizeof(DEFAULT_CONFIG_VALUES) / sizeof(DEFAULT_CONFIG_VALUES[0]); i++) {
        sqlite3_reset(stmt);
        sqlite3_bind_text(stmt, 1, DEFAULT_CONFIG_VALUES[i].key, -1, SQLITE_STATIC);
        sqlite3_bind_text(stmt, 2, DEFAULT_CONFIG_VALUES[i].value, -1, SQLITE_STATIC);
        
        rc = sqlite3_step(stmt);
        if (rc != SQLITE_DONE) {
            DEBUG_ERROR("Failed to insert config key '%s': %s", 
                       DEFAULT_CONFIG_VALUES[i].key, sqlite3_errmsg(g_database));
            sqlite3_finalize(stmt);
            sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
            return -1;
        }
    }

    // Insert admin_pubkey from cache
    sqlite3_reset(stmt);
    sqlite3_bind_text(stmt, 1, "admin_pubkey", -1, SQLITE_STATIC);
    sqlite3_bind_text(stmt, 2, g_unified_cache.admin_pubkey, -1, SQLITE_STATIC);
    rc = sqlite3_step(stmt);
    if (rc != SQLITE_DONE) {
        DEBUG_ERROR("Failed to insert admin_pubkey: %s", sqlite3_errmsg(g_database));
        sqlite3_finalize(stmt);
        sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
        return -1;
    }

    // Insert relay_pubkey from cache
    sqlite3_reset(stmt);
    sqlite3_bind_text(stmt, 1, "relay_pubkey", -1, SQLITE_STATIC);
    sqlite3_bind_text(stmt, 2, g_unified_cache.relay_pubkey, -1, SQLITE_STATIC);
    rc = sqlite3_step(stmt);
    if (rc != SQLITE_DONE) {
        DEBUG_ERROR("Failed to insert relay_pubkey: %s", sqlite3_errmsg(g_database));
        sqlite3_finalize(stmt);
        sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
        return -1;
    }

    sqlite3_finalize(stmt);

    // Commit transaction
    rc = sqlite3_exec(g_database, "COMMIT;", NULL, NULL, &err_msg);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to commit transaction: %s", err_msg);
        sqlite3_free(err_msg);
        sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
        return -1;
    }

    DEBUG_INFO("Successfully populated all config values atomically");
    return 0;
}
```

**Testing**:
- Verify transaction atomicity (all or nothing)
- Verify all DEFAULT_CONFIG_VALUES inserted
- Verify admin_pubkey and relay_pubkey inserted
- Verify error handling on failure

---

### Step 1.2: Implement `apply_cli_overrides_atomic()`

**Location**: `src/config.c`

**Purpose**: Apply CLI overrides to existing config table in single transaction

**Function Signature**:
```c
int apply_cli_overrides_atomic(const cli_options_t* cli_options);
```

**Implementation Details**:
```c
int apply_cli_overrides_atomic(const cli_options_t* cli_options) {
    if (!g_database) {
        DEBUG_ERROR("Database not initialized");
        return -1;
    }

    if (!cli_options) {
        DEBUG_ERROR("CLI options is NULL");
        return -1;
    }

    // Check if any overrides exist
    bool has_overrides = false;
    if (cli_options->port_override > 0) has_overrides = true;
    if (cli_options->admin_pubkey_override[0] != '\0') has_overrides = true;
    if (cli_options->relay_privkey_override[0] != '\0') has_overrides = true;

    if (!has_overrides) {
        DEBUG_INFO("No CLI overrides to apply");
        return 0;
    }

    // Begin transaction
    char* err_msg = NULL;
    int rc = sqlite3_exec(g_database, "BEGIN TRANSACTION;", NULL, NULL, &err_msg);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to begin transaction: %s", err_msg);
        sqlite3_free(err_msg);
        return -1;
    }

    // Prepare UPDATE statement
    sqlite3_stmt* stmt = NULL;
    const char* sql = "UPDATE config SET value = ? WHERE key = ?";
    rc = sqlite3_prepare_v2(g_database, sql, -1, &stmt, NULL);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to prepare statement: %s", sqlite3_errmsg(g_database));
        sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
        return -1;
    }

    // Apply port override
    if (cli_options->port_override > 0) {
        char port_str[16];
        snprintf(port_str, sizeof(port_str), "%d", cli_options->port_override);
        
        sqlite3_reset(stmt);
        sqlite3_bind_text(stmt, 1, port_str, -1, SQLITE_TRANSIENT);
        sqlite3_bind_text(stmt, 2, "relay_port", -1, SQLITE_STATIC);
        
        rc = sqlite3_step(stmt);
        if (rc != SQLITE_DONE) {
            DEBUG_ERROR("Failed to update relay_port: %s", sqlite3_errmsg(g_database));
            sqlite3_finalize(stmt);
            sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
            return -1;
        }
        DEBUG_INFO("Applied CLI override: relay_port = %s", port_str);
    }

    // Apply admin_pubkey override
    if (cli_options->admin_pubkey_override[0] != '\0') {
        sqlite3_reset(stmt);
        sqlite3_bind_text(stmt, 1, cli_options->admin_pubkey_override, -1, SQLITE_STATIC);
        sqlite3_bind_text(stmt, 2, "admin_pubkey", -1, SQLITE_STATIC);
        
        rc = sqlite3_step(stmt);
        if (rc != SQLITE_DONE) {
            DEBUG_ERROR("Failed to update admin_pubkey: %s", sqlite3_errmsg(g_database));
            sqlite3_finalize(stmt);
            sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
            return -1;
        }
        DEBUG_INFO("Applied CLI override: admin_pubkey");
    }

    // Apply relay_privkey override
    if (cli_options->relay_privkey_override[0] != '\0') {
        sqlite3_reset(stmt);
        sqlite3_bind_text(stmt, 1, cli_options->relay_privkey_override, -1, SQLITE_STATIC);
        sqlite3_bind_text(stmt, 2, "relay_privkey", -1, SQLITE_STATIC);
        
        rc = sqlite3_step(stmt);
        if (rc != SQLITE_DONE) {
            DEBUG_ERROR("Failed to update relay_privkey: %s", sqlite3_errmsg(g_database));
            sqlite3_finalize(stmt);
            sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
            return -1;
        }
        DEBUG_INFO("Applied CLI override: relay_privkey");
    }

    sqlite3_finalize(stmt);

    // Commit transaction
    rc = sqlite3_exec(g_database, "COMMIT;", NULL, NULL, &err_msg);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to commit transaction: %s", err_msg);
        sqlite3_free(err_msg);
        sqlite3_exec(g_database, "ROLLBACK;", NULL, NULL, NULL);
        return -1;
    }

    // Invalidate cache to force refresh
    invalidate_config_cache();

    DEBUG_INFO("Successfully applied CLI overrides atomically");
    return 0;
}
```

**Testing**:
- Verify transaction atomicity
- Verify each override type (port, admin_pubkey, relay_privkey)
- Verify cache invalidation after overrides
- Verify no-op when no overrides present

---

### Step 1.3: Implement `validate_config_table_completeness()`

**Location**: `src/config.c`

**Purpose**: Validate config table has all required keys, populate missing ones

**Function Signature**:
```c
int validate_config_table_completeness(void);
```

**Implementation Details**:
```c
int validate_config_table_completeness(void) {
    if (!g_database) {
        DEBUG_ERROR("Database not initialized");
        return -1;
    }

    DEBUG_INFO("Validating config table completeness");

    // Check each default config key
    for (size_t i = 0; i < sizeof(DEFAULT_CONFIG_VALUES) / sizeof(DEFAULT_CONFIG_VALUES[0]); i++) {
        const char* key = DEFAULT_CONFIG_VALUES[i].key;
        
        // Check if key exists
        sqlite3_stmt* stmt = NULL;
        const char* sql = "SELECT COUNT(*) FROM config WHERE key = ?";
        int rc = sqlite3_prepare_v2(g_database, sql, -1, &stmt, NULL);
        if (rc != SQLITE_OK) {
            DEBUG_ERROR("Failed to prepare statement: %s", sqlite3_errmsg(g_database));
            return -1;
        }

        sqlite3_bind_text(stmt, 1, key, -1, SQLITE_STATIC);
        rc = sqlite3_step(stmt);
        
        int count = 0;
        if (rc == SQLITE_ROW) {
            count = sqlite3_column_int(stmt, 0);
        }
        sqlite3_finalize(stmt);

        // If key missing, populate it
        if (count == 0) {
            DEBUG_WARN("Config key '%s' missing, populating with default", key);
            rc = populate_missing_config_key(key, DEFAULT_CONFIG_VALUES[i].value);
            if (rc != 0) {
                DEBUG_ERROR("Failed to populate missing key '%s'", key);
                return -1;
            }
        }
    }

    DEBUG_INFO("Config table validation complete");
    return 0;
}
```

**Helper Function**:
```c
static int populate_missing_config_key(const char* key, const char* value) {
    sqlite3_stmt* stmt = NULL;
    const char* sql = "INSERT INTO config (key, value) VALUES (?, ?)";
    
    int rc = sqlite3_prepare_v2(g_database, sql, -1, &stmt, NULL);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to prepare statement: %s", sqlite3_errmsg(g_database));
        return -1;
    }

    sqlite3_bind_text(stmt, 1, key, -1, SQLITE_STATIC);
    sqlite3_bind_text(stmt, 2, value, -1, SQLITE_STATIC);
    
    rc = sqlite3_step(stmt);
    sqlite3_finalize(stmt);

    if (rc != SQLITE_DONE) {
        DEBUG_ERROR("Failed to insert config key '%s': %s", key, sqlite3_errmsg(g_database));
        return -1;
    }

    return 0;
}
```

**Testing**:
- Verify detection of missing keys
- Verify population of missing keys with defaults
- Verify no changes when all keys present
- Verify error handling

---

### Step 1.4: Implement `has_cli_overrides()`

**Location**: `src/config.c`

**Purpose**: Check if any CLI overrides are present

**Function Signature**:
```c
bool has_cli_overrides(const cli_options_t* cli_options);
```

**Implementation Details**:
```c
bool has_cli_overrides(const cli_options_t* cli_options) {
    if (!cli_options) {
        return false;
    }

    return (cli_options->port_override > 0 ||
            cli_options->admin_pubkey_override[0] != '\0' ||
            cli_options->relay_privkey_override[0] != '\0');
}
```

**Testing**:
- Verify returns true when any override present
- Verify returns false when no overrides
- Verify NULL safety

---

## Phase 2: Update Function Declarations in config.h

### Step 2.1: Add New Function Declarations

**Location**: `src/config.h`

**Changes**:
```c
// Add after existing function declarations

// Atomic config population for first-time startup
int populate_all_config_values_atomic(const cli_options_t* cli_options);

// Atomic CLI override application
int apply_cli_overrides_atomic(const cli_options_t* cli_options);

// Config validation for existing databases
int validate_config_table_completeness(void);

// Helper function to check for CLI overrides
bool has_cli_overrides(const cli_options_t* cli_options);
```

---

## Phase 3: Refactor Startup Flow in main.c

### Step 3.1: Update First-Time Startup Branch

**Location**: `src/main.c` (around lines 1624-1740)

**Current Code**:
```c
if (is_first_time_startup()) {
    first_time_startup_sequence(&cli_options);
    init_database(g_database_path);
    
    // Current incremental approach
    populate_default_config_values();
    if (cli_options.port_override > 0) {
        char port_str[16];
        snprintf(port_str, sizeof(port_str), "%d", cli_options.port_override);
        update_config_in_table("relay_port", port_str);
    }
    add_pubkeys_to_config_table();
    
    store_relay_private_key(relay_privkey);
    refresh_unified_cache_from_table();
}
```

**New Code**:
```c
if (is_first_time_startup()) {
    // 1. Generate keys and set database path
    first_time_startup_sequence(&cli_options);
    
    // 2. Create database with schema
    init_database(g_database_path);
    
    // 3. Populate ALL config values atomically (defaults + pubkeys)
    if (populate_all_config_values_atomic(&cli_options) != 0) {
        DEBUG_ERROR("Failed to populate config values");
        return EXIT_FAILURE;
    }
    
    // 4. Apply CLI overrides atomically (separate transaction)
    if (apply_cli_overrides_atomic(&cli_options) != 0) {
        DEBUG_ERROR("Failed to apply CLI overrides");
        return EXIT_FAILURE;
    }
    
    // 5. Store relay private key securely
    store_relay_private_key(relay_privkey);
    
    // 6. Load complete config into cache
    refresh_unified_cache_from_table();
}
```

**Testing**:
- Verify first-time startup creates complete config
- Verify CLI overrides applied correctly
- Verify cache loads complete config
- Verify error handling at each step

---

### Step 3.2: Update Existing Relay Startup Branch

**Location**: `src/main.c` (around lines 1741-1928)

**Current Code**:
```c
else {
    char** existing_files = find_existing_db_files();
    char* relay_pubkey = extract_pubkey_from_filename(existing_files[0]);
    startup_existing_relay(relay_pubkey);
    
    init_database(g_database_path);
    
    // Current approach - unclear when overrides applied
    populate_default_config_values();
    if (cli_options.port_override > 0) {
        // ... override logic ...
    }
    
    refresh_unified_cache_from_table();
}
```

**New Code**:
```c
else {
    // 1. Discover existing database
    char** existing_files = find_existing_db_files();
    if (!existing_files || !existing_files[0]) {
        DEBUG_ERROR("No existing database files found");
        return EXIT_FAILURE;
    }
    
    char* relay_pubkey = extract_pubkey_from_filename(existing_files[0]);
    startup_existing_relay(relay_pubkey);
    
    // 2. Open existing database
    init_database(g_database_path);
    
    // 3. Validate config table completeness (populate missing keys)
    if (validate_config_table_completeness() != 0) {
        DEBUG_ERROR("Failed to validate config table");
        return EXIT_FAILURE;
    }
    
    // 4. Apply CLI overrides if present (separate transaction)
    if (has_cli_overrides(&cli_options)) {
        if (apply_cli_overrides_atomic(&cli_options) != 0) {
            DEBUG_ERROR("Failed to apply CLI overrides");
            return EXIT_FAILURE;
        }
    }
    
    // 5. Load complete config into cache
    refresh_unified_cache_from_table();
}
```

**Testing**:
- Verify existing relay startup with complete config
- Verify missing keys populated
- Verify CLI overrides applied when present
- Verify no changes when no overrides
- Verify cache loads correctly

---

## Phase 4: Deprecate Old Functions

### Step 4.1: Mark Functions as Deprecated

**Location**: `src/config.c`

**Functions to Deprecate**:
1. `populate_default_config_values()` - replaced by `populate_all_config_values_atomic()`
2. `add_pubkeys_to_config_table()` - logic moved to `populate_all_config_values_atomic()`

**Changes**:
```c
// Mark as deprecated in comments
// DEPRECATED: Use populate_all_config_values_atomic() instead
// This function will be removed in a future version
int populate_default_config_values(void) {
    // ... existing implementation ...
}

// DEPRECATED: Use populate_all_config_values_atomic() instead
// This function will be removed in a future version
int add_pubkeys_to_config_table(void) {
    // ... existing implementation ...
}
```

---

## Phase 5: Testing Strategy

### Unit Tests

1. **Test `populate_all_config_values_atomic()`**
   - Test with valid cli_options
   - Test transaction rollback on error
   - Test all config keys inserted
   - Test pubkeys inserted correctly

2. **Test `apply_cli_overrides_atomic()`**
   - Test port override
   - Test admin_pubkey override
   - Test relay_privkey override
   - Test multiple overrides
   - Test no overrides
   - Test transaction rollback on error

3. **Test `validate_config_table_completeness()`**
   - Test with complete config
   - Test with missing keys
   - Test population of missing keys

4. **Test `has_cli_overrides()`**
   - Test with each override type
   - Test with no overrides
   - Test with NULL cli_options

### Integration Tests

1. **First-Time Startup**
   ```bash
   # Clean environment
   rm -f *.db
   
   # Start relay with defaults
   ./build/c_relay_x86
   
   # Verify config table complete
   sqlite3 <relay_pubkey>.db "SELECT COUNT(*) FROM config;"
   # Expected: 20+ rows (all defaults + pubkeys)
   
   # Verify cache loaded
   # Check relay.log for cache refresh message
   ```

2. **First-Time Startup with CLI Overrides**
   ```bash
   # Clean environment
   rm -f *.db
   
   # Start relay with port override
   ./build/c_relay_x86 --port 9999
   
   # Verify port override applied
   sqlite3 <relay_pubkey>.db "SELECT value FROM config WHERE key='relay_port';"
   # Expected: 9999
   ```

3. **Restart with Existing Database**
   ```bash
   # Start relay (creates database)
   ./build/c_relay_x86
   
   # Stop relay
   pkill -f c_relay_
   
   # Restart relay
   ./build/c_relay_x86
   
   # Verify config unchanged
   # Check relay.log for validation message
   ```

4. **Restart with CLI Overrides**
   ```bash
   # Start relay (creates database)
   ./build/c_relay_x86
   
   # Stop relay
   pkill -f c_relay_
   
   # Restart with port override
   ./build/c_relay_x86 --port 9999
   
   # Verify port override applied
   sqlite3 <relay_pubkey>.db "SELECT value FROM config WHERE key='relay_port';"
   # Expected: 9999
   ```

### Regression Tests

Run existing test suite to ensure no breakage:
```bash
./tests/run_all_tests.sh
```

---

## Phase 6: Documentation Updates

### Files to Update

1. **docs/configuration_guide.md**
   - Update startup sequence description
   - Document new atomic config creation
   - Document CLI override behavior

2. **docs/startup_flows_complete.md**
   - Update with new flow diagrams
   - Document new function calls

3. **README.md**
   - Update CLI options documentation
   - Document override behavior

---

## Implementation Timeline

### Week 1: Core Functions
- Day 1-2: Implement `populate_all_config_values_atomic()`
- Day 3-4: Implement `apply_cli_overrides_atomic()`
- Day 5: Implement `validate_config_table_completeness()` and `has_cli_overrides()`

### Week 2: Integration
- Day 1-2: Update main.c startup flow
- Day 3-4: Testing and bug fixes
- Day 5: Documentation updates

### Week 3: Cleanup
- Day 1-2: Deprecate old functions
- Day 3-4: Final testing and validation
- Day 5: Code review and merge

---

## Risk Mitigation

### Potential Issues

1. **Database Lock Contention**
   - Risk: Multiple transactions could cause locks
   - Mitigation: Use BEGIN IMMEDIATE for write transactions

2. **Cache Invalidation Timing**
   - Risk: Cache could be read before overrides applied
   - Mitigation: Invalidate cache immediately after overrides

3. **Backward Compatibility**
   - Risk: Existing databases might have incomplete config
   - Mitigation: `validate_config_table_completeness()` handles this

4. **Transaction Rollback**
   - Risk: Partial config on error
   - Mitigation: All operations in transactions with proper rollback

---

## Success Criteria

1. ✅ All config values created atomically in first-time startup
2. ✅ CLI overrides applied in separate atomic transaction
3. ✅ Existing databases validated and missing keys populated
4. ✅ Cache only loaded after complete config exists
5. ✅ All existing tests pass
6. ✅ No race conditions in config creation
7. ✅ Clear separation between config creation and override phases

---

## Rollback Plan

If issues arise during implementation:

1. **Revert main.c changes** - restore original startup flow
2. **Keep new functions** - they can coexist with old code
3. **Add feature flag** - allow toggling between old and new behavior
4. **Gradual migration** - enable new behavior per scenario

```c
// Feature flag approach
#define USE_ATOMIC_CONFIG_CREATION 1

#if USE_ATOMIC_CONFIG_CREATION
    // New atomic approach
    populate_all_config_values_atomic(&cli_options);
    apply_cli_overrides_atomic(&cli_options);
#else
    // Old incremental approach
    populate_default_config_values();
    // ... existing code ...
#endif
```