c-relay/docs/monitoring_simplified_plan.md

# Simplified Monitoring Implementation Plan
## Kind 34567 Event Kind Distribution Reporting

**Date:** 2025-10-16  
**Status:** Implementation Ready

---

## Overview

Simplified real-time monitoring system that:
- Reports event kind distribution (which includes total event count)
- Uses kind 34567 addressable events with `d=event_kinds`
- Controlled by two config variables
- Enabled on-demand when admin logs in
- Uses simple throttling to prevent performance impact

---

## Configuration Variables

### Database Config Table

Add two new configuration keys:

```sql
INSERT INTO config (key, value, data_type, description, category) VALUES
  ('kind_34567_reporting_enabled', 'false', 'boolean', 
   'Enable/disable kind 34567 event kind distribution reporting', 'monitoring'),
  ('kind_34567_reporting_throttling_sec', '5', 'integer', 
   'Minimum seconds between kind 34567 reports (throttling)', 'monitoring');
```

### Configuration Access

```c
// In src/monitoring.c or src/api.c
int is_monitoring_enabled(void) {
    return get_config_bool("kind_34567_reporting_enabled", 0);
}

int get_monitoring_throttle_seconds(void) {
    return get_config_int("kind_34567_reporting_throttling_sec", 5);
}
```

---

## Event Structure

### Kind 34567 Event Format

```json
{
  "id": "<event_id>",
  "pubkey": "<relay_pubkey>",
  "created_at": 1697123456,
  "kind": 34567,
  "content": "{\"data_type\":\"event_kinds\",\"timestamp\":1697123456,\"data\":{\"total_events\":125000,\"distribution\":[{\"kind\":1,\"count\":45000,\"percentage\":36.0},{\"kind\":3,\"count\":12500,\"percentage\":10.0}]}}",
  "tags": [
    ["d", "event_kinds"],
    ["relay", "<relay_pubkey>"]
  ],
  "sig": "<signature>"
}
```

### Content JSON Structure

```json
{
  "data_type": "event_kinds",
  "timestamp": 1697123456,
  "data": {
    "total_events": 125000,
    "distribution": [
      {
        "kind": 1,
        "count": 45000,
        "percentage": 36.0
      },
      {
        "kind": 3,
        "count": 12500,
        "percentage": 10.0
      }
    ]
  },
  "metadata": {
    "query_time_ms": 18
  }
}
```

---

## Implementation

### File Structure

```
src/
  monitoring.h          # New file - monitoring system header
  monitoring.c          # New file - monitoring implementation
  main.c               # Modified - add trigger hook
  config.c             # Modified - add config keys (or use migration)
```

### 1. Header File: `src/monitoring.h`

```c
#ifndef MONITORING_H
#define MONITORING_H

#include <time.h>
#include <cjson/cJSON.h>

// Initialize monitoring system
int init_monitoring_system(void);

// Cleanup monitoring system
void cleanup_monitoring_system(void);

// Called when an event is stored (from main.c)
void monitoring_on_event_stored(void);

// Enable/disable monitoring (called from admin API)
int set_monitoring_enabled(int enabled);

// Get monitoring status
int is_monitoring_enabled(void);

// Get throttle interval
int get_monitoring_throttle_seconds(void);

#endif /* MONITORING_H */
```

### 2. Implementation: `src/monitoring.c`

```c
#include "monitoring.h"
#include "config.h"
#include "debug.h"
#include "../nostr_core_lib/nostr_core/nostr_core.h"
#include <sqlite3.h>
#include <string.h>
#include <time.h>

// External references
extern sqlite3* g_db;
extern int broadcast_event_to_subscriptions(cJSON* event);
extern int store_event(cJSON* event);
extern const char* get_config_value(const char* key);
extern int get_config_bool(const char* key, int default_value);
extern int get_config_int(const char* key, int default_value);
extern char* get_relay_private_key(void);

// Throttling state
static time_t last_report_time = 0;

// Initialize monitoring system
int init_monitoring_system(void) {
    DEBUG_LOG("Monitoring system initialized");
    last_report_time = 0;
    return 0;
}

// Cleanup monitoring system
void cleanup_monitoring_system(void) {
    DEBUG_LOG("Monitoring system cleaned up");
}

// Check if monitoring is enabled
int is_monitoring_enabled(void) {
    return get_config_bool("kind_34567_reporting_enabled", 0);
}

// Get throttle interval
int get_monitoring_throttle_seconds(void) {
    return get_config_int("kind_34567_reporting_throttling_sec", 5);
}

// Enable/disable monitoring
int set_monitoring_enabled(int enabled) {
    // Update config table
    const char* value = enabled ? "true" : "false";
    
    // This would call update_config_in_table() or similar
    // For now, assume we have a function to update config
    extern int update_config_in_table(const char* key, const char* value);
    return update_config_in_table("kind_34567_reporting_enabled", value);
}

// Query event kind distribution from database
static char* query_event_kind_distribution(void) {
    if (!g_db) {
        DEBUG_ERROR("Database not available for monitoring query");
        return NULL;
    }
    
    struct timespec start_time;
    clock_gettime(CLOCK_MONOTONIC, &start_time);
    
    // Query total events
    sqlite3_stmt* stmt;
    int total_events = 0;
    
    if (sqlite3_prepare_v2(g_db, "SELECT COUNT(*) FROM events", -1, &stmt, NULL) == SQLITE_OK) {
        if (sqlite3_step(stmt) == SQLITE_ROW) {
            total_events = sqlite3_column_int(stmt, 0);
        }
        sqlite3_finalize(stmt);
    }
    
    // Query kind distribution
    cJSON* response = cJSON_CreateObject();
    cJSON_AddStringToObject(response, "data_type", "event_kinds");
    cJSON_AddNumberToObject(response, "timestamp", (double)time(NULL));
    
    cJSON* data = cJSON_CreateObject();
    cJSON_AddNumberToObject(data, "total_events", total_events);
    
    cJSON* distribution = cJSON_CreateArray();
    
    const char* sql = 
        "SELECT kind, COUNT(*) as count, "
        "ROUND(COUNT(*) * 100.0 / (SELECT COUNT(*) FROM events), 2) as percentage "
        "FROM events GROUP BY kind ORDER BY count DESC";
    
    if (sqlite3_prepare_v2(g_db, sql, -1, &stmt, NULL) == SQLITE_OK) {
        while (sqlite3_step(stmt) == SQLITE_ROW) {
            cJSON* kind_obj = cJSON_CreateObject();
            cJSON_AddNumberToObject(kind_obj, "kind", sqlite3_column_int(stmt, 0));
            cJSON_AddNumberToObject(kind_obj, "count", sqlite3_column_int64(stmt, 1));
            cJSON_AddNumberToObject(kind_obj, "percentage", sqlite3_column_double(stmt, 2));
            cJSON_AddItemToArray(distribution, kind_obj);
        }
        sqlite3_finalize(stmt);
    }
    
    cJSON_AddItemToObject(data, "distribution", distribution);
    cJSON_AddItemToObject(response, "data", data);
    
    // Calculate query time
    struct timespec end_time;
    clock_gettime(CLOCK_MONOTONIC, &end_time);
    double query_time_ms = (end_time.tv_sec - start_time.tv_sec) * 1000.0 +
                          (end_time.tv_nsec - start_time.tv_nsec) / 1000000.0;
    
    cJSON* metadata = cJSON_CreateObject();
    cJSON_AddNumberToObject(metadata, "query_time_ms", query_time_ms);
    cJSON_AddItemToObject(response, "metadata", metadata);
    
    char* json_string = cJSON_Print(response);
    cJSON_Delete(response);
    
    return json_string;
}

// Generate and broadcast kind 34567 event
static int generate_monitoring_event(const char* json_content) {
    if (!json_content) return -1;
    
    // Get relay keys
    const char* relay_pubkey = get_config_value("relay_pubkey");
    char* relay_privkey_hex = get_relay_private_key();
    if (!relay_pubkey || !relay_privkey_hex) {
        if (relay_privkey_hex) free(relay_privkey_hex);
        DEBUG_ERROR("Could not get relay keys for monitoring event");
        return -1;
    }
    
    // Convert relay private key to bytes
    unsigned char relay_privkey[32];
    if (nostr_hex_to_bytes(relay_privkey_hex, relay_privkey, sizeof(relay_privkey)) != 0) {
        free(relay_privkey_hex);
        DEBUG_ERROR("Failed to convert relay private key");
        return -1;
    }
    free(relay_privkey_hex);
    
    // Create tags array
    cJSON* tags = cJSON_CreateArray();
    
    // d tag for addressable event
    cJSON* d_tag = cJSON_CreateArray();
    cJSON_AddItemToArray(d_tag, cJSON_CreateString("d"));
    cJSON_AddItemToArray(d_tag, cJSON_CreateString("event_kinds"));
    cJSON_AddItemToArray(tags, d_tag);
    
    // relay tag
    cJSON* relay_tag = cJSON_CreateArray();
    cJSON_AddItemToArray(relay_tag, cJSON_CreateString("relay"));
    cJSON_AddItemToArray(relay_tag, cJSON_CreateString(relay_pubkey));
    cJSON_AddItemToArray(tags, relay_tag);
    
    // Create and sign event
    cJSON* event = nostr_create_and_sign_event(
        34567,              // kind
        json_content,       // content
        tags,               // tags
        relay_privkey,      // private key
        time(NULL)          // timestamp
    );
    
    if (!event) {
        DEBUG_ERROR("Failed to create and sign monitoring event");
        return -1;
    }
    
    // Broadcast to subscriptions
    broadcast_event_to_subscriptions(event);
    
    // Store in database
    int result = store_event(event);
    
    cJSON_Delete(event);
    
    return result;
}

// Called when an event is stored
void monitoring_on_event_stored(void) {
    // Check if monitoring is enabled
    if (!is_monitoring_enabled()) {
        return;
    }
    
    // Check throttling
    time_t now = time(NULL);
    int throttle_seconds = get_monitoring_throttle_seconds();
    
    if (now - last_report_time < throttle_seconds) {
        return;  // Too soon, skip this update
    }
    
    // Query event kind distribution
    char* json_content = query_event_kind_distribution();
    if (!json_content) {
        DEBUG_ERROR("Failed to query event kind distribution");
        return;
    }
    
    // Generate and broadcast monitoring event
    int result = generate_monitoring_event(json_content);
    free(json_content);
    
    if (result == 0) {
        last_report_time = now;
        DEBUG_LOG("Generated kind 34567 monitoring event");
    } else {
        DEBUG_ERROR("Failed to generate monitoring event");
    }
}
```

### 3. Integration: Modify `src/main.c`

Add monitoring hook to event storage:

```c
// At top of file
#include "monitoring.h"

// In main() function, after init_database()
if (init_monitoring_system() != 0) {
    DEBUG_WARN("Failed to initialize monitoring system");
    // Continue anyway - monitoring is optional
}

// In store_event() function, after successful storage
int store_event(cJSON* event) {
    // ... existing code ...
    
    if (rc != SQLITE_DONE) {
        // ... error handling ...
    }
    
    free(tags_json);
    
    // Trigger monitoring update
    monitoring_on_event_stored();
    
    return 0;
}

// In cleanup section of main()
cleanup_monitoring_system();
```

### 4. Admin API: Enable/Disable Monitoring

Add admin command to enable monitoring (in `src/dm_admin.c` or `src/api.c`):

```c
// Handle admin command to enable monitoring
if (strcmp(command, "enable_monitoring") == 0) {
    set_monitoring_enabled(1);
    send_nip17_response(sender_pubkey, 
        "✅ Kind 34567 monitoring enabled", 
        error_msg, sizeof(error_msg));
    return 0;
}

// Handle admin command to disable monitoring
if (strcmp(command, "disable_monitoring") == 0) {
    set_monitoring_enabled(0);
    send_nip17_response(sender_pubkey, 
        "🔴 Kind 34567 monitoring disabled", 
        error_msg, sizeof(error_msg));
    return 0;
}

// Handle admin command to set throttle interval
if (strncmp(command, "set_monitoring_throttle ", 24) == 0) {
    int seconds = atoi(command + 24);
    if (seconds >= 1 && seconds <= 3600) {
        char value[16];
        snprintf(value, sizeof(value), "%d", seconds);
        update_config_in_table("kind_34567_reporting_throttling_sec", value);
        
        char response[128];
        snprintf(response, sizeof(response), 
            "✅ Monitoring throttle set to %d seconds", seconds);
        send_nip17_response(sender_pubkey, response, error_msg, sizeof(error_msg));
    }
    return 0;
}
```

---

## Frontend Integration

### Admin Dashboard Subscription

```javascript
// When admin logs in to dashboard
async function enableMonitoring() {
  // Send admin command to enable monitoring
  await sendAdminCommand(['enable_monitoring']);
  
  // Subscribe to kind 34567 events
  const subscription = {
    kinds: [34567],
    authors: [relayPubkey],
    "#d": ["event_kinds"]
  };
  
  relay.subscribe([subscription], {
    onevent: (event) => {
      handleMonitoringEvent(event);
    }
  });
}

// Handle incoming monitoring events
function handleMonitoringEvent(event) {
  const content = JSON.parse(event.content);
  
  if (content.data_type === 'event_kinds') {
    updateEventKindsChart(content.data);
    updateTotalEventsDisplay(content.data.total_events);
  }
}

// When admin logs out or closes dashboard
async function disableMonitoring() {
  await sendAdminCommand(['disable_monitoring']);
}
```

### Display Event Kind Distribution

```javascript
function updateEventKindsChart(data) {
  const { total_events, distribution } = data;
  
  // Update total events display
  document.getElementById('total-events').textContent = 
    total_events.toLocaleString();
  
  // Update chart/table with distribution
  const tableBody = document.getElementById('kind-distribution-table');
  tableBody.innerHTML = '';
  
  distribution.forEach(item => {
    const row = document.createElement('tr');
    row.innerHTML = `
      <td>Kind ${item.kind}</td>
      <td>${item.count.toLocaleString()}</td>
      <td>${item.percentage}%</td>
    `;
    tableBody.appendChild(row);
  });
}
```

---

## Configuration Migration

### Add to Schema or Migration Script

```sql
-- Add monitoring configuration
INSERT INTO config (key, value, data_type, description, category) VALUES
  ('kind_34567_reporting_enabled', 'false', 'boolean', 
   'Enable/disable kind 34567 event kind distribution reporting', 'monitoring'),
  ('kind_34567_reporting_throttling_sec', '5', 'integer', 
   'Minimum seconds between kind 34567 reports (throttling)', 'monitoring');
```

Or add to existing config initialization in `src/config.c`.

---

## Testing

### 1. Enable Monitoring

```bash
# Via admin command (NIP-17 DM)
echo '["enable_monitoring"]' | nak event --kind 14 --content - ws://localhost:8888
```

### 2. Subscribe to Monitoring Events

```bash
# Subscribe to kind 34567 events
nak req --kinds 34567 --authors <relay_pubkey> ws://localhost:8888
```

### 3. Generate Events

```bash
# Send some test events to trigger monitoring
for i in {1..10}; do
  nak event -c "Test event $i" ws://localhost:8888
  sleep 1
done
```

### 4. Verify Monitoring Events

You should see kind 34567 events every 5 seconds (or configured throttle interval) with event kind distribution.

---

## Performance Impact

### With 3 events/second (relay.damus.io scale)

**Query execution**:
- Frequency: Every 5 seconds (throttled)
- Query time: ~700ms (for 1M events)
- Overhead: 700ms / 5000ms = 14% (acceptable)

**Per-event overhead**:
- Check if enabled: < 0.01ms
- Check throttle: < 0.01ms
- Total: < 0.02ms per event (negligible)

**Overall impact**: < 1% on event processing, 14% on query thread (separate from event processing)

---

## Future Enhancements

Once this is working, easy to add:

1. **More data types**: Add `d=connections`, `d=subscriptions`, etc.
2. **Materialized counters**: Optimize queries for very large databases
3. **Historical data**: Store monitoring events for trending
4. **Alerts**: Trigger on thresholds (e.g., > 90% capacity)

---

## Summary

This simplified plan provides:

✅ **Single data type**: Event kind distribution (includes total events)  
✅ **Two config variables**: Enable/disable and throttle control  
✅ **On-demand activation**: Enabled when admin logs in  
✅ **Simple throttling**: Prevents performance impact  
✅ **Clean implementation**: ~200 lines of code  
✅ **Easy to extend**: Add more data types later  

**Estimated implementation time**: 4-6 hours

**Files to create/modify**:
- Create: `src/monitoring.h` (~30 lines)
- Create: `src/monitoring.c` (~200 lines)
- Modify: `src/main.c` (~10 lines)
- Modify: `src/config.c` or migration (~5 lines)
- Modify: `src/dm_admin.c` or `src/api.c` (~30 lines)
- Create: `api/monitoring.js` (frontend, ~100 lines)

**Total new code**: ~375 lines