ginxsom/IMPLEMENTATION.md

Ginxsom Blossom Server Implementation Checklist

This document outlines the implementation plan for ginxsom, a FastCGI-based Blossom server designed to work with nginx for optimal performance.

Architecture Overview

• nginx: Handles static file serving (GET /) for maximum performance
• FastCGI Application: Handles authenticated operations, metadata queries, uploads
• SQLite Database: Stores blob metadata and server configuration
• File Storage: Flat directory structure initially, hierarchical optimization later

---

Phase 1: Basic File Serving & Retrieval (BUD-01)

1.1 Infrastructure Setup

• Create basic directory structure
  • Create blobs/ directory for file storage
  • Create db/ directory for SQLite database
  • Create logs/ directory for application logs
  • Set up proper permissions (nginx readable, app writable)

1.2 Database Schema

• Design SQLite schema for blob metadata
  • blobs table: sha256, size, type, uploaded_at, uploader_pubkey, filename
  • server_config table: key-value pairs for server settings
  • Create database initialization script
  • Add proper indexes on sha256 hash

1.3 nginx Configuration

• Configure nginx for static file serving
  • Set up location block for GET /<sha256> pattern with extension support
  • Configure try_files directive for multiple extension fallbacks
  • Configure proper MIME type detection
  • Add proper headers (Cache-Control, ETag, etc.)
  • Handle 404s gracefully when blob doesn't exist
  • Configure FastCGI pass-through for HEAD and non-GET requests

Future Enhancement Note: Consider implementing nginx Lua extension for true Blossom compliance with dynamic file discovery. The current approach uses explicit extension lists in try_files, which works well for common extensions but may not serve files with unusual extensions. Lua module would allow runtime directory scanning for hash-matching files regardless of extension.

1.4 Basic HEAD Endpoint

• Implement FastCGI handler for HEAD /<sha256>
  • Query database for blob metadata (single source of truth)
  • Extract SHA-256 from URL (strip extensions)
  • Return proper headers (Content-Type, Content-Length, etc.)
  • Return 404 if blob doesn't exist in database
  • Add server timing headers for debugging

1.5 Testing & Validation

• Create test blobs with known SHA-256 hashes
• Verify nginx serves files correctly with extension support
• Verify HEAD requests return proper metadata
• Test with missing files (404 responses)
• Test HEAD requests with and without extensions
• Performance test with large files

---

Phase 2: Upload & Authentication (BUD-02)

2.1 Nostr Authentication Setup

• Integrate nostr_core_lib submodule

  • Update Makefile to include nostr_core_lib paths and static library
  • Build libnostr_core_x64.a using provided build.sh script
  • Add system dependencies: -lsecp256k1 -lssl -lcrypto -lcurl -lz -ldl -lpthread -lm

• Implement authentication functions in main.c (BUD-02 section):

  • parse_authorization_header() - Extract JSON from "Nostr base64(event)" header
  • validate_blossom_event() - Validate Blossom-specific requirements (kind 24242, content hash, method, expiration)
  • authenticate_request() - Main orchestrator function

• Leverage existing nostr_core_lib functions:

  • Use nostr_validate_event() for structure + signature validation (from nip001.h)
  • Use standardized error codes from nostr_common.h (NOSTR_SUCCESS, NOSTR_ERROR_EVENT_INVALID_SIGNATURE, etc.)
  • Use nostr_strerror() for error message translation

2.2 Upload Endpoint Implementation

• Implement PUT /upload endpoint
  • Parse Authorization header (Nostr base64 event extraction)
  • Stream file upload to temporary location
  • Calculate SHA-256 hash during upload
  • Validate hash matches authorization if provided
  • Move file to permanent location
  • Store metadata in database (including uploader_pubkey and filename)
  • Return blob descriptor JSON response

2.3 Blob Descriptor Response

• Implement blob descriptor structure
  • Required fields: url, sha256, size, type, uploaded
  • Handle MIME type detection
  • Generate proper blob URLs
  • Add optional server-specific fields (uploader_pubkey, filename)

2.4 Error Handling

• Implement proper HTTP status codes
  • 400 Bad Request for invalid data
  • 401 Unauthorized for auth failures
  • 409 Conflict for hash mismatches
  • 413 Payload Too Large for size limits
  • 500 Internal Server Error for system issues
• Add detailed error messages
• Implement request logging

2.5 List Blobs Endpoint

• Implement GET /list/<pubkey> endpoint
  • Extract pubkey from URL path
  • Query database for blobs uploaded by specified pubkey
  • Support since and until query parameters for date filtering
  • Return JSON array of blob descriptors
  • Handle empty results gracefully
  • Implement optional authorization with kind 24242 event validation
    • Validate t tag is set to "list"
    • Check authorization expiration
    • Verify event signature and structure

2.6 Delete Blob Endpoint

• Implement DELETE /<sha256> endpoint
  • Extract SHA-256 hash from URL path
  • Require authorization with kind 24242 event validation
    • Validate t tag is set to "delete"
    • Verify at least one x tag matches the requested hash
    • Check authorization expiration
    • Verify event signature and structure
  • Check blob exists in database
  • Verify uploader_pubkey matches authorized pubkey (ownership check)
  • Remove blob file from filesystem
  • Remove blob metadata from database
  • Handle file deletion errors gracefully
  • Return appropriate success/error responses

2.7 Testing & Validation

• Test uploads without authentication
• Test uploads with valid nostr auth ✅ WORKING (HTTP 200 success)
• Test uploads with invalid auth ✅ WORKING (proper error responses with specific error types)
• Test hash mismatch scenarios ✅ WORKING (409 Conflict responses)
• Test file size limits
• Verify blob descriptors are correct
• Verify database metadata storage (uploader_pubkey and filename)

---

Phase 3: Upload Requirements (BUD-06)

3.1 Upload Policy Configuration

• Add server configuration options
  • Maximum file size limits
  • Allowed MIME types
  • Authentication requirements
  • Rate limiting settings
  • Storage quota limits
  • Hash-based banning/filtering

3.2 HEAD /upload Endpoint Implementation

• Implement HEAD /upload endpoint for pre-flight upload validation
  • Parse client headers:
    • X-SHA-256: blob's SHA-256 hash
    • X-Content-Length: blob size in bytes
    • X-Content-Type: blob's MIME type
  • Handle optional Authorization header (same as PUT /upload)
  • Perform validation checks without file transfer:
    • Validate SHA-256 format
    • Check file size against limits
    • Validate MIME type restrictions
    • Check authentication if required
    • Check if hash already exists (duplicate detection)
    • Check if hash is banned
  • Return appropriate HTTP status codes:
    • 200 OK - upload can proceed
    • 400 Bad Request - invalid headers
    • 401 Unauthorized - auth required
    • 403 Forbidden - not permitted (banned hash, etc.)
    • 411 Length Required - missing content length
    • 413 Content Too Large - file too large
    • 415 Unsupported Media Type - invalid MIME type
  • Add X-Reason header with human-readable error messages

3.3 Upload Pre-validation Logic

• Create validation functions that can be shared between HEAD and PUT endpoints
  • validate_upload_headers() - check required headers present and valid
  • check_file_size_limits() - enforce maximum size restrictions
  • check_mime_type_allowed() - validate against allowed types list
  • check_hash_restrictions() - check banned hashes, duplicates
  • check_upload_permissions() - user-specific upload rights

3.4 DOS Protection Benefits

• Implement early rejection before file transfer:
  • Authentication happens before any file data sent
  • Size validation prevents large file uploads that would be rejected
  • MIME type checking prevents unwanted file types
  • Hash checking prevents duplicate uploads
• Update PUT /upload to use same validation functions for consistency

3.5 Client Integration Support

• Update nginx configuration to properly handle HEAD requests to /upload
• Ensure FastCGI handles HEAD method for /upload endpoint
• Add CORS headers for preflight requests

3.6 Testing & Validation

• Test HEAD /upload with valid headers
• Test various error scenarios (missing headers, invalid formats)
• Test authorization requirements
• Test policy enforcement (size limits, MIME types, banned hashes)
• Verify error responses match BUD-06 specification
• Test client workflow: HEAD check → PUT upload
• Verify DOS protection effectiveness

---

Phase 4: Advanced Authentication & Administration System

4.1 Flexible Authentication Rules System

4.1.1 Database Schema Extension

• Create authentication rules tables
  • auth_rules table: rule_type, rule_target, operation, rule_value, enabled, expires_at
  • auth_cache table: performance caching for rule evaluation results
  • Add indexes on rule_type, rule_target, operation for performance

4.1.2 Authentication Rule Types Implementation

• Basic rule types:
  • pubkey_whitelist: Only specific pubkeys allowed
  • pubkey_blacklist: Specific pubkeys banned
  • hash_blacklist: Specific file hashes cannot be uploaded
  • mime_type_whitelist: Only specific content types allowed
  • mime_type_blacklist: Specific content types banned
• Advanced rule types:
  • rate_limit: Limit operations per pubkey/IP per time period
  • size_limit: Per-pubkey or global size limits
  • conditional: Complex JSON-based rules (time-based, size-based, etc.)

4.1.3 Rule Evaluation Engine

• Core authentication functions:
  • evaluate_auth_rules(): Main rule evaluation with caching
  • check_rule_cache(): Performance optimization layer
  • process_rule_priority(): Handle rule precedence and conflicts
  • update_auth_cache(): Store evaluation results for reuse
• Integration points:
  • Extend handle_upload_request() with rule evaluation
  • Extend handle_delete_request() with rule evaluation
  • Extend handle_list_request() with rule evaluation (optional)

4.1.4 Rule Management Interface

• SQL-based rule management:
  • add_auth_rule(): Add new authentication rules
  • remove_auth_rule(): Remove rules by ID
  • list_auth_rules(): Query existing rules with filters
  • update_auth_rule(): Modify existing rule parameters

4.2 Nostr-Native Administrative Interface

4.2.1 Server Identity Management

• Server keypair generation and storage:
  • Generate server public/private keypair on first run
  • Store server pubkey in server_config table
  • Secure private key storage (encrypted file or environment)
  • Key rotation capabilities for security

4.2.2 Administrator Management System

• Administrator database schema:
  • administrators table: pubkey, permissions, added_by, expires_at
  • Permission levels: rules, config, users, stats, * (full access)
  • Initial admin setup during server deployment
• Administrative functions:
  • check_admin_permissions(): Verify admin authorization
  • add_administrator(): Grant admin privileges
  • remove_administrator(): Revoke admin privileges
  • list_administrators(): Query admin list with permissions

4.2.3 Administrative Event Types

• Event kind definitions:
  • Kind 30242: Administrative commands (rule_add, rule_remove, config_set, etc.)
  • Kind 30243: Administrative queries (stats_get, rule_list, audit_log, etc.)
  • Kind 30244: Administrative responses (command results, query data)
• Command implementations:
  • Rule management: rule_add, rule_remove, rule_update, rule_list
  • System management: config_set, config_get, admin_add, admin_remove
  • Query operations: stats_get, blob_list, audit_log, storage_cleanup

4.2.4 Administrative Event Processing

• HTTP administrative endpoint:
  • POST /admin with nostr event authorization
  • JSON command interface with parameter validation
  • Synchronous response with operation results
• Direct nostr relay integration (future enhancement):
  • Subscribe to administrative events on configured relays
  • Real-time event processing and response
  • Publish response events back to relays

4.2.5 Administrative Audit Trail

• Administrative logging system:
  • admin_log table: track all administrative actions
  • Event ID references for nostr event traceability
  • Success/failure tracking with detailed error messages
  • Audit query capabilities for compliance

4.2.6 Security & Permission Framework

• Multi-level permission system:
  • Granular permissions: rules, config, users, stats
  • Permission inheritance and delegation
  • Time-limited administrative access (expires_at)
• Authentication security:
  • Strong nostr signature validation
  • Administrator authorization chain verification
  • Command-specific permission checks
  • Rate limiting for administrative operations

4.3 Integration & Testing

• Authentication system integration:
  • Integrate rule evaluation into existing authentication flow
  • Maintain backward compatibility with nostr-only authentication
  • Performance testing with rule caching
• Administrative system testing:
  • Test all administrative commands and queries
  • Verify permission enforcement and security
  • Test audit logging and compliance features
  • Load testing for administrative operations

---

Phase 5: Optional Features

---

Phase 5: Optional Features

4.1 User Server Lists (BUD-03) - Optional

• Implement server list advertisement
• Handle kind:10063 events
• Create server discovery endpoint
• Test client fallback scenarios

4.2 Blob Mirroring (BUD-04) - Optional

• Implement PUT /mirror endpoint
• Add URL downloading capability
• Implement hash verification
• Handle authorization for mirroring
• Test inter-server mirroring

4.3 Media Optimization (BUD-05) - Optional

• Implement PUT /media endpoint
• Add media processing libraries
• Implement optimization algorithms
• Handle various media formats
• Test optimization pipeline

4.4 Payment Integration (BUD-07) - Optional

• Implement 402 Payment Required responses
• Add Lightning payment support
• Add Cashu payment support
• Implement payment verification
• Test payment flows

4.5 NIP-94 Metadata (BUD-08) - Optional

• Add NIP-94 tag generation
• Extend blob descriptor responses
• Generate magnet links if supported
• Test metadata compatibility

4.6 Blob Reporting (BUD-09) - Optional

• Implement PUT /report endpoint
• Handle NIP-56 report events
• Add moderation interface
• Implement content filtering
• Test reporting workflow

---

Development Milestones

Milestone 1: Basic Functionality (Phase 1 Complete)

• nginx serves files by hash with extension support
• HEAD requests return metadata from database
• Database stores blob information with proper schema

Milestone 2: Full Upload Support (Phase 2 Pending)

• Basic upload functionality working (PUT requests accepted)
• SHA-256 hash calculation during upload
• File storage to blobs/ directory
• Blob descriptor JSON response
• Authenticated uploads working (Nostr kind 24242 event validation)
• Proper error handling for upload scenarios
• Database metadata storage during upload (with uploader_pubkey and filename)
• List blobs endpoint implemented (GET /list/)
• Delete blob endpoint implemented (DELETE /)

Milestone 3: Policy Compliance (Phase 3 Pending)

• Upload requirements implemented
• Server policies configurable
• Spec compliance verified

Milestone 4: Advanced Authentication (Phase 4 Complete)

• Flexible authentication rules system operational
• Nostr-native administrative interface implemented
• Rule evaluation engine with caching performance
• Administrative audit trail and compliance features

Milestone 5: Production Ready (Phase 5 Complete)

• Optional features implemented as needed
• Performance optimized
• Security hardened
• Documentation complete

---

Testing Strategy

Unit Tests

• Authentication validation functions
• SHA-256 hash calculation
• Database operations
• Configuration parsing

Integration Tests

• nginx + FastCGI integration
• End-to-end upload/download flows
• Error scenario handling
• Multi-client concurrent access

Performance Tests

• Large file uploads/downloads
• Concurrent request handling
• Database query performance
• Memory usage optimization

Compliance Tests

• Blossom protocol compliance
• Nostr event validation
• HTTP specification compliance
• Security best practices

---

Future Improvements

Upload Security & Performance Enhancements

Current Issue: The existing upload flow has a DOS vulnerability where large files are loaded entirely into memory before authentication occurs. This allows unauthenticated attackers to exhaust server memory.

Current Flow:

Client → nginx → FastCGI ginxsom
                 ├─ reads entire file into memory (malloc + fread)
                 ├─ validates auth (after file in memory)
                 └─ saves to blobs/ or errors

Proposed Solution - nginx Upload Module:

Client → nginx upload module → temp file → FastCGI ginxsom
         ├─ saves to /tmp/uploads/           ├─ validates auth quickly
         └─ passes metadata only            ├─ moves file to blobs/ 
                                            └─ or deletes temp file

Benefits:

• Eliminates DOS vulnerability - nginx handles large files efficiently
• Fast auth validation - no waiting for full upload
• Leverages nginx strengths - what it's designed for
• Better scalability - memory usage independent of file size

Implementation Requirements:

• nginx upload module configuration
• Temp file cleanup handling
• Modified FastCGI code to process file paths instead of stdin
• Proper error handling for temp file operations

Alternative Enhancement - HTTP 100 Continue: Could propose new Blossom BUD for two-phase upload:

1. Client sends headers with Expect: 100-continue + auth event
2. Server validates early (signature, expiration, pubkey)
3. Server responds 100 Continue or 401 Unauthorized
4. Client only sends file data if authorized

Priority: Implement after core BUD compliance is complete.

---

Security Considerations

• Input validation on all endpoints
• Rate limiting to prevent abuse
• Secure file storage permissions
• Database injection prevention
• Memory safety in C implementation
• Proper error message sanitization
• Log security (no sensitive data)
• secp256k1 Context Initialization Fixed - Authentication system now fully functional
• API Refactoring Complete - Upgraded from low-level crypto headers to high-level nostr_crypto_init() API
• Enhanced Error Messages - Specific error types: event_expired, invalid_signature, invalid_pubkey, etc.
• Upload DOS vulnerability - Current implementation vulnerable to memory exhaustion attacks

---

Performance Optimizations

• nginx direct file serving (bypasses application)
• FastCGI connection pooling
• Database connection management
• Efficient hash calculation
• Memory-mapped file operations
• Hierarchical file storage (future)
• CDN integration support

---

Deployment Checklist

• nginx configuration template
• FastCGI service configuration
• Database initialization scripts
• Log rotation setup
• Monitoring and health checks
• Backup procedures
• Security hardening guide
• Documentation and examples