Files
ginxsom/IMPLEMENTATION.md
2025-08-20 06:20:32 -04:00

512 lines
20 KiB
Markdown

# Ginxsom Blossom Server Implementation Checklist
This document outlines the implementation plan for ginxsom, a FastCGI-based Blossom server designed to work with nginx for optimal performance.
## Architecture Overview
- **nginx**: Handles static file serving (GET /<sha256>) for maximum performance
- **FastCGI Application**: Handles authenticated operations, metadata queries, uploads
- **SQLite Database**: Stores blob metadata and server configuration
- **File Storage**: Flat directory structure initially, hierarchical optimization later
---
## Phase 1: Basic File Serving & Retrieval (BUD-01)
### 1.1 Infrastructure Setup
- [x] Create basic directory structure
- [x] Create `blobs/` directory for file storage
- [x] Create `db/` directory for SQLite database
- [x] Create `logs/` directory for application logs
- [x] Set up proper permissions (nginx readable, app writable)
### 1.2 Database Schema
- [x] Design SQLite schema for blob metadata
- [x] `blobs` table: sha256, size, type, uploaded_at, uploader_pubkey, filename
- [x] `server_config` table: key-value pairs for server settings
- [x] Create database initialization script
- [x] Add proper indexes on sha256 hash
### 1.3 nginx Configuration
- [x] Configure nginx for static file serving
- [x] Set up location block for `GET /<sha256>` pattern with extension support
- [x] Configure try_files directive for multiple extension fallbacks
- [x] Configure proper MIME type detection
- [x] Add proper headers (Cache-Control, ETag, etc.)
- [x] Handle 404s gracefully when blob doesn't exist
- [x] Configure FastCGI pass-through for HEAD and non-GET requests
**Future Enhancement Note**: Consider implementing nginx Lua extension for true Blossom compliance with dynamic file discovery. The current approach uses explicit extension lists in `try_files`, which works well for common extensions but may not serve files with unusual extensions. Lua module would allow runtime directory scanning for hash-matching files regardless of extension.
### 1.4 Basic HEAD Endpoint
- [x] Implement FastCGI handler for `HEAD /<sha256>`
- [x] Query database for blob metadata (single source of truth)
- [x] Extract SHA-256 from URL (strip extensions)
- [x] Return proper headers (Content-Type, Content-Length, etc.)
- [x] Return 404 if blob doesn't exist in database
- [x] Add server timing headers for debugging
### 1.5 Testing & Validation
- [x] Create test blobs with known SHA-256 hashes
- [x] Verify nginx serves files correctly with extension support
- [x] Verify HEAD requests return proper metadata
- [x] Test with missing files (404 responses)
- [x] Test HEAD requests with and without extensions
- [ ] Performance test with large files
---
## Phase 2: Upload & Authentication (BUD-02)
### 2.1 Nostr Authentication Setup
- [x] Integrate nostr_core_lib submodule
- [x] Update Makefile to include nostr_core_lib paths and static library
- [x] Build libnostr_core_x64.a using provided build.sh script
- [x] Add system dependencies: -lsecp256k1 -lssl -lcrypto -lcurl -lz -ldl -lpthread -lm
- [x] Implement authentication functions in main.c (BUD-02 section):
- [x] `parse_authorization_header()` - Extract JSON from "Nostr base64(event)" header
- [x] `validate_blossom_event()` - Validate Blossom-specific requirements (kind 24242, content hash, method, expiration)
- [x] `authenticate_request()` - Main orchestrator function
- [x] Leverage existing nostr_core_lib functions:
- [x] Use `nostr_validate_event()` for structure + signature validation (from nip001.h)
- [x] Use standardized error codes from nostr_common.h (NOSTR_SUCCESS, NOSTR_ERROR_EVENT_INVALID_SIGNATURE, etc.)
- [x] Use `nostr_strerror()` for error message translation
### 2.2 Upload Endpoint Implementation
- [x] Implement `PUT /upload` endpoint
- [x] Parse Authorization header (Nostr base64 event extraction)
- [x] Stream file upload to temporary location
- [x] Calculate SHA-256 hash during upload
- [x] Validate hash matches authorization if provided
- [x] Move file to permanent location
- [x] Store metadata in database (including uploader_pubkey and filename)
- [x] Return blob descriptor JSON response
### 2.3 Blob Descriptor Response
- [x] Implement blob descriptor structure
- [x] Required fields: url, sha256, size, type, uploaded
- [x] Handle MIME type detection
- [x] Generate proper blob URLs
- [x] Add optional server-specific fields (uploader_pubkey, filename)
### 2.4 Error Handling
- [x] Implement proper HTTP status codes
- [x] 400 Bad Request for invalid data
- [x] 401 Unauthorized for auth failures
- [x] 409 Conflict for hash mismatches
- [x] 413 Payload Too Large for size limits
- [x] 500 Internal Server Error for system issues
- [x] Add detailed error messages
- [x] Implement request logging
### 2.5 List Blobs Endpoint
- [x] Implement `GET /list/<pubkey>` endpoint
- [x] Extract pubkey from URL path
- [x] Query database for blobs uploaded by specified pubkey
- [x] Support `since` and `until` query parameters for date filtering
- [x] Return JSON array of blob descriptors
- [x] Handle empty results gracefully
- [x] Implement optional authorization with kind 24242 event validation
- [x] Validate `t` tag is set to "list"
- [x] Check authorization expiration
- [x] Verify event signature and structure
### 2.6 Delete Blob Endpoint
- [x] Implement `DELETE /<sha256>` endpoint
- [x] Extract SHA-256 hash from URL path
- [x] Require authorization with kind 24242 event validation
- [x] Validate `t` tag is set to "delete"
- [x] Verify at least one `x` tag matches the requested hash
- [x] Check authorization expiration
- [x] Verify event signature and structure
- [x] Check blob exists in database
- [x] Verify uploader_pubkey matches authorized pubkey (ownership check)
- [x] Remove blob file from filesystem
- [x] Remove blob metadata from database
- [x] Handle file deletion errors gracefully
- [x] Return appropriate success/error responses
### 2.7 Testing & Validation
- [x] Test uploads without authentication
- [x] Test uploads with valid nostr auth
- [x] Test uploads with invalid auth
- [x] Test hash mismatch scenarios
- [ ] Test file size limits
- [x] Verify blob descriptors are correct
- [x] Verify database metadata storage (uploader_pubkey and filename)
---
## Phase 3: Upload Requirements (BUD-06)
### 3.1 Upload Policy Configuration
- [ ] Add server configuration options
- [ ] Maximum file size limits
- [ ] Allowed MIME types
- [ ] Authentication requirements
- [ ] Rate limiting settings
- [ ] Storage quota limits
- [ ] Hash-based banning/filtering
### 3.2 HEAD /upload Endpoint Implementation
- [ ] Implement `HEAD /upload` endpoint for pre-flight upload validation
- [ ] Parse client headers:
- [ ] `X-SHA-256`: blob's SHA-256 hash
- [ ] `X-Content-Length`: blob size in bytes
- [ ] `X-Content-Type`: blob's MIME type
- [ ] Handle optional Authorization header (same as PUT /upload)
- [ ] Perform validation checks without file transfer:
- [ ] Validate SHA-256 format
- [ ] Check file size against limits
- [ ] Validate MIME type restrictions
- [ ] Check authentication if required
- [ ] Check if hash already exists (duplicate detection)
- [ ] Check if hash is banned
- [ ] Return appropriate HTTP status codes:
- [ ] `200 OK` - upload can proceed
- [ ] `400 Bad Request` - invalid headers
- [ ] `401 Unauthorized` - auth required
- [ ] `403 Forbidden` - not permitted (banned hash, etc.)
- [ ] `411 Length Required` - missing content length
- [ ] `413 Content Too Large` - file too large
- [ ] `415 Unsupported Media Type` - invalid MIME type
- [ ] Add `X-Reason` header with human-readable error messages
### 3.3 Upload Pre-validation Logic
- [ ] Create validation functions that can be shared between HEAD and PUT endpoints
- [ ] `validate_upload_headers()` - check required headers present and valid
- [ ] `check_file_size_limits()` - enforce maximum size restrictions
- [ ] `check_mime_type_allowed()` - validate against allowed types list
- [ ] `check_hash_restrictions()` - check banned hashes, duplicates
- [ ] `check_upload_permissions()` - user-specific upload rights
### 3.4 DOS Protection Benefits
- [ ] Implement early rejection before file transfer:
- [ ] Authentication happens before any file data sent
- [ ] Size validation prevents large file uploads that would be rejected
- [ ] MIME type checking prevents unwanted file types
- [ ] Hash checking prevents duplicate uploads
- [ ] Update PUT /upload to use same validation functions for consistency
### 3.5 Client Integration Support
- [ ] Update nginx configuration to properly handle HEAD requests to /upload
- [ ] Ensure FastCGI handles HEAD method for /upload endpoint
- [ ] Add CORS headers for preflight requests
### 3.6 Testing & Validation
- [ ] Test HEAD /upload with valid headers
- [ ] Test various error scenarios (missing headers, invalid formats)
- [ ] Test authorization requirements
- [ ] Test policy enforcement (size limits, MIME types, banned hashes)
- [ ] Verify error responses match BUD-06 specification
- [ ] Test client workflow: HEAD check → PUT upload
- [ ] Verify DOS protection effectiveness
---
## Phase 4: Advanced Authentication & Administration System
### 4.1 Flexible Authentication Rules System
#### 4.1.1 Database Schema Extension
- [ ] Create authentication rules tables
- [ ] `auth_rules` table: rule_type, rule_target, operation, rule_value, enabled, expires_at
- [ ] `auth_cache` table: performance caching for rule evaluation results
- [ ] Add indexes on rule_type, rule_target, operation for performance
#### 4.1.2 Authentication Rule Types Implementation
- [ ] Basic rule types:
- [ ] `pubkey_whitelist`: Only specific pubkeys allowed
- [ ] `pubkey_blacklist`: Specific pubkeys banned
- [ ] `hash_blacklist`: Specific file hashes cannot be uploaded
- [ ] `mime_type_whitelist`: Only specific content types allowed
- [ ] `mime_type_blacklist`: Specific content types banned
- [ ] Advanced rule types:
- [ ] `rate_limit`: Limit operations per pubkey/IP per time period
- [ ] `size_limit`: Per-pubkey or global size limits
- [ ] `conditional`: Complex JSON-based rules (time-based, size-based, etc.)
#### 4.1.3 Rule Evaluation Engine
- [ ] Core authentication functions:
- [ ] `evaluate_auth_rules()`: Main rule evaluation with caching
- [ ] `check_rule_cache()`: Performance optimization layer
- [ ] `process_rule_priority()`: Handle rule precedence and conflicts
- [ ] `update_auth_cache()`: Store evaluation results for reuse
- [ ] Integration points:
- [ ] Extend `handle_upload_request()` with rule evaluation
- [ ] Extend `handle_delete_request()` with rule evaluation
- [ ] Extend `handle_list_request()` with rule evaluation (optional)
#### 4.1.4 Rule Management Interface
- [ ] SQL-based rule management:
- [ ] `add_auth_rule()`: Add new authentication rules
- [ ] `remove_auth_rule()`: Remove rules by ID
- [ ] `list_auth_rules()`: Query existing rules with filters
- [ ] `update_auth_rule()`: Modify existing rule parameters
### 4.2 Nostr-Native Administrative Interface
#### 4.2.1 Server Identity Management
- [ ] Server keypair generation and storage:
- [ ] Generate server public/private keypair on first run
- [ ] Store server pubkey in `server_config` table
- [ ] Secure private key storage (encrypted file or environment)
- [ ] Key rotation capabilities for security
#### 4.2.2 Administrator Management System
- [ ] Administrator database schema:
- [ ] `administrators` table: pubkey, permissions, added_by, expires_at
- [ ] Permission levels: rules, config, users, stats, * (full access)
- [ ] Initial admin setup during server deployment
- [ ] Administrative functions:
- [ ] `check_admin_permissions()`: Verify admin authorization
- [ ] `add_administrator()`: Grant admin privileges
- [ ] `remove_administrator()`: Revoke admin privileges
- [ ] `list_administrators()`: Query admin list with permissions
#### 4.2.3 Administrative Event Types
- [ ] Event kind definitions:
- [ ] Kind 30242: Administrative commands (rule_add, rule_remove, config_set, etc.)
- [ ] Kind 30243: Administrative queries (stats_get, rule_list, audit_log, etc.)
- [ ] Kind 30244: Administrative responses (command results, query data)
- [ ] Command implementations:
- [ ] Rule management: `rule_add`, `rule_remove`, `rule_update`, `rule_list`
- [ ] System management: `config_set`, `config_get`, `admin_add`, `admin_remove`
- [ ] Query operations: `stats_get`, `blob_list`, `audit_log`, `storage_cleanup`
#### 4.2.4 Administrative Event Processing
- [ ] HTTP administrative endpoint:
- [ ] `POST /admin` with nostr event authorization
- [ ] JSON command interface with parameter validation
- [ ] Synchronous response with operation results
- [ ] Direct nostr relay integration (future enhancement):
- [ ] Subscribe to administrative events on configured relays
- [ ] Real-time event processing and response
- [ ] Publish response events back to relays
#### 4.2.5 Administrative Audit Trail
- [ ] Administrative logging system:
- [ ] `admin_log` table: track all administrative actions
- [ ] Event ID references for nostr event traceability
- [ ] Success/failure tracking with detailed error messages
- [ ] Audit query capabilities for compliance
#### 4.2.6 Security & Permission Framework
- [ ] Multi-level permission system:
- [ ] Granular permissions: rules, config, users, stats
- [ ] Permission inheritance and delegation
- [ ] Time-limited administrative access (expires_at)
- [ ] Authentication security:
- [ ] Strong nostr signature validation
- [ ] Administrator authorization chain verification
- [ ] Command-specific permission checks
- [ ] Rate limiting for administrative operations
### 4.3 Integration & Testing
- [ ] Authentication system integration:
- [ ] Integrate rule evaluation into existing authentication flow
- [ ] Maintain backward compatibility with nostr-only authentication
- [ ] Performance testing with rule caching
- [ ] Administrative system testing:
- [ ] Test all administrative commands and queries
- [ ] Verify permission enforcement and security
- [ ] Test audit logging and compliance features
- [ ] Load testing for administrative operations
---
## Phase 5: Optional Features
---
## Phase 5: Optional Features
### 4.1 User Server Lists (BUD-03) - Optional
- [ ] Implement server list advertisement
- [ ] Handle kind:10063 events
- [ ] Create server discovery endpoint
- [ ] Test client fallback scenarios
### 4.2 Blob Mirroring (BUD-04) - Optional
- [ ] Implement `PUT /mirror` endpoint
- [ ] Add URL downloading capability
- [ ] Implement hash verification
- [ ] Handle authorization for mirroring
- [ ] Test inter-server mirroring
### 4.3 Media Optimization (BUD-05) - Optional
- [ ] Implement `PUT /media` endpoint
- [ ] Add media processing libraries
- [ ] Implement optimization algorithms
- [ ] Handle various media formats
- [ ] Test optimization pipeline
### 4.4 Payment Integration (BUD-07) - Optional
- [ ] Implement 402 Payment Required responses
- [ ] Add Lightning payment support
- [ ] Add Cashu payment support
- [ ] Implement payment verification
- [ ] Test payment flows
### 4.5 NIP-94 Metadata (BUD-08) - Optional
- [ ] Add NIP-94 tag generation
- [ ] Extend blob descriptor responses
- [ ] Generate magnet links if supported
- [ ] Test metadata compatibility
### 4.6 Blob Reporting (BUD-09) - Optional
- [ ] Implement `PUT /report` endpoint
- [ ] Handle NIP-56 report events
- [ ] Add moderation interface
- [ ] Implement content filtering
- [ ] Test reporting workflow
---
## Development Milestones
### Milestone 1: Basic Functionality (Phase 1 Complete)
- [x] nginx serves files by hash with extension support
- [x] HEAD requests return metadata from database
- [x] Database stores blob information with proper schema
### Milestone 2: Full Upload Support (Phase 2 Pending)
- [x] Basic upload functionality working (PUT requests accepted)
- [x] SHA-256 hash calculation during upload
- [x] File storage to blobs/ directory
- [x] Blob descriptor JSON response
- [x] Authenticated uploads working (Nostr kind 24242 event validation)
- [x] Proper error handling for upload scenarios
- [x] Database metadata storage during upload (with uploader_pubkey and filename)
- [x] List blobs endpoint implemented (GET /list/<pubkey>)
- [x] Delete blob endpoint implemented (DELETE /<sha256>)
### Milestone 3: Policy Compliance (Phase 3 Pending)
- [ ] Upload requirements implemented
- [ ] Server policies configurable
- [ ] Spec compliance verified
### Milestone 4: Advanced Authentication (Phase 4 Complete)
- [ ] Flexible authentication rules system operational
- [ ] Nostr-native administrative interface implemented
- [ ] Rule evaluation engine with caching performance
- [ ] Administrative audit trail and compliance features
### Milestone 5: Production Ready (Phase 5 Complete)
- Optional features implemented as needed
- Performance optimized
- Security hardened
- Documentation complete
---
## Testing Strategy
### Unit Tests
- [ ] Authentication validation functions
- [ ] SHA-256 hash calculation
- [ ] Database operations
- [ ] Configuration parsing
### Integration Tests
- [ ] nginx + FastCGI integration
- [ ] End-to-end upload/download flows
- [ ] Error scenario handling
- [ ] Multi-client concurrent access
### Performance Tests
- [ ] Large file uploads/downloads
- [ ] Concurrent request handling
- [ ] Database query performance
- [ ] Memory usage optimization
### Compliance Tests
- [ ] Blossom protocol compliance
- [ ] Nostr event validation
- [ ] HTTP specification compliance
- [ ] Security best practices
---
## Future Improvements
### Upload Security & Performance Enhancements
**Current Issue**: The existing upload flow has a DOS vulnerability where large files are loaded entirely into memory before authentication occurs. This allows unauthenticated attackers to exhaust server memory.
**Current Flow**:
```
Client → nginx → FastCGI ginxsom
├─ reads entire file into memory (malloc + fread)
├─ validates auth (after file in memory)
└─ saves to blobs/ or errors
```
**Proposed Solution - nginx Upload Module**:
```
Client → nginx upload module → temp file → FastCGI ginxsom
├─ saves to /tmp/uploads/ ├─ validates auth quickly
└─ passes metadata only ├─ moves file to blobs/
└─ or deletes temp file
```
**Benefits**:
- Eliminates DOS vulnerability - nginx handles large files efficiently
- Fast auth validation - no waiting for full upload
- Leverages nginx strengths - what it's designed for
- Better scalability - memory usage independent of file size
**Implementation Requirements**:
- nginx upload module configuration
- Temp file cleanup handling
- Modified FastCGI code to process file paths instead of stdin
- Proper error handling for temp file operations
**Alternative Enhancement - HTTP 100 Continue**:
Could propose new Blossom BUD for two-phase upload:
1. Client sends headers with `Expect: 100-continue` + auth event
2. Server validates early (signature, expiration, pubkey)
3. Server responds `100 Continue` or `401 Unauthorized`
4. Client only sends file data if authorized
**Priority**: Implement after core BUD compliance is complete.
---
## Security Considerations
- [ ] Input validation on all endpoints
- [ ] Rate limiting to prevent abuse
- [ ] Secure file storage permissions
- [ ] Database injection prevention
- [ ] Memory safety in C implementation
- [ ] Proper error message sanitization
- [ ] Log security (no sensitive data)
- [ ] **Upload DOS vulnerability** - Current implementation vulnerable to memory exhaustion attacks
---
## Performance Optimizations
- [ ] nginx direct file serving (bypasses application)
- [ ] FastCGI connection pooling
- [ ] Database connection management
- [ ] Efficient hash calculation
- [ ] Memory-mapped file operations
- [ ] Hierarchical file storage (future)
- [ ] CDN integration support
---
## Deployment Checklist
- [ ] nginx configuration template
- [ ] FastCGI service configuration
- [ ] Database initialization scripts
- [ ] Log rotation setup
- [ ] Monitoring and health checks
- [ ] Backup procedures
- [ ] Security hardening guide
- [ ] Documentation and examples