laantungir/ginxsom
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

readme.md and implementation.md

Browse Source
This commit is contained in:
Your Name
2025-08-18 17:42:51 -04:00
parent 5f7d2197e8
commit e641c813eb
8 changed files with 1505 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.gitmodules vendored Normal file
View File

@@ -0,0 +1,6 @@
[submodule "blossom"]
path = blossom
url = ssh://git@git.laantungir.net:222/laantungir/blossom.git
[submodule "nostr_core_lib"]
path = nostr_core_lib
url = ssh://git@git.laantungir.net:222/laantungir/nostr_core_lib.git

509
BLOSSOM_FLOW.md Normal file
View File

@@ -0,0 +1,509 @@
# Blossom Protocol Flow Charts
This document provides ASCII flow charts illustrating how each Blossom Upgrade Document (BUD) works in practice.
---
## BUD-01: Basic Blob Retrieval
### GET Request Flow
```
Client Request nginx Database
| | |
| GET /<sha256> | |
|--------------------------->| |
| | |
| | Check file exists |
| | in blobs/ directory |
| | |
| File Found | |
|<---------------------------| |
| 200 OK + File Content | |
| | |
| File Not Found | |
|<---------------------------| |
| 404 Not Found | |
```
### HEAD Request Flow
```
Client Request FastCGI App Database
| | |
| HEAD /<sha256> | |
|------------------------->| |
| | |
| | Query blob metadata |
| |-------------------------->|
| | |
| | Blob exists |
| |<--------------------------|
| | size, type, uploaded_at |
| | |
| 200 OK + Headers | |
|<-------------------------| |
| Content-Type: image/png | |
| Content-Length: 12345 | |
| | |
| | Blob not found |
| |<--------------------------|
| | |
| 404 Not Found | |
|<-------------------------| |
```
---
## BUD-02: Blob Upload & Authentication
### Upload Without Authentication
```
Client FastCGI App File System Database
| | | |
| PUT /upload | | |
|--------------------------->| | |
| Content: <binary data> | | |
| | | |
| | Calculate SHA-256 | |
| | hash of data | |
| | | |
| | Write to blobs/ | |
| |-------------------------->| |
| | | File saved |
| |<--------------------------| |
| | | |
| | Store metadata | |
| |----------------------------------------------->|
| | | | INSERT |
| |<-----------------------------------------------|
| | | |
| 200 OK | | |
|<---------------------------| | |
| { | | |
| "url": "https://...", | | |
| "sha256": "abc123...", | | |
| "size": 12345, | | |
| "type": "image/png", | | |
| "uploaded": 1234567890 | | |
| } | | |
```
### Upload With Nostr Authentication
```
Client FastCGI App Nostr Validation File System Database
| | | | |
| PUT /upload | | | |
|--------------------------->| | | |
| Authorization: Nostr <evt> | | | |
| Content: <binary data> | | | |
| | | | |
| | Parse auth event | | |
| |-------------------------->| | |
| | | • Verify signature | |
| | | • Check expiration | |
| | | • Validate tags | |
| |<--------------------------| | |
| | Valid ✓ | | |
| | | | |
| | Calculate hash | | |
| | | | |
| | Compare with 'x' tag | | |
| | in auth event | | |
| | | | |
| | Hash matches ✓ | | |
| | | | |
| | Save file | | |
| |----------------------------------------------->| |
| | | | Write | |
| |<-----------------------------------------------| |
| | | | |
| | Store metadata | | |
| |-------------------------------------------------------------->|
| | (include uploader_pubkey) | | | INSERT |
| |<--------------------------------------------------------------|
| | | | |
| 200 OK + Blob Descriptor | | | |
|<---------------------------| | | |
```
### Authentication Failure Flows
```
Invalid Signature:
Client --> FastCGI --> Nostr Validation --> 401 Unauthorized
Expired Event:
Client --> FastCGI --> Nostr Validation --> 401 Unauthorized
Hash Mismatch:
Client --> FastCGI --> Hash Check --> 409 Conflict
```
---
## BUD-03: User Server Lists
### Server List Publication
```
User/Client Nostr Relay Other Clients
| | |
| Publish kind:10063 | |
| event with server tags | |
|------------------------>| |
| { | |
| "kind": 10063, | |
| "tags": [ | |
| ["server", "cdn1"], | |
| ["server", "cdn2"] | |
| ] | |
| } | |
| | |
| | Store event |
| | |
| | |
| | Query for user's |
| | server list |
| |<--------------------------|
| | |
| | Return server list |
| |-------------------------->|
```
### Client Blob Discovery
```
Client Original URL Author's Servers
| | |
| GET blob from URL | |
|------------------------->| |
| | |
| 404 Not Found | |
|<-------------------------| |
| | |
| Extract SHA-256 hash | |
| from URL | |
| | |
| Query nostr for author's | |
| server list (kind:10063) | |
| | |
| Try each server in order | |
|--------------------------------------------------->|
| | |
| GET /<sha256> | |
| | |
| 200 OK + File Content | |
|<---------------------------------------------------|
```
---
## BUD-04: Blob Mirroring
### Mirror Request Flow
```
Client Server B Server A Database
| | | |
| PUT /mirror | | |
|------------------------>| | |
| { | | |
| "url": "https:// | | |
| serverA/abc123..." | | |
| } | | |
| Authorization: <event> | | |
| | | |
| | Validate auth event | |
| | (check x tag matches) | |
| | | |
| | Download from URL | |
| |------------------------>| |
| | | |
| | Stream blob content | |
| |<------------------------| |
| | | |
| | Calculate SHA-256 | |
| | during download | |
| | | |
| | Verify hash matches | |
| | x tag in auth event | |
| | | |
| | Save blob locally | |
| | | |
| | Store metadata | |
| |------------------------------------------------>|
| | | | INSERT |
| |<------------------------------------------------|
| | | |
| 200 OK | | |
|<------------------------| | |
| Blob Descriptor | | |
```
---
## BUD-05: Media Optimization
### Media Processing Flow
```
Client Media Server Optimization Engine File System
| | | |
| PUT /media | | |
|------------------------>| | |
| Content: <raw image> | | |
| Content-Type: image/png | | |
| Authorization: <event> | | |
| | | |
| | Validate auth | |
| | (type="media") | |
| | | |
| | Process media | |
| |------------------------>| |
| | | • Resize/compress |
| | | • Format conversion |
| | | • Quality optimization|
| |<------------------------| Optimized media |
| | | |
| | Calculate new hash | |
| | | |
| | Save optimized blob | |
| |------------------------------------------------>|
| | | | Write |
| |<------------------------------------------------|
| | | |
| 200 OK | | |
|<------------------------| | |
| { | | |
| "url": "new_hash...", | | |
| "sha256": "def456...",| | |
| "size": 8765, | | |
| "type": "image/webp" | | |
| } | | |
```
---
## BUD-06: Upload Requirements
### Upload Requirement Check
```
Client FastCGI App Configuration
| | |
| HEAD /upload | |
|------------------------>| |
| Authorization: <event> | |
| (optional) | |
| | |
| | Check server config |
| |------------------------>|
| | | • Max file size
| | | • Auth required?
| | | • Allowed types
| |<------------------------| • Rate limits
| | |
| | Validate auth if |
| | provided |
| | |
| 200 OK | |
|<------------------------| |
| X-Upload-Size-Limit: | |
| 10485760 | |
| X-Upload-Auth-Required: | |
| true | |
| X-Upload-Types: | |
| image/*,video/* | |
```
### Upload Policy Enforcement
```
Client FastCGI App Policy Check
| | |
| PUT /upload | |
|------------------------>| |
| Content-Length: 50MB | |
| | |
| | Check against limits |
| |------------------------>|
| | | Size: 50MB > 10MB ✗
| |<------------------------| REJECT
| | |
| 413 Payload Too Large | |
|<------------------------| |
| { | |
| "error": "File too | |
| large. Max: 10MB" | |
| } | |
```
---
## BUD-07: Paid Upload/Download
### Payment Required Flow
```
Client Paid Server Payment Provider
| | |
| PUT /upload | |
|------------------------>| |
| | |
| | Check payment required |
| | |
| 402 Payment Required | |
|<------------------------| |
| X-Lightning: lnbc... | |
| X-Cashu: creq... | |
| | |
| User pays invoice | |
|-------------------------------------------------->|
| | |
| PUT /upload (retry) | |
|------------------------>| |
| X-Lightning: <preimage> | |
| | |
| | Verify payment proof |
| |------------------------>|
| | | Valid ✓
| |<------------------------|
| | |
| | Process upload |
| | |
| 200 OK + Blob Desc | |
|<------------------------| |
```
### Payment Methods
```
Lightning Payment:
Client --> Server --> Lightning Node --> Payment Verification --> Upload Success
Cashu Payment:
Client --> Server --> Cashu Mint --> Token Validation --> Upload Success
```
---
## BUD-08: NIP-94 File Metadata
### Enhanced Blob Descriptor
```
Client Upload FastCGI App Metadata Generation Response
| | | |
| PUT /upload | | |
|---------------------> | | |
| | | |
| | Process file | |
| | | |
| | Generate NIP-94 tags | |
| |------------------------>| |
| | | • ["url", "..."] |
| | | • ["m", "image/png"] |
| | | • ["x", "hash..."] |
| | | • ["size", "12345"] |
| | | • ["magnet", "..."] |
| |<------------------------| NIP-94 tags |
| | | |
| | Build response | |
| |------------------------------------------------>|
| Enhanced Response | | |
|<--------------------- | | |
| { | | |
| "url": "...", | | |
| "sha256": "...", | | |
| "nip94": [ | | |
| ["url", "..."], | | |
| ["m", "..."], | | |
| ["x", "..."] | | |
| ] | | |
| } | | |
```
---
## BUD-09: Blob Reporting
### Content Moderation Flow
```
Client/User FastCGI App Moderation System Action
| | | |
| PUT /report | | |
|---------------------> | | |
| NIP-56 report event | | |
| { | | |
| "kind": 1984, | | |
| "tags": [ | | |
| ["x", "hash..."] | | |
| ], | | |
| "content": "spam" | | |
| } | | |
| | | |
| | Validate report | |
| | | |
| | Store report | |
| |------------------------>| |
| | | • Log report |
| | | • Check reporter |
| | | • Queue for review |
| |<------------------------| Stored |
| | | |
| 200 OK | | |
|<--------------------- | | |
| | | |
| | | Manual Review |
| | |---------------------> |
| | | | Remove blob
| | | | Block hash
| | | | Ban user
```
### Automated Moderation
```
Trusted Reporter Report --> Immediate Action --> Blob Removed
Multiple Reports --> Temporary Hide --> Manual Review --> Final Decision
```
---
## Complete Ginxsom Architecture Flow
### Nginx + FastCGI Integration
```
Internet nginx FastCGI App Database File System
| | | | |
| GET /<sha256> | | | |
|-------------------->| | | |
| | Direct file serve | | |
| |----------------------------------------->| |
| | | | blobs/ |
| File Content |<-----------------------------------------| <hash> |
|<--------------------| | | |
| | | | |
| HEAD /<sha256> | | | |
|-------------------->| | | |
| | Forward to FastCGI | | |
| |-------------------->| | |
| | | Query metadata | |
| | |------------------->| |
| | | | SELECT |
| | |<-------------------| |
| | Headers response | | |
| |<--------------------| | |
| Metadata Headers | | | |
|<--------------------| | | |
| | | | |
| PUT /upload | | | |
|-------------------->| | | |
| | Forward to FastCGI | | |
| |-------------------->| | |
| | | Process upload | |
| | | | |
| | | Store metadata | |
| | |------------------->| |
| | | | INSERT |
| | |<-------------------| |
| | | | |
| | | Save file | |
| | |-------------------------------------->|
| | | | Write |
| | |<--------------------------------------|
| | JSON response | | |
| |<--------------------| | |
| Blob Descriptor | | | |
|<--------------------| | | |
```

265
IMPLEMENTATION.md Normal file
View File

@@ -0,0 +1,265 @@
# Ginxsom Blossom Server Implementation Checklist
This document outlines the implementation plan for ginxsom, a FastCGI-based Blossom server designed to work with nginx for optimal performance.
## Architecture Overview
- **nginx**: Handles static file serving (GET /<sha256>) for maximum performance
- **FastCGI Application**: Handles authenticated operations, metadata queries, uploads
- **SQLite Database**: Stores blob metadata and server configuration
- **File Storage**: Flat directory structure initially, hierarchical optimization later
---
## Phase 1: Basic File Serving & Retrieval (BUD-01)
### 1.1 Infrastructure Setup
- [ ] Create basic directory structure
- [ ] Create `blobs/` directory for file storage
- [ ] Create `db/` directory for SQLite database
- [ ] Create `logs/` directory for application logs
- [ ] Set up proper permissions (nginx readable, app writable)
### 1.2 Database Schema
- [ ] Design SQLite schema for blob metadata
- [ ] `blobs` table: sha256, size, type, uploaded_at, uploader_pubkey
- [ ] `server_config` table: key-value pairs for server settings
- [ ] Create database initialization script
- [ ] Add proper indexes on sha256 hash
### 1.3 nginx Configuration
- [ ] Configure nginx for static file serving
- [ ] Set up location block for `GET /<sha256>` pattern
- [ ] Configure proper MIME type detection
- [ ] Add proper headers (Cache-Control, ETag, etc.)
- [ ] Handle 404s gracefully when blob doesn't exist
- [ ] Configure FastCGI pass-through for non-GET requests
### 1.4 Basic HEAD Endpoint
- [ ] Implement FastCGI handler for `HEAD /<sha256>`
- [ ] Query database for blob metadata
- [ ] Return proper headers (Content-Type, Content-Length, etc.)
- [ ] Return 404 if blob doesn't exist
- [ ] Add server timing headers for debugging
### 1.5 Testing & Validation
- [ ] Create test blobs with known SHA-256 hashes
- [ ] Verify nginx serves files correctly
- [ ] Verify HEAD requests return proper metadata
- [ ] Test with missing files (404 responses)
- [ ] Performance test with large files
---
## Phase 2: Upload & Authentication (BUD-02)
### 2.1 Nostr Authentication Setup
- [ ] Integrate nostr_core_lib submodule
- [ ] Implement nostr event validation
- [ ] Verify event signature (schnorr)
- [ ] Validate event structure (kind 24242)
- [ ] Check required fields (t, expiration, x tags)
- [ ] Implement expiration checking
- [ ] Create authentication middleware
### 2.2 Upload Endpoint Implementation
- [ ] Implement `PUT /upload` endpoint
- [ ] Parse Authorization header (optional but recommended)
- [ ] Stream file upload to temporary location
- [ ] Calculate SHA-256 hash during upload
- [ ] Validate hash matches authorization if provided
- [ ] Move file to permanent location
- [ ] Store metadata in database
- [ ] Return blob descriptor JSON response
### 2.3 Blob Descriptor Response
- [ ] Implement blob descriptor structure
- [ ] Required fields: url, sha256, size, type, uploaded
- [ ] Handle MIME type detection
- [ ] Generate proper blob URLs
- [ ] Add optional server-specific fields
### 2.4 Error Handling
- [ ] Implement proper HTTP status codes
- [ ] 400 Bad Request for invalid data
- [ ] 401 Unauthorized for auth failures
- [ ] 409 Conflict for hash mismatches
- [ ] 413 Payload Too Large for size limits
- [ ] 500 Internal Server Error for system issues
- [ ] Add detailed error messages
- [ ] Implement request logging
### 2.5 Testing & Validation
- [ ] Test uploads without authentication
- [ ] Test uploads with valid nostr auth
- [ ] Test uploads with invalid auth
- [ ] Test hash mismatch scenarios
- [ ] Test file size limits
- [ ] Verify blob descriptors are correct
---
## Phase 3: Upload Requirements (BUD-06)
### 3.1 Upload Policy Configuration
- [ ] Add server configuration options
- [ ] Maximum file size limits
- [ ] Allowed MIME types
- [ ] Authentication requirements
- [ ] Rate limiting settings
- [ ] Storage quota limits
### 3.2 HEAD /upload Endpoint
- [ ] Implement `HEAD /upload` endpoint
- [ ] Return upload requirements in headers
- [ ] Handle optional Authorization header
- [ ] Return proper status codes for policy checks
- [ ] Add custom headers for requirements
### 3.3 Upload Validation
- [ ] Implement pre-upload validation
- [ ] Check file size before processing
- [ ] Validate MIME types if restricted
- [ ] Check authentication requirements
- [ ] Verify user permissions/quotas
### 3.4 Testing & Validation
- [ ] Test upload requirements endpoint
- [ ] Test policy enforcement
- [ ] Test with various client scenarios
- [ ] Verify error responses match spec
---
## Phase 4: Optional Features
### 4.1 User Server Lists (BUD-03) - Optional
- [ ] Implement server list advertisement
- [ ] Handle kind:10063 events
- [ ] Create server discovery endpoint
- [ ] Test client fallback scenarios
### 4.2 Blob Mirroring (BUD-04) - Optional
- [ ] Implement `PUT /mirror` endpoint
- [ ] Add URL downloading capability
- [ ] Implement hash verification
- [ ] Handle authorization for mirroring
- [ ] Test inter-server mirroring
### 4.3 Media Optimization (BUD-05) - Optional
- [ ] Implement `PUT /media` endpoint
- [ ] Add media processing libraries
- [ ] Implement optimization algorithms
- [ ] Handle various media formats
- [ ] Test optimization pipeline
### 4.4 Payment Integration (BUD-07) - Optional
- [ ] Implement 402 Payment Required responses
- [ ] Add Lightning payment support
- [ ] Add Cashu payment support
- [ ] Implement payment verification
- [ ] Test payment flows
### 4.5 NIP-94 Metadata (BUD-08) - Optional
- [ ] Add NIP-94 tag generation
- [ ] Extend blob descriptor responses
- [ ] Generate magnet links if supported
- [ ] Test metadata compatibility
### 4.6 Blob Reporting (BUD-09) - Optional
- [ ] Implement `PUT /report` endpoint
- [ ] Handle NIP-56 report events
- [ ] Add moderation interface
- [ ] Implement content filtering
- [ ] Test reporting workflow
---
## Development Milestones
### Milestone 1: Basic Functionality ✓ (Phase 1 Complete)
- nginx serves files by hash
- HEAD requests return metadata
- Database stores blob information
### Milestone 2: Full Upload Support (Phase 2 Complete)
- Authenticated uploads working
- Proper error handling
- Blob descriptors returned correctly
### Milestone 3: Policy Compliance (Phase 3 Complete)
- Upload requirements implemented
- Server policies configurable
- Spec compliance verified
### Milestone 4: Production Ready (Phase 4 Complete)
- Optional features implemented as needed
- Performance optimized
- Security hardened
- Documentation complete
---
## Testing Strategy
### Unit Tests
- [ ] Authentication validation functions
- [ ] SHA-256 hash calculation
- [ ] Database operations
- [ ] Configuration parsing
### Integration Tests
- [ ] nginx + FastCGI integration
- [ ] End-to-end upload/download flows
- [ ] Error scenario handling
- [ ] Multi-client concurrent access
### Performance Tests
- [ ] Large file uploads/downloads
- [ ] Concurrent request handling
- [ ] Database query performance
- [ ] Memory usage optimization
### Compliance Tests
- [ ] Blossom protocol compliance
- [ ] Nostr event validation
- [ ] HTTP specification compliance
- [ ] Security best practices
---
## Security Considerations
- [ ] Input validation on all endpoints
- [ ] Rate limiting to prevent abuse
- [ ] Secure file storage permissions
- [ ] Database injection prevention
- [ ] Memory safety in C implementation
- [ ] Proper error message sanitization
- [ ] Log security (no sensitive data)
---
## Performance Optimizations
- [ ] nginx direct file serving (bypasses application)
- [ ] FastCGI connection pooling
- [ ] Database connection management
- [ ] Efficient hash calculation
- [ ] Memory-mapped file operations
- [ ] Hierarchical file storage (future)
- [ ] CDN integration support
---
## Deployment Checklist
- [ ] nginx configuration template
- [ ] FastCGI service configuration
- [ ] Database initialization scripts
- [ ] Log rotation setup
- [ ] Monitoring and health checks
- [ ] Backup procedures
- [ ] Security hardening guide
- [ ] Documentation and examples

316
README.md
View File

@@ -1 +1,315 @@
Ginxsom # ginxsom 🌸
A high-performance [Blossom](https://github.com/hzrd149/blossom) server implementation in C, designed for optimal integration with nginx for blazing-fast file serving.
## Overview
ginxsom is a Blossom protocol server implemented as a FastCGI application that integrates seamlessly with nginx. nginx handles static file serving directly while ginxsom processes authenticated operations (uploads, deletes, management) via FastCGI. This architecture provides optimal performance with nginx's excellent static file serving and C's efficiency for cryptographic operations.
### Why ginxsom?
- **Performance**: C application with nginx static serving = maximum throughput
- **Simplicity**: Clean separation between static serving (nginx) and dynamic operations (C app)
- **Scalability**: nginx handles the heavy file serving load, C app handles auth and metadata
- **Standards Compliant**: Full Blossom protocol implementation with nostr authentication
## Architecture
```
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
│ Client │ │ nginx │ │ ginxsom FastCGI │
│ │───▶│ │───▶│ │
│ │ │ │ │ │
└─────────────┘ └──────────────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────┐ ┌─────────────────┐
│ Blobs │ │ SQLite DB │
│ (flat files)│ │ (metadata) │
└─────────────┘ └─────────────────┘
```
### Request Flow
1. **File Retrieval (`GET /<sha256>`)**: nginx serves directly from disk
2. **File Upload (`PUT /upload`)**: nginx calls ginxsom via FastCGI for authentication + storage
3. **File Management (`DELETE`, `LIST`)**: nginx calls ginxsom via FastCGI
4. **Metadata Operations**: ginxsom manages SQLite database
## Blossom Protocol Support
ginxsom implements the following Blossom Upgrade Documents (BUDs):
- **BUD-01**: Server requirements and blob retrieval ✅
- **BUD-02**: Blob upload and management ✅
- **BUD-06**: Upload requirements ✅
### Supported Endpoints
| Endpoint | Method | Description | Handler |
|----------|---------|-------------|---------|
| `/<sha256>` | GET | Retrieve blob | nginx → disk |
| `/<sha256>` | HEAD | Check blob exists | nginx → disk |
| `/upload` | PUT | Upload new blob | nginx → FastCGI ginxsom |
| `/upload` | HEAD | Check upload requirements | nginx → FastCGI ginxsom |
| `/list/<pubkey>` | GET | List user's blobs | nginx → FastCGI ginxsom |
| `/<sha256>` | DELETE | Delete blob | nginx → FastCGI ginxsom |
## Installation
### Prerequisites
```bash
# Ubuntu/Debian
sudo apt-get update
sudo apt-get install build-essential libssl-dev libsqlite3-dev libfcgi-dev pkg-config
# RHEL/CentOS/Fedora
sudo dnf install gcc make openssl-devel sqlite-devel fcgi-devel pkgconfig
# macOS
brew install openssl sqlite fcgi pkg-config
```
### Building ginxsom
```bash
git clone https://github.com/yourusername/ginxsom.git
cd ginxsom
make
```
### Installation
```bash
sudo make install
# Installs to /usr/local/bin/ginxsom
```
## Configuration
### ginxsom Configuration
Create `/etc/ginxsom/config.conf`:
```ini
[server]
socket_path = /run/ginxsom.sock
max_file_size = 104857600 # 100MB
storage_path = /var/lib/ginxsom/blobs
database_path = /var/lib/ginxsom/blobs.db
[auth]
require_auth_upload = true
require_auth_get = false
require_auth_list = false
[limits]
max_blobs_per_user = 1000
rate_limit_uploads = 10 # per minute
```
### nginx Configuration
Add to your nginx configuration:
```nginx
server {
listen 80;
server_name your-blossom-server.com;
# CORS headers for all responses
add_header Access-Control-Allow-Origin *;
add_header Access-Control-Allow-Methods "GET, HEAD, PUT, DELETE, OPTIONS";
add_header Access-Control-Allow-Headers "Authorization, Content-Type, Content-Length, X-SHA-256, X-Content-Type, X-Content-Length";
add_header Access-Control-Max-Age 86400;
# Handle preflight OPTIONS requests
if ($request_method = 'OPTIONS') {
return 204;
}
# Static blob serving - nginx handles directly
location ~ ^/([a-f0-9]{64})(\.[a-zA-Z0-9]+)?$ {
root /var/lib/ginxsom/blobs;
try_files /$1$2 =404;
add_header Accept-Ranges bytes;
# Set content type based on file extension if not detected
location ~* \.(pdf)$ { add_header Content-Type application/pdf; }
location ~* \.(jpg|jpeg)$ { add_header Content-Type image/jpeg; }
location ~* \.(png)$ { add_header Content-Type image/png; }
location ~* \.(mp4)$ { add_header Content-Type video/mp4; }
}
# All other requests go to ginxsom FastCGI
location / {
include fastcgi_params;
fastcgi_pass unix:/run/ginxsom.sock;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
# Handle large uploads
client_max_body_size 100M;
fastcgi_request_buffering off;
}
}
```
## Usage
### Starting the Server
```bash
# Start ginxsom FastCGI daemon
sudo spawn-fcgi -s /run/ginxsom.sock -f /usr/local/bin/ginxsom -u www-data -g www-data
# Or with systemd
sudo systemctl enable ginxsom
sudo systemctl start ginxsom
```
### Testing
```bash
# Check server is running
curl -I http://localhost/upload
# Upload a file (requires nostr authorization)
curl -X PUT http://localhost/upload \
-H "Authorization: Nostr eyJ..." \
-H "Content-Type: application/pdf" \
--data-binary @document.pdf
# Retrieve a file
curl http://localhost/b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553.pdf
```
## API Documentation
### Authentication
All write operations require nostr event authentication via the `Authorization` header:
```
Authorization: Nostr <base64-encoded-nostr-event>
```
The nostr event must be kind `24242` with appropriate tags:
- `t` tag: `upload`, `delete`, `list`, or `get`
- `x` tag: SHA-256 hash for blob-specific operations
- `expiration` tag: Unix timestamp when event expires
### Blob Descriptors
Successful uploads return blob descriptors:
```json
{
"url": "https://cdn.example.com/b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553.pdf",
"sha256": "b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553",
"size": 184292,
"type": "application/pdf",
"uploaded": 1725105921
}
```
## File Storage
### Current (Flat) Structure
```
/var/lib/ginxsom/blobs/
├── b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553.pdf
├── a8472f6d93e42c1e5b4e9f3a7b2c8d4e6f9a1b3c5d7e8f0a1b2c3d4e5f6a7b8.png
└── ...
```
### Future (Optimized) Structure
```
/var/lib/ginxsom/blobs/
├── b1/
│ └── 67/
│ └── b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553.pdf
├── a8/
│ └── 47/
│ └── a8472f6d93e42c1e5b4e9f3a7b2c8d4e6f9a1b3c5d7e8f0a1b2c3d4e5f6a7b8.png
```
## Development
### Project Structure
```
ginxsom/
├── src/
│ ├── main.c # Main server loop
│ ├── config.c # Configuration parsing
│ ├── server.c # HTTP server logic
│ ├── auth.c # Nostr authentication
│ ├── storage.c # File storage operations
│ ├── database.c # SQLite operations
│ └── utils.c # Utility functions
├── include/
│ └── ginxsom.h # Main header file
├── tests/
│ └── test_*.c # Unit tests
├── docs/
│ └── api.md # API documentation
├── Makefile
└── README.md
```
### Building for Development
```bash
make debug # Build with debug symbols
make test # Run test suite
make clean # Clean build artifacts
```
### Dependencies
- **libssl**: For cryptographic operations (nostr event verification)
- **libsqlite3**: For metadata storage
- **libfcgi**: For FastCGI functionality
## Performance Characteristics
- **Static File Serving**: nginx performance (typically >10k req/s)
- **Upload Processing**: Limited by disk I/O and crypto verification
- **Memory Usage**: Minimal - FastCGI processes are lightweight
- **Concurrent Operations**: nginx manages FastCGI process pool efficiently
- **Process Management**: nginx spawns/manages ginxsom FastCGI processes
## Security Considerations
- All uploads require valid nostr event signatures
- SHA-256 verification prevents data corruption
- Rate limiting prevents abuse
- File size limits prevent disk exhaustion
- No script execution - blobs stored as static files
## Monitoring
ginxsom provides metrics via `/metrics` endpoint:
```
# HELP ginxsom_uploads_total Total number of uploads processed
# TYPE ginxsom_uploads_total counter
ginxsom_uploads_total 1234
# HELP ginxsom_storage_bytes Total bytes stored
# TYPE ginxsom_storage_bytes gauge
ginxsom_storage_bytes 1073741824
```
## License
[License information here]
## Contributing
[Contributing guidelines here]
---
**Note**: This project is under active development. Please report bugs and feature requests via GitHub issues.

259
SUBMODULES.md Normal file
View File

@@ -0,0 +1,259 @@
# Git Submodules Guide for Ginxsom
This project uses git submodules to include external dependencies that we reference but never modify.
## Current Submodules
- **blossom**: The official Blossom protocol specification and BUD documents
- **nostr_core_lib**: Core nostr functionality for authentication and event validation
## Key Principles
1. **Read-Only**: We never modify submodule content directly
2. **Version Pinning**: Submodules are pinned to specific commits for stability
3. **Upstream Updates**: We periodically update to newer versions when needed
---
## Common Git Submodule Commands
### Initial Setup (for new clones)
```bash
# Clone the main repository
git clone ssh://git@git.laantungir.net:222/laantungir/ginxsom.git
# Initialize and update all submodules
git submodule update --init --recursive
```
### One-Line Clone (includes submodules)
```bash
git clone --recursive ssh://git@git.laantungir.net:222/laantungir/ginxsom.git
```
### Update Submodules to Latest
```bash
# Update to latest commit on default branch
git submodule update --remote
# Update specific submodule
git submodule update --remote blossom
git submodule update --remote nostr_core_lib
```
### Pin Submodule to Specific Version
```bash
# Go into submodule directory
cd blossom
# Checkout specific commit/tag
git checkout v1.2.3 # or specific commit hash
# Go back to main repo
cd ..
# Stage the submodule change
git add blossom
# Commit the pin
git commit -m "Pin blossom submodule to v1.2.3"
```
### Check Submodule Status
```bash
# Show current submodule commits
git submodule status
# Show if submodules have uncommitted changes
git status
# Show submodule branch/commit info
git submodule foreach git log --oneline -1
```
---
## Workflow Best Practices
### Daily Development
1. **Never edit submodule files directly** - they're read-only references
2. **Use `git status` regularly** - it shows submodule state changes
3. **Commit submodule updates separately** - makes history cleaner
### When Submodule Updates Available
```bash
# Check what's new upstream
cd blossom
git fetch
git log HEAD..origin/main --oneline
cd ..
# If updates look good, update the pin
git submodule update --remote blossom
git add blossom
git commit -m "Update blossom submodule to latest"
```
### Team Collaboration
```bash
# After pulling main repo changes
git pull
# Update submodules if they changed
git submodule update --init --recursive
# Or use this one-liner for pulls
git pull --recurse-submodules
```
---
## Common Issues & Solutions
### "Submodule path contains unstaged changes"
```bash
# This means files in submodule were accidentally modified
cd <submodule_path>
git checkout . # Discard changes
cd ..
```
### "Submodule not initialized"
```bash
# Initialize and update
git submodule update --init <submodule_name>
```
### "Detached HEAD state in submodule"
```bash
# This is normal! Submodules are pinned to specific commits
# Only worry if you need to track a branch instead of a commit
```
### Accidentally Committed Changes in Submodule
```bash
# Go to submodule
cd <submodule_path>
# Reset to the pinned commit
git reset --hard <pinned_commit_hash>
cd ..
# The main repo should now show no changes to submodule
```
---
## Integration with Build System
### Makefile Integration
```makefile
# Ensure submodules are updated before building
build: update-submodules
# build commands here
update-submodules:
git submodule update --init --recursive
.PHONY: build update-submodules
```
### CI/CD Integration
```bash
# In your CI/CD pipeline, always initialize submodules
git submodule update --init --recursive
```
---
## Project-Specific Usage
### Blossom Submodule
- **Purpose**: Read BUD specifications and protocol documentation
- **Usage**: Reference only, never modify
- **Update frequency**: When new BUDs are released or existing ones updated
### nostr_core_lib Submodule
- **Purpose**: Core nostr functionality for ginxsom authentication
- **Usage**: Link against in build process, never modify source
- **Update frequency**: When bug fixes or new features needed
---
## Advanced Submodule Operations
### Change Submodule URL
```bash
# Edit .gitmodules file
vim .gitmodules
# Update git config
git submodule sync
# Update the submodule
git submodule update --init
```
### Remove a Submodule
```bash
# Remove from .gitmodules
git config -f .gitmodules --remove-section submodule.<name>
# Remove from .git/config
git config -f .git/config --remove-section submodule.<name>
# Remove directory
git rm --cached <submodule_path>
rm -rf <submodule_path>
# Commit changes
git add .gitmodules
git commit -m "Remove <name> submodule"
```
### Track a Branch Instead of Commit
```bash
# Edit .gitmodules to add branch
[submodule "blossom"]
path = blossom
url = ssh://git@git.laantungir.net:222/laantungir/blossom.git
branch = main
# Update to track branch
git submodule update --remote --merge
```
---
## Security Considerations
1. **Verify submodule sources** - only use trusted repositories
2. **Pin to specific commits** - avoid automatically tracking latest
3. **Review submodule updates** - check changes before updating pins
4. **Use SSH URLs** - more secure than HTTPS for private repos
---
## Troubleshooting Checklist
- [ ] Did you run `git submodule update --init --recursive` after cloning?
- [ ] Are you trying to edit files in submodule directories? (Don't!)
- [ ] Did you commit submodule changes to the main repository?
- [ ] Are your SSH keys set up for the submodule repositories?
- [ ] Is the submodule URL accessible from your current environment?
---
## Quick Reference
| Action | Command |
|--------|---------|
| Clone with submodules | `git clone --recursive <repo>` |
| Initialize after clone | `git submodule update --init --recursive` |
| Update all submodules | `git submodule update --remote` |
| Update one submodule | `git submodule update --remote <name>` |
| Check status | `git submodule status` |
| Reset submodule | `cd <sub> && git checkout . && cd ..` |
Remember: **Submodules are for code we use but never change!**

1
blossom Submodule

Submodule blossom added at e8d0a1ec44

149
include/ginxsom.h Normal file
View File

@@ -0,0 +1,149 @@
#ifndef GINXSOM_H
#define GINXSOM_H
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <stdint.h>
#include <stdbool.h>
#include <unistd.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <errno.h>
#include <fcgi_stdio.h>
#include <sqlite3.h>
#include <openssl/sha.h>
#include <openssl/evp.h>
#include <openssl/ec.h>
#include <openssl/obj_mac.h>
/* Constants */
#define GINXSOM_VERSION "0.1.0"
#define MAX_PATH_LEN 4096
#define MAX_CONFIG_LINE 1024
#define SHA256_HEX_LEN 64
#define SHA256_DIGEST_LEN 32
#define MAX_PUBKEY_HEX_LEN 64
#define MAX_MIME_TYPE_LEN 128
#define MAX_FILENAME_LEN 256
#define MAX_CONTENT_LEN 1024
#define DEFAULT_MAX_FILE_SIZE (100 * 1024 * 1024) // 100MB
#define DEFAULT_MAX_BLOBS_PER_USER 1000
#define DEFAULT_RATE_LIMIT 10
/* Error codes */
typedef enum {
GINXSOM_OK = 0,
GINXSOM_ERROR = -1,
GINXSOM_ERROR_CONFIG = -2,
GINXSOM_ERROR_DB = -3,
GINXSOM_ERROR_AUTH = -4,
GINXSOM_ERROR_STORAGE = -5,
GINXSOM_ERROR_MEMORY = -6,
GINXSOM_ERROR_INVALID_REQUEST = -7,
GINXSOM_ERROR_FILE_TOO_LARGE = -8,
GINXSOM_ERROR_RATE_LIMIT = -9,
GINXSOM_ERROR_NOT_FOUND = -10
} ginxsom_error_t;
/* Configuration structure */
typedef struct {
char socket_path[MAX_PATH_LEN];
size_t max_file_size;
char storage_path[MAX_PATH_LEN];
char database_path[MAX_PATH_LEN];
bool require_auth_upload;
bool require_auth_get;
bool require_auth_list;
int max_blobs_per_user;
int rate_limit_uploads;
} ginxsom_config_t;
/* Blob descriptor structure */
typedef struct {
char sha256[SHA256_HEX_LEN + 1];
char pubkey[MAX_PUBKEY_HEX_LEN + 1];
size_t size;
char mime_type[MAX_MIME_TYPE_LEN];
time_t uploaded;
char filename[MAX_FILENAME_LEN];
} ginxsom_blob_t;
/* Nostr event structure */
typedef struct {
char id[SHA256_HEX_LEN + 1];
char pubkey[MAX_PUBKEY_HEX_LEN + 1];
int kind;
char content[MAX_CONTENT_LEN];
time_t created_at;
time_t expiration;
char verb[16]; // t tag value: get, upload, list, delete
char x_hashes[10][SHA256_HEX_LEN + 1]; // x tag values (up to 10)
int x_count;
char server[MAX_PATH_LEN]; // server tag value
char signature[128];
} nostr_event_t;
/* FastCGI context structure */
typedef struct {
ginxsom_config_t config;
sqlite3 *db;
char *request_method;
char *request_uri;
char *content_type;
char *content_length_str;
size_t content_length;
char *authorization;
} ginxsom_context_t;
/* Function declarations */
/* config.c */
int ginxsom_load_config(const char *config_path, ginxsom_config_t *config);
void ginxsom_config_init_defaults(ginxsom_config_t *config);
/* database.c */
int ginxsom_db_init(sqlite3 **db, const char *db_path);
int ginxsom_db_close(sqlite3 *db);
int ginxsom_db_store_blob(sqlite3 *db, const ginxsom_blob_t *blob);
int ginxsom_db_delete_blob(sqlite3 *db, const char *sha256, const char *pubkey);
int ginxsom_db_list_blobs(sqlite3 *db, const char *pubkey, ginxsom_blob_t **blobs, int *count);
int ginxsom_db_blob_exists(sqlite3 *db, const char *sha256);
int ginxsom_db_get_user_blob_count(sqlite3 *db, const char *pubkey);
/* auth.c */
int ginxsom_parse_auth_header(const char *auth_header, nostr_event_t *event);
int ginxsom_verify_nostr_event(const nostr_event_t *event);
int ginxsom_validate_auth_for_upload(const nostr_event_t *event, const char *sha256);
int ginxsom_validate_auth_for_delete(const nostr_event_t *event, const char *sha256);
int ginxsom_validate_auth_for_list(const nostr_event_t *event);
/* storage.c */
int ginxsom_store_blob(const char *storage_path, const char *sha256,
const char *data, size_t size, const char *mime_type);
int ginxsom_delete_blob(const char *storage_path, const char *sha256);
char *ginxsom_detect_mime_type(const char *data, size_t size);
int ginxsom_calculate_sha256(const char *data, size_t size, char *sha256_hex);
/* server.c */
int ginxsom_handle_upload(ginxsom_context_t *ctx);
int ginxsom_handle_upload_head(ginxsom_context_t *ctx);
int ginxsom_handle_list(ginxsom_context_t *ctx, const char *pubkey);
int ginxsom_handle_delete(ginxsom_context_t *ctx, const char *sha256);
void ginxsom_send_response(int status, const char *content_type,
const char *body, size_t body_len);
void ginxsom_send_error(int status, const char *reason);
/* utils.c */
char *ginxsom_read_request_body(size_t content_length);
int ginxsom_hex_to_bytes(const char *hex, unsigned char *bytes, size_t bytes_len);
void ginxsom_bytes_to_hex(const unsigned char *bytes, size_t bytes_len, char *hex);
int ginxsom_parse_query_params(const char *query_string, char *params[][2], int max_params);
char *ginxsom_url_decode(const char *encoded);
int ginxsom_create_directory(const char *path);
#endif /* GINXSOM_H */

1
nostr_core_lib Submodule

Submodule nostr_core_lib added at 1da4f6751e