Merge branch 'master' into chunked-blobs

2025-12-09 23:18:50 +00:00 · 2025-11-25 08:50:30 -06:00
parent 1d855e1633 42b6c568fc
commit d05228aae1
13 changed files with 654 additions and 201 deletions
--- a/buds/12.md
+++ b/buds/12.md
@@ -0,0 +1,116 @@
+# BUD-12
+
+## Chunked blobs
+
+`draft` `optional`
+
+Breaking large blobs into smaller chunks for distribution
+
+### Chunking
+
+The client MAY break large blobs into any number or size of chunks. although its recommended to use the size `1Mb` or `4Mb` for small and large chunks
+
+Clients MUST NOT pad the remaining chunk, If clients need privacy they should use random chunk sizes and optionally encrypt the large blob
+
+Clients MUST create a merkle tree using the chunk hashes as the leaf nodes
+
+### Publishing
+
+Clients should publish a `2001` kind event after chunking the file in order to store the list of chunks
+
+The events MUST contain an ordered list of `chunk` tags with the sha256 hashes of the chunks
+
+The `content` field MUST be a human readable description of the chunked file
+
+```json
+{
+  "tags": [
+    [ "chunk", "7e668b56a58c7891e0cf263ea3f093b75eebade23d663a45aa9920f347b3d671"],
+    [ "chunk", "9b9c44a91396f19fd8700986eb0586dff2dcccf96c75bc2caefef302bcd78da1"],
+    [ "chunk", "7a281548f1223664b855b10b08e59e84389ccabeb742517f6cd75eda2724a798"],
+    [ "chunk", "fadeccee86b123088bbc452df10e8fbc99d4c2f22a70ef7a35605ec8e439c345"],
+    [ "chunk", "5d62398419e6d136771541f3d2215e0ce31b1be45e99dbc64b43a4b734b447ca"],
+  ]
+}
+```
+
+The event MUST have an `x` tag with the sha256 hash of the ordered concatenated hashes of each chunk
+
+Example using `xxd` and `sha256sum`:
+```bash
+#!/bin/bash
+hashes=(
+  "7e668b56a58c7891e0cf263ea3f093b75eebade23d663a45aa9920f347b3d671"
+  "9b9c44a91396f19fd8700986eb0586dff2dcccf96c75bc2caefef302bcd78da1"
+  "7a281548f1223664b855b10b08e59e84389ccabeb742517f6cd75eda2724a798"
+  "fadeccee86b123088bbc452df10e8fbc99d4c2f22a70ef7a35605ec8e439c345"
+  "5d62398419e6d136771541f3d2215e0ce31b1be45e99dbc64b43a4b734b447ca"
+)
+
+concatenated=""
+for hex in "${hex_array[@]}"; do
+  concatenated="${concatenated}${hex}"
+done
+
+echo "Chunks:"
+printf '%s\n' "${hex_array[@]}"
+
+echo -e "\nConcatenated:"
+echo "$concatenated"
+
+echo -e "\nRoot hash:"
+echo "$concatenated" | xxd -r -p | sha256sum
+```
+
+The final `chunk` tags and `x` tag should look something like this
+
+```json
+{
+  "tags": [
+    [ "chunk", "7e668b56a58c7891e0cf263ea3f093b75eebade23d663a45aa9920f347b3d671"],
+    [ "chunk", "9b9c44a91396f19fd8700986eb0586dff2dcccf96c75bc2caefef302bcd78da1"],
+    [ "chunk", "7a281548f1223664b855b10b08e59e84389ccabeb742517f6cd75eda2724a798"],
+    [ "chunk", "fadeccee86b123088bbc452df10e8fbc99d4c2f22a70ef7a35605ec8e439c345"],
+    [ "chunk", "5d62398419e6d136771541f3d2215e0ce31b1be45e99dbc64b43a4b734b447ca"],
+    [ "x", "2d839865ac17d8bb10168490a88107637619f79dac21275fcec1705162581f39" ],
+  ]
+}
+```
+
+### Metadata
+
+The `2001` event MAY include additional metadata tags to help other clients know the filename, mime type or servers to download from
+
+Metadata tags:
+  - `name` Filename
+  - `mime` Mime type of file
+  - `size` Total size in bytes of the file
+  - `server` (multiple) Recommended servers to download chunks from
+
+### Examples
+
+Example `2001` event
+```json
+{
+  "id": "fd2f4da046851170576e7a09009e63f752dc469c61a24faca514b2cedfe641aa",
+  "pubkey": "5328e6c743a50271745e16476590ba7ea3dc591c65f3a5e2b03430814c1dabc0",
+  "created_at": 1731405194,
+  "kind": 2001,
+  "content": "Here is the original version",
+  "tags": [
+    [ "chunk", "7e668b56a58c7891e0cf263ea3f093b75eebade23d663a45aa9920f347b3d671"],
+    [ "chunk", "9b9c44a91396f19fd8700986eb0586dff2dcccf96c75bc2caefef302bcd78da1"],
+    [ "chunk", "7a281548f1223664b855b10b08e59e84389ccabeb742517f6cd75eda2724a798"],
+    [ "chunk", "fadeccee86b123088bbc452df10e8fbc99d4c2f22a70ef7a35605ec8e439c345"],
+    [ "chunk", "5d62398419e6d136771541f3d2215e0ce31b1be45e99dbc64b43a4b734b447ca"],
+    [ "x", "2d839865ac17d8bb10168490a88107637619f79dac21275fcec1705162581f39" ],
+    [ "name", "example.mp4" ],
+    [ "mime", "video/mp4" ],
+    [ "size", "4823449" ],
+    [ "server", "https://cdn.example.com" ],
+    [ "server", "https://nostr.download" ]
+  ],
+  "sig": "cd589a4ea702185c69d1fec93a7704265071b674c5708a1da04d08bcba873f5277bc0abb379cbb014852fafa2ac951ed49abbdba60028a4f02ef4e31ff2599cb"
+}
+```
+