gitoria

22.9 KB

1. # gitoria.worldapi.org
3. Git hosting for all projects, written in **Hybriel** (hl:web), login via **ident**. Source of truth: `CONCEPT.md`.
4. Built so far: repos with their own address (#6), the repo homepage (#8), code browsing (#9), tickets (#10), pull requests (#11) with a Merge button for the owner (#17), releases (#12),
5. every repo view as its own server-rendered page (#16), push and pull over HTTPS with access tokens and over SSH with keys (#7).
7. ## Hybriel (vendored)
8. * `bin/hybriel` + `plugins/` (core crypto data fetch fs http http1 mpackdb proc time web) = hybriel **master 317d4754** (2026-09-26,
9. mission 038; includes 7cb9f8fc = #107: an emit in flight when its socket closes is carried over, a page being left starts nothing
10. — fixes the Firefox pull-back that rolled mission 037 back; also #103/#104, #105 `headers`, #106 `let` per loop pass, #94 files in
11. emit, #82 reconnect; before: #83 hashed `/__hl/…?v=` URLs + immutable cache, #88–#91 Bytes / chunked bodies / base64 / run stdin,
12. #95–#97, #100, #101, #45 hl:web, #80 `run()`, #81 `*path`, #44 `sessionDomain`), built from a read-only
13. `git archive master` (never inside Anton's repo) into `~/scratch-038/src`: `/media/STORAGE/projects/hybriel/native/zig-toolchain/zig
14. build -Doptimize=ReleaseFast -Dtarget=x86_64-linux-gnu.2.39` in `native/`, binary `native/zig-out/bin/hybriel`, sha256 `fc7481fb…48670a8`.
15. Older copies: `.scratch/pre-038/` (13ef4f9b bin/ + plugins/ + the pre-038 `browser.mjs`), `.scratch/pre-037/`, `.scratch/pre-035/`, `.scratch/pre-033*/`.
16. * **No local patch** (`grep -rn "LOCAL PATCH" plugins` finds nothing): a re-vendor is copy binary + plugins, run the gate.
17. The session cookie's `Domain` is the manifest setting `sessionDomain` (project.hl).
19. ## Run (dev, Loreana)
20. ```bash
21. cd /media/STORAGE/projects/gitoria.worldapi.org
22. GITORIA_PORT=8360 GITORIA_PUBLIC_URL=http://localhost:8360 ./bin/hybriel project.hl
23. ```
24. Config (environment, or a `.env` beside `project.hl` — never printed or committed):
26. | Variable | Default | |
27. |---|---|---|
28. | `GITORIA_PUBLIC_URL` | `https://gitoria.worldapi.org` | the main address; a repo lives at `<scheme>://<slug>.<host[:port] of this>` |
29. | `GITORIA_PORT` | 8360 | |
30. | `HL_HOST` | 0.0.0.0 | `127.0.0.1` on Byrodin behind nginx |
31. | `GITORIA_WATCH` | on | `0` = no dev watcher (the container) |
32. | `GITORIA_STORAGE` | `./storage/mpackdb` | table directory (`repos.db`, `users.db`) |
33. | `GITORIA_GIT` | `<launch dir>/storage/git` | the bare git repositories, `<slug>.git` each (absolute path; the `git` binary must be installed) |
34. | `GITORIA_TICKETS_URL` | `https://tickets.worldapi.org` | where a repo's tickets live (see "Tickets") |
35. | `GITORIA_SESSIONS` | `.sessions/` | |
36. | `GITORIA_COOKIE_DOMAIN` | `.<host of the public url>` | the session cookie's Domain (`-` = host-only; a host without a dot or an IP gets host-only) |
37. | `IDENT_URL`, `IDENT_EXCHANGE_URL`, `IDENT_API_KEY`, `IDENT_API_SECRET` | as in tickets | login via ident |
39. ## How a repo gets its address
40. * A repo is one row in `storage/mpackdb/repos.db` (`@id` key, unique index on `slug`): `{ slug, description, owner, created }`.
41. * **Slug**: unique in the whole system (one table, unique index); 2–40 characters `a–z 0–9 -`, starts and ends with a
42. letter/digit, no `--`, not one of the reserved names (`repos.hl` `reserved`: technical host names only — www, api, git, mail, …; app names like ident or tickets are allowed).
43. * **Address** = `<slug>.gitoria.worldapi.org`. nginx sends `gitoria.worldapi.org` and `*.gitoria.worldapi.org` to this one
44. app (wildcard vhost, wildcard DNS and certificate: the architect's).
45. * **Pages (gitoria#16)**: every page component declares `host = null` and hl:web hands it the request's host without port
46. (hybriel#74) — on the server, on the first load and on every hl:web navigation. `users.hl slugOfHost` → '' (main address)
47. or the slug. Routes (`project.hl`): `/` → `components/index.hl` (main address: `list.hl`, the repo list + "Create a repository";
48. repo address: `readme.hl`), `/code` `/code/*` → `code.hl`, `/branch/*` → `branch.hl`, `/commit/*` → `commit.hl` (all three
49. show `codebrowser.hl`), `/pulls` → `pulls.hl`, `/releases` → `releases.hl`, `/tickets` → `tickets.hl`, `/login/failed` →
50. `loginfailed.hl`. Each repo page starts with `repohead.hl` (name, description, nav; unknown address → "No such repository").
51. Links inside a repo are hl:web navigations (no page load). Rendered on the server with their content: the repo list,
52. the repo head, the Readme's address/owner/created, the Tickets list (hl:fetch answers at once).
53. * **Git on the server**: git is read with hl:proc `run()`, which waits for the program (hybriel#80), so the README, the file
54. list / file, branches, commits, pulls and releases are in the first HTML and in every hl:web navigation (`git.hl`
55. `homepageNow`, `browseNow`, `pullsNow`, `releasesNow`). No browser script is involved; `/host.js` is gone.
56. * **The path of `/code/*path`, `/branch/*path`, `/commit/*path`**: hl:web binds the rest of the URL to the page member `path`
57. (hybriel#81); the pages hand it to `codebrowser.hl`.
58. * **Login**: ident's login *button* flow (`<ident>/login?key=&return=<main>/login/callback`), the code exchanged server side.
59. * **Identity selector** (gitoria#14, as in tickets): ident's `<ident-selector>` sits beside the button in the header; choosing an identity
60. hands its one-time code (`/login.js` → hidden `#identcode` → face `gitoriaLogin`) to the server for the same exchange — no reload, open
61. tabs of the session follow. ident answers only a registered origin, so the selector shows on the main address only (the shell gives it
62. the class `onrepo` from the request's host, `styles.hl` hides it); the button stays the way in there.
63. The app is registered in ident with ONE origin, the main address. The session cookie carries `Domain=.gitoria.worldapi.org`
64. (hl:web `sessionDomain`, hybriel#44), so the login holds on every repo address. A login started at
65. `<slug>.…` returns through the main address and on to that repo (`?next=`, added by `/login.js` at the click: a path, or a full URL of `<valid-slug>.<host>`;
66. `project.hl` `safeNext`). The first login asks for a display name (shown as the repo's owner). A failed login redirects to
67. `/login/failed` (the reason parked in `session.data.loginError`).
68. * **Short ids (ident#23, mission 039)**: `users.identity` holds what ident's exchange answers — since ident#23 the identity's
69. public 5-character short id (`a68sz`), before that the per-app id (32 hex); `users.hl isIdentId` accepts both (the old
70. `isHex` check refused short ids). The switch: one-off `tools/migrate-short-ids.hl` (old → short id, idempotent, never
71. `finish`), gate `node tests/short-id-switch.mjs` (ports 8724/8725, no browser), runbook
72. `antcolony-docs/docs/short-id-switch.md` (Byrodin: `/CONTAINERS/projects/antcolony/docs/short-id-switch.md` once synced).
73. * **Create** (web form, face `gitoriaCreate`): any logged-in user with a display name. New repos reach every open list live.
74. * **API** (public reads): `GET /api/repos`, `GET /api/repos/:slug`; `Accept: text/markdown` gives the Markdown read view.
75. Creating is not in the API yet (needs API tokens — as in tickets `users.hl`).
76. * Creating a repo also runs `git init --bare -b main` in `<GITORIA_GIT>/<slug>.git` (`git.hl`); pushing to it: see "Push and pull".
78. ## Push and pull (gitoria#7)
79. Git over **HTTPS**, answered by this app itself with the `git` binary (`transport.hl`; design: `docs/git-backend.md`). SSH: see below.
80. * **Clone URL**: `https://<slug>.gitoria.worldapi.org/<slug>.git` (on the repo address, so the clone's folder is named like the repo). Paths
81. `/<slug>.git/info/refs?service=…`, `/<slug>.git/git-upload-pack`, `/<slug>.git/git-receive-pack` (function routes, after the page routes in `project.hl`).
82. * **Read** (clone, fetch, pull) needs no login: every repo is public. **Write** (push) needs the **access token of the repo's owner** as the
83. password (any user name); anybody else's token → 403, no/unknown token → 401 with the way to get one. The token check is in `gitTransport`,
84. before git is started.
85. * **Access tokens** (`tokens.hl`, `components/tokens.hl`, section `#tokens` of the main address for a logged-in user): name → token `gtr_` + 40 hex, shown ONCE;
86. only its sha256 is stored (`storage/mpackdb/tokens.db`); list, remove (works at once); at most 20 per user. Faces `gitoriaMakeToken`, `gitoriaRemoveToken`.
87. * **The "Add code to this repository" box** (`components/repohead.hl`, gitoria#20): plain text (no toggle), only on the
88. **Code page** of a repo that has no branch yet (`git.hl isEmptyNow`) — never on Readme / Pull requests / Releases /
89. Tickets / Settings, and never once there is a commit. The clone command, the commands for a new project and for an
90. existing one, and where the token comes from.
91. * **How the body gets to git**: hl:proc `run()` has no stdin, so the request body is written to `<GITORIA_GIT>/.tmp/<random>.in` (a String holds raw bytes;
92. hl:fs writes them exactly) and `sh -c 'git upload-pack|receive-pack --stateless-rpc "$1" < "$2" > "$3"'` (paths as arguments, no user text in the script)
93. writes the answer to `.out`, which is read back as the response; both files are removed. A gzip request is unpacked first. `Git-Protocol: version=…`
94. → `GIT_PROTOCOL` (v0, v1 and v2 tested). No hook: pulls and releases are read from the commits.
95. * **Behind nginx** (the architect's vhost): `client_max_body_size` must allow pushes (say `500m`) and `proxy_request_buffering` stays **on** (default).
96. git sends a big push chunked; hl:http1 reads chunked request bodies since hybriel#89 (mission 035: the old 411 hint is gone, a direct
97. client without proxy pushes too — gate-checked). `proxy_read_timeout` ≥ 300s. The whole body is held in memory (≤ 500 MB).
98. Kept on purpose (035): the temp files + `sh -c` (hl:proc `run()` could now take `stdin` + `binary`, hybriel#88/#91, but stdin
99. crosses the plugin ABI as hex = twice the memory for a ≤ 500 MB body, and gzip would still need a second program) and `base64 -d`
100. for the Basic header (hl:crypto `fromBase64(...).toString()` aborts the request on bytes that are not UTF-8).
101. * Test: `node tests/push.mjs` (own ident + server + Chrome + the real git client; a small proxy in the gate plays nginx).
103. ## Git over SSH (gitoria#7)
104. A small **sshd container** (`docker/sshd`, service `gitoria-sshd` in `docker-compose.yml`) whose only job is `git-upload-pack` / `git-receive-pack`. It keeps no user list:
105. * **Keys** (`sshkeys.hl`, `components/sshkeys.hl`, section `#sshkeys` of the main address for a logged-in user): paste a PUBLIC key + a name; checked with `ssh-keygen -l`
106. (ed25519, ecdsa, sk-…, RSA ≥ 2048; a private key, DSA, junk, a second copy of a key are refused); list with fingerprint; remove; ≤ 20 per user. Table `storage/mpackdb/sshkeys.db`.
107. Faces `gitoria{Add,Remove}Key`. A key works at once and stops at once (sshd asks again on every login).
108. * **Login**: `AuthorizedKeysCommand` (`gitoria-keys`) → `GET /__git/keys?type&key` (`sshgate.hl`) → `restrict,command="gitoria-shell <user id>" <key>`. The forced command
109. `gitoria-shell` accepts only `git-upload-pack|git-receive-pack '<slug>.git'` (slug validated, no shell, no forwarding, no tty), asks `GET /__git/access?user&slug&write`
110. and only then runs git on `/repos/<slug>.git`. Same rule as HTTPS: **read = everyone, push = the repo's owner**.
111. * **The two internal routes** exist only when `GITORIA_SSH_SECRET` is set; every call needs `X-Gitoria-Secret` = that secret, and a call that came through the public proxy
112. (`X-Forwarded-For` / `X-Real-IP`) is refused. **SSH is offered on the site only when the secret is set** (the keys section and the ssh commands in the "add code" box are hidden otherwise).
113. * **Deploy (the architect)**: `GITORIA_SSH_SECRET=<long random text>` (and optionally `GITORIA_SSH_PORT`, default 2222) in `.env` beside `docker-compose.yml` (both services read it);
114. open the port in the firewall (port 22 belongs to the host's sshd, hence 2222: address `ssh://[email protected]:2222/<slug>.git`; with `GITORIA_SSH_PORT=22` the box shows `[email protected]:<slug>.git`);
115. `gitoria.worldapi.org` (not only the wildcard) must resolve to Byrodin directly — **Cloudflare's proxy does not carry ssh** (use a grey-cloud/DNS-only record for the ssh host or a
116. separate name; the address in the box uses the site's host name). The Hybriel image needs `openssh-client` (Dockerfile). `./storage/git` is mounted into the sshd container; its `git` account takes
117. the uid of that folder's owner; host keys live in `storage/sshd-hostkeys/` (clients keep trusting the server). `docker compose up -d --build` builds both.
118. * Test: `node tests/ssh.mjs` (builds and starts the REAL sshd container on port 8708 with host networking; real ssh + git: clone, push 3 MB, RSA + ed25519 keys, pull, unknown key, no shell,
119. path tricks, no forwarding, another user may read not push, removed key stops at once). Needs docker.
121. ## The repo homepage (`/` of a repo address)
122. * The repo's **README.md** (root, any case) is shown as a page; if the repo has a **`$docs`** folder, **all Markdown files inside it**
123. (subfolders too, in path order, at most 30) form the homepage instead and the root README is not shown. Read from the
124. repo's `HEAD` (the branch setting comes with #9). No commit / no README → "This repository has no README.md yet."
125. * Reading = the `git` binary via hl:proc (`git.hl`: `ls-tree -r`, `cat-file blob HEAD:<path>`, argv list, paths only from git,
126. 15 s limit, ≤ 5000 lines a file),
127. read while the page is built on the server (`homepageNow`).
128. * Markdown → HTML (`markdown.hl` copied from tickets, plus tables, block quotes, rules; `components/markdown.hl`): built as
129. elements from parsed data, never an HTML string — raw HTML in a README is shown as text, only http(s)/mailto/`/…`/`#…` links
130. are links. Not rendered: images, nested lists (shown as typed).
132. ## Browsing code (`/code`, `/branch/<name>`, `/commit/<id>` of a repo address)
133. * **`/code`** = the repo's main branch at its last commit. Main branch = the owner's setting (repo field `branch`), else `main`,
134. else the first branch. The owner sets it on the code page: "Make main" beside each other branch (only the owner sees it; the
135. face checks the owner again). The homepage (README / `$docs`) reads the same main branch.
136. * **`/branch/<name>`** = that branch at its last commit (a name with slashes works: the longest existing branch name wins).
137. **`/commit/<id>`** = the whole project at that commit (`<id>` = 4–40 hex characters, resolved to the full id).
138. * After each of them a path: `/code/src/a.txt`, `/branch/feature/x/src`, `/commit/<id>/src`. A folder shows its entries
139. (folders first, then files, with size), a file its numbered lines (at most 2000 lines, at most 1 MB). Also on the page: the
140. crumbs, the latest commit, all branches, the latest 20 commits (each links to `/commit/<id>`).
141. * Read with the `git` binary (`git.hl` `browse`: `for-each-ref`, `log`, `cat-file`, `ls-tree`, argv lists, `--literal-pathspecs`).
142. Read on the server while the page is built (`browseNow`). `/code`, `/branch/*`, `/commit/*` are three pages sharing
143. `codebrowser.hl`; a link inside the app is an hl:web navigation. "Make main" (face `gitoriaSetBranch`) answers with the view again.
144. * **Only UTF-8 text is shown**: a binary file or one that is not valid UTF-8 says so instead (its bytes would break the
145. page's socket). Paths with `:`, quotes, backslashes or control characters are not browsable (`git.hl` `safePath`).
146. * Not built: the Markdown read view / API of code, a diff of a commit, images, syntax highlighting, downloading a tree.
148. ## Tickets (`/tickets` of a repo address)
149. * The tickets are **not stored in gitoria**: they live in tickets.worldapi.org, in a project named `<slug>.<host of GITORIA_PUBLIC_URL>`
150. (e.g. `myrepo.gitoria.worldapi.org`; a repo slug has no dot, so it never meets another repo's project or the dotted app projects).
151. Anyone logged in to tickets sees them like any tickets project. `tickets.hl` talks to tickets' public API.
152. * **List**: `GET <tickets>/api/projects/<project>/tickets` (public), newest update first, at most 200 shown: number, subject (as text), state,
153. last update; each links to the ticket in tickets (read, comment and change the state there — the list view only is in gitoria).
154. No project yet (404) → "No ticket yet". Tickets down → "tickets.worldapi.org did not answer". Read when the page is built on
155. the server (hl:fetch is synchronous): the list is in the first HTML. A ticket opened here shows up live on every open tickets page of that repo.
156. * **Connect (gitoria#18)**: the repo's **settings** (`/settings`, owner only) has "Tickets: connect". It sends the owner to
157. `<tickets>/connect?app=gitoria&label=<slug>&return=<repo address>/settings/connected&state=<nonce>` (tickets asks which project they are
158. admin of); tickets returns `?code&state`; gitoria checks the nonce (parked in the session, bound to the repo), exchanges the code from
159. the server (`POST /api/connect/exchange`) and stores the key per repo in `repos.db` (`tktKey`, never sent to a page). "Forget the
160. connection" clears it locally (tickets can also disconnect). Not connected → `/tickets` says so and points to Settings.
161. `GITORIA_TICKETS_TOKEN` and the auto-created `<slug>.<host>` project are gone (they were the old way).
162. * **Open a ticket**: any logged-in user with a display name: `POST <api>/tickets` with `Authorization: Bearer <key>` and
163. `X-Tickets-Identity: <the user's ident id>` — the person is the author in tickets, under the project's roles (they must have logged
164. in to tickets once).
165. * **`#N` links both ways**: `#12` in a commit subject (code page, latest commits) or a pull request title links ticket 12. A push
166. (and the Merge button) runs `notifyPush`: every commit of the last 100 on any branch whose subject names `#N` posts a comment
167. "Mentioned in commit …" on ticket N (once per commit and ticket, remembered in `ticketlinks.db`), and a **merged** `|||PR` whose
168. title says `fixes|closes|resolves #N` sets ticket N to state `review` with a comment. Done as the repo's owner. Commits that
169. exist when the repo is connected are only marked, not announced.
170. * Not built: showing a ticket's text / comments inside gitoria, editing a ticket from gitoria, a state filter, API of gitoria for tickets.
172. ## Pull requests (`/pulls` of a repo address)
173. * Nothing is stored: the list is read from git each time (`git.hl` `pulls`). A **pull request is a commit** whose subject starts
174. `|||PR ` (target = the repo's main branch, i.e. the owner's setting, else `main`) or `|||PR|<branch>] ` (target = `<branch>`).
175. Anything else (marker not at the start, no space after `]`, an invalid branch name) is a normal commit.
176. * Title = the text after the marker (empty → the source branch's name). Source = the branch that holds the commit and is not the target
177. (`for-each-ref --contains`); one request per source branch (its newest marker commit); 50 at most, from the latest 500 commits of all branches.
178. * State: **merged** when the commit is in the target branch (`merge-base --is-ancestor`), else **open**; a target that does not exist is
179. shown as "(no such branch)". Merging is done with git itself (push to the target) — no merge button yet.
180. * Read on the server while the page is built (`pullsNow`).
182. ## Releases (`/releases` of a repo address)
183. * Nothing is stored: read from git each time (`git.hl` `releases`). A **release is a commit on the repo's main branch** whose subject starts
184. `|||RL ` (patch +1), `|||RL|med ` (minor +1, patch 0), `|||RL|mj ` (major +1, minor and patch 0) or `|||RL|<version> ` (set by hand,
185. `1.1.1a`: three numbers, then letters/digits/`.`/`-`). `|||RL` alone counts as `|||RL `. Anything else is a normal commit.
186. * Versions are counted from 0.0.0 over the main branch's history, oldest to newest (first `|||RL ` = 0.0.1); after a hand-set version the count
187. goes on from its numbers. A hand-set version already released is not a release. Shown newest first (200 at most, latest marked): version, text
188. after the marker (else the short commit id; links to `/commit/<id>`), author, date. Read on the server while the page is built (`releasesNow`).
189. * A release is only that: version + commit. No git tag is written, no archive to download (the concept does not say what else it holds).
191. ## Test
192. `node tests/browser.mjs` (149 checks; commits real files into the gate's bare repos) — own gitoria + own ident + own tickets (copies without `.env`, mail sink; `tests/ticketskit.mjs`) + headless Chromes; `*.gitoria.test`
193. is mapped to 127.0.0.1 inside Chrome (`HL_CHROME_ARGS`, `tests/cdp.mjs`). Ports 8700–8709 (gitoria 8700, ident 8701, tickets 8702, Chromes 8703–8709); another range:
194. `GITORIA_GATE_PORT=8710 GITORIA_GATE_IDENT_PORT=8711 GITORIA_GATE_TICKETS_PORT=8712 GITORIA_GATE_CHROME=8713-8719 node tests/browser.mjs`. Screenshots in `.scratch/gate-*.png`, server log `.scratch/gate-server.log`.
195. * gitoria#16 block: `firstHtml(path, host)` fetches the FIRST HTML with a Host header (node's fetch cannot set one) — `/` (README), /code,
196. /code/src, a file, /branch/main, /branch/feature/x, /commit/<id>, /pulls, /releases, /tickets hold their real content and no "Loading";
197. the main address the repo list; hybriel#43: a code view in a second tab of the session keeps its elements through a login and a
198. logout of the other tab; every view of an unknown address "No such repository"; `/host.js` 404; in Chrome every view loads
199. directly, and Readme → Code → Releases → Tickets → Pulls → Readme plus a folder + Back keep `window.__navMarker` (no page load).
200. * Re-vendor block: `/__hl/app.css` has the token file's `--dark` in `:root` (hybriel#39); a repo description
201. `</script><b id="xss">…` stays inside the seed (`\u003c`) and shows as text (hybriel#34); `Domain=.gitoria.test` on the cookie (`sessionDomain`).
202. * Navigation LOGGED IN (the creator saw full page loads in Firefox): a fresh Chrome logs in with the button on a repo address, then
203. real clicks Readme → Code → Releases → Pulls → Tickets → Code → a folder keep `window.__navMarker`; the same on an empty repo the user
204. owns, and with the WebSocket closed (POST fallback). The same two runs in a **real Firefox** (`tests/firefox.mjs`: headless
205. `/usr/bin/firefox` over WebDriver BiDi, no driver/npm; hosts mapped with the pref `network.dns.localDomains`; BiDi port
206. `GITORIA_GATE_FIREFOX`, default 8699).
207. * By hand: `curl -s -H 'Host: <slug>.gitoria.test:8720' http://127.0.0.1:8720/code` against a running dev server.
208. * Lesson: the views are in the FIRST HTML now, so "content is there" no longer means "page is live" — `await hydrated(page)` before
209. clicking a button with a handler ("Make main" was clicked on the dead SSR page and flaked).
211. ## Deploy
212. `./deploy.sh` (gates → rsync → restart → 200). First deploy = the architect's: folder, `.env`, wildcard vhost
213. (`server_name gitoria.worldapi.org *.gitoria.worldapi.org;`, WebSocket upgrade headers), wildcard DNS + certificate, and the app
214. registered in ident with origin `https://gitoria.worldapi.org`. Container port 45004 (`docker-compose.yml`).