gitoria

All repositories: gitoria

Readme Code Pull requests Releases Tickets Settings

Commitfd981932

fd981932State of 2026-09-27; bin/ no longer tracked (Hybriel commit is in README)mre2026-09-27 12:20

fd981932/README.md

22.8 KB

# gitoria.worldapi.org
Git hosting for all projects, written in **Hybriel** (hl:web), login via **ident**. Source of truth: `CONCEPT.md`.
Built so far: repos with their own address (#6), the repo homepage (#8), code browsing (#9), tickets (#10), pull requests (#11) with a Merge button for the owner (#17), releases (#12),
every repo view as its own server-rendered page (#16), push and pull over HTTPS with access tokens and over SSH with keys (#7).
## Hybriel (vendored)
* `bin/hybriel` + `plugins/` (core crypto data fetch fs http http1 mpackdb proc time web) = hybriel **master 317d4754** (2026-09-26,
mission 038; includes 7cb9f8fc = #107: an emit in flight when its socket closes is carried over, a page being left starts nothing
— fixes the Firefox pull-back that rolled mission 037 back; also #103/#104, #105 `headers`, #106 `let` per loop pass, #94 files in
emit, #82 reconnect; before: #83 hashed `/__hl/…?v=` URLs + immutable cache, #88–#91 Bytes / chunked bodies / base64 / run stdin,
#95–#97, #100, #101, #45 hl:web, #80 `run()`, #81 `*path`, #44 `sessionDomain`), built from a read-only
`git archive master` (never inside Anton's repo) into `~/scratch-038/src`: `/media/STORAGE/projects/hybriel/native/zig-toolchain/zig
build -Doptimize=ReleaseFast -Dtarget=x86_64-linux-gnu.2.39` in `native/`, binary `native/zig-out/bin/hybriel`, sha256 `fc7481fb…48670a8`.
Older copies: `.scratch/pre-038/` (13ef4f9b bin/ + plugins/ + the pre-038 `browser.mjs`), `.scratch/pre-037/`, `.scratch/pre-035/`, `.scratch/pre-033*/`.
* **No local patch** (`grep -rn "LOCAL PATCH" plugins` finds nothing): a re-vendor is copy binary + plugins, run the gate.
The session cookie's `Domain` is the manifest setting `sessionDomain` (project.hl).
## Run (dev, Loreana)
```bash
cd /media/STORAGE/projects/gitoria.worldapi.org
GITORIA_PORT=8360 GITORIA_PUBLIC_URL=http://localhost:8360 ./bin/hybriel project.hl
```
Config (environment, or a `.env` beside `project.hl` — never printed or committed):
| Variable | Default | |
|---|---|---|
| `GITORIA_PUBLIC_URL` | `https://gitoria.worldapi.org` | the main address; a repo lives at `<scheme>://<slug>.<host[:port] of this>` |
| `GITORIA_PORT` | 8360 | |
| `HL_HOST` | 0.0.0.0 | `127.0.0.1` on Byrodin behind nginx |
| `GITORIA_WATCH` | on | `0` = no dev watcher (the container) |
| `GITORIA_STORAGE` | `./storage/mpackdb` | table directory (`repos.db`, `users.db`) |
| `GITORIA_GIT` | `<launch dir>/storage/git` | the bare git repositories, `<slug>.git` each (absolute path; the `git` binary must be installed) |
| `GITORIA_TICKETS_URL` | `https://tickets.worldapi.org` | where a repo's tickets live (see "Tickets") |
| `GITORIA_SESSIONS` | `.sessions/` | |
| `GITORIA_COOKIE_DOMAIN` | `.<host of the public url>` | the session cookie's Domain (`-` = host-only; a host without a dot or an IP gets host-only) |
| `IDENT_URL`, `IDENT_EXCHANGE_URL`, `IDENT_API_KEY`, `IDENT_API_SECRET` | as in tickets | login via ident |
## How a repo gets its address
* A repo is one row in `storage/mpackdb/repos.db` (`@id` key, unique index on `slug`): `{ slug, description, owner, created }`.
* **Slug**: unique in the whole system (one table, unique index); 2–40 characters `a–z 0–9 -`, starts and ends with a
letter/digit, no `--`, not one of the reserved names (`repos.hl` `reserved`: technical host names only — www, api, git, mail, …; app names like ident or tickets are allowed).
* **Address** = `<slug>.gitoria.worldapi.org`. nginx sends `gitoria.worldapi.org` and `*.gitoria.worldapi.org` to this one
app (wildcard vhost, wildcard DNS and certificate: the architect's).
* **Pages (gitoria#16)**: every page component declares `host = null` and hl:web hands it the request's host without port
(hybriel#74) — on the server, on the first load and on every hl:web navigation. `users.hl slugOfHost` → '' (main address)
or the slug. Routes (`project.hl`): `/` → `components/index.hl` (main address: `list.hl`, the repo list + "Create a repository";
repo address: `readme.hl`), `/code` `/code/*` → `code.hl`, `/branch/*` → `branch.hl`, `/commit/*` → `commit.hl` (all three
show `codebrowser.hl`), `/pulls` → `pulls.hl`, `/releases` → `releases.hl`, `/tickets` → `tickets.hl`, `/login/failed` →
`loginfailed.hl`. Each repo page starts with `repohead.hl` (name, description, nav; unknown address → "No such repository").
Links inside a repo are hl:web navigations (no page load). Rendered on the server with their content: the repo list,
the repo head, the Readme's address/owner/created, the Tickets list (hl:fetch answers at once).
* **Git on the server**: git is read with hl:proc `run()`, which waits for the program (hybriel#80), so the README, the file
list / file, branches, commits, pulls and releases are in the first HTML and in every hl:web navigation (`git.hl`
`homepageNow`, `browseNow`, `pullsNow`, `releasesNow`). No browser script is involved; `/host.js` is gone.
* **The path of `/code/*path`, `/branch/*path`, `/commit/*path`**: hl:web binds the rest of the URL to the page member `path`
(hybriel#81); the pages hand it to `codebrowser.hl`.
* **Login**: ident's login *button* flow (`<ident>/login?key=&return=<main>/login/callback`), the code exchanged server side.
* **Identity selector** (gitoria#14, as in tickets): ident's `<ident-selector>` sits beside the button in the header; choosing an identity
hands its one-time code (`/login.js` → hidden `#identcode` → face `gitoriaLogin`) to the server for the same exchange — no reload, open
tabs of the session follow. ident answers only a registered origin, so the selector shows on the main address only (the shell gives it
the class `onrepo` from the request's host, `styles.hl` hides it); the button stays the way in there.
The app is registered in ident with ONE origin, the main address. The session cookie carries `Domain=.gitoria.worldapi.org`
(hl:web `sessionDomain`, hybriel#44), so the login holds on every repo address. A login started at
`<slug>.…` returns through the main address and on to that repo (`?next=`, added by `/login.js` at the click: a path, or a full URL of `<valid-slug>.<host>`;
`project.hl` `safeNext`). The first login asks for a display name (shown as the repo's owner). A failed login redirects to
`/login/failed` (the reason parked in `session.data.loginError`).
* **Short ids (ident#23, mission 039)**: `users.identity` holds what ident's exchange answers — since ident#23 the identity's
public 5-character short id (`a68sz`), before that the per-app id (32 hex); `users.hl isIdentId` accepts both (the old
`isHex` check refused short ids). The switch: one-off `tools/migrate-short-ids.hl` (old → short id, idempotent, never
`finish`), gate `node tests/short-id-switch.mjs` (ports 8724/8725, no browser), runbook
`antcolony-docs/docs/short-id-switch.md` (Byrodin: `/CONTAINERS/projects/antcolony/docs/short-id-switch.md` once synced).
* **Create** (web form, face `gitoriaCreate`): any logged-in user with a display name. New repos reach every open list live.
* **API** (public reads): `GET /api/repos`, `GET /api/repos/:slug`; `Accept: text/markdown` gives the Markdown read view.
Creating is not in the API yet (needs API tokens — as in tickets `users.hl`).
* Creating a repo also runs `git init --bare -b main` in `<GITORIA_GIT>/<slug>.git` (`git.hl`); pushing to it: see "Push and pull".
## Push and pull (gitoria#7)
Git over **HTTPS**, answered by this app itself with the `git` binary (`transport.hl`; design: `docs/git-backend.md`). SSH: see below.
* **Clone URL**: `https://<slug>.gitoria.worldapi.org/<slug>.git` (on the repo address, so the clone's folder is named like the repo). Paths
`/<slug>.git/info/refs?service=…`, `/<slug>.git/git-upload-pack`, `/<slug>.git/git-receive-pack` (function routes, after the page routes in `project.hl`).
* **Read** (clone, fetch, pull) needs no login: every repo is public. **Write** (push) needs the **access token of the repo's owner** as the
password (any user name); anybody else's token → 403, no/unknown token → 401 with the way to get one. The token check is in `gitTransport`,
before git is started.
* **Access tokens** (`tokens.hl`, `components/tokens.hl`, section `#tokens` of the main address for a logged-in user): name → token `gtr_` + 40 hex, shown ONCE;
only its sha256 is stored (`storage/mpackdb/tokens.db`); list, remove (works at once); at most 20 per user. Faces `gitoriaMakeToken`, `gitoriaRemoveToken`.
* **The "Add code to this repository" box** (`components/repohead.hl`, on EVERY repo page, in the first HTML): the clone command, the commands for a
new project and for an existing one, and where the token comes from; open by itself while the repo has no branch (`git.hl isEmptyNow`), else closed.
* **How the body gets to git**: hl:proc `run()` has no stdin, so the request body is written to `<GITORIA_GIT>/.tmp/<random>.in` (a String holds raw bytes;
hl:fs writes them exactly) and `sh -c 'git upload-pack|receive-pack --stateless-rpc "$1" < "$2" > "$3"'` (paths as arguments, no user text in the script)
writes the answer to `.out`, which is read back as the response; both files are removed. A gzip request is unpacked first. `Git-Protocol: version=…`
→ `GIT_PROTOCOL` (v0, v1 and v2 tested). No hook: pulls and releases are read from the commits.
* **Behind nginx** (the architect's vhost): `client_max_body_size` must allow pushes (say `500m`) and `proxy_request_buffering` stays **on** (default).
git sends a big push chunked; hl:http1 reads chunked request bodies since hybriel#89 (mission 035: the old 411 hint is gone, a direct
client without proxy pushes too — gate-checked). `proxy_read_timeout` ≥ 300s. The whole body is held in memory (≤ 500 MB).
Kept on purpose (035): the temp files + `sh -c` (hl:proc `run()` could now take `stdin` + `binary`, hybriel#88/#91, but stdin
crosses the plugin ABI as hex = twice the memory for a ≤ 500 MB body, and gzip would still need a second program) and `base64 -d`
for the Basic header (hl:crypto `fromBase64(...).toString()` aborts the request on bytes that are not UTF-8).
* Test: `node tests/push.mjs` (own ident + server + Chrome + the real git client; a small proxy in the gate plays nginx).
## Git over SSH (gitoria#7)
A small **sshd container** (`docker/sshd`, service `gitoria-sshd` in `docker-compose.yml`) whose only job is `git-upload-pack` / `git-receive-pack`. It keeps no user list:
* **Keys** (`sshkeys.hl`, `components/sshkeys.hl`, section `#sshkeys` of the main address for a logged-in user): paste a PUBLIC key + a name; checked with `ssh-keygen -l`
(ed25519, ecdsa, sk-…, RSA ≥ 2048; a private key, DSA, junk, a second copy of a key are refused); list with fingerprint; remove; ≤ 20 per user. Table `storage/mpackdb/sshkeys.db`.
Faces `gitoria{Add,Remove}Key`. A key works at once and stops at once (sshd asks again on every login).
* **Login**: `AuthorizedKeysCommand` (`gitoria-keys`) → `GET /__git/keys?type&key` (`sshgate.hl`) → `restrict,command="gitoria-shell <user id>" <key>`. The forced command
`gitoria-shell` accepts only `git-upload-pack|git-receive-pack '<slug>.git'` (slug validated, no shell, no forwarding, no tty), asks `GET /__git/access?user&slug&write`
and only then runs git on `/repos/<slug>.git`. Same rule as HTTPS: **read = everyone, push = the repo's owner**.
* **The two internal routes** exist only when `GITORIA_SSH_SECRET` is set; every call needs `X-Gitoria-Secret` = that secret, and a call that came through the public proxy
(`X-Forwarded-For` / `X-Real-IP`) is refused. **SSH is offered on the site only when the secret is set** (the keys section and the ssh commands in the "add code" box are hidden otherwise).
* **Deploy (the architect)**: `GITORIA_SSH_SECRET=<long random text>` (and optionally `GITORIA_SSH_PORT`, default 2222) in `.env` beside `docker-compose.yml` (both services read it);
open the port in the firewall (port 22 belongs to the host's sshd, hence 2222: address `ssh://[email protected]:2222/<slug>.git`; with `GITORIA_SSH_PORT=22` the box shows `[email protected]:<slug>.git`);
`gitoria.worldapi.org` (not only the wildcard) must resolve to Byrodin directly — **Cloudflare's proxy does not carry ssh** (use a grey-cloud/DNS-only record for the ssh host or a
separate name; the address in the box uses the site's host name). The Hybriel image needs `openssh-client` (Dockerfile). `./storage/git` is mounted into the sshd container; its `git` account takes
the uid of that folder's owner; host keys live in `storage/sshd-hostkeys/` (clients keep trusting the server). `docker compose up -d --build` builds both.
* Test: `node tests/ssh.mjs` (builds and starts the REAL sshd container on port 8708 with host networking; real ssh + git: clone, push 3 MB, RSA + ed25519 keys, pull, unknown key, no shell,
path tricks, no forwarding, another user may read not push, removed key stops at once). Needs docker.
## The repo homepage (`/` of a repo address)
* The repo's **README.md** (root, any case) is shown as a page; if the repo has a **`$docs`** folder, **all Markdown files inside it**
(subfolders too, in path order, at most 30) form the homepage instead and the root README is not shown. Read from the
repo's `HEAD` (the branch setting comes with #9). No commit / no README → "This repository has no README.md yet."
* Reading = the `git` binary via hl:proc (`git.hl`: `ls-tree -r`, `cat-file blob HEAD:<path>`, argv list, paths only from git,
15 s limit, ≤ 5000 lines a file),
read while the page is built on the server (`homepageNow`).
* Markdown → HTML (`markdown.hl` copied from tickets, plus tables, block quotes, rules; `components/markdown.hl`): built as
elements from parsed data, never an HTML string — raw HTML in a README is shown as text, only http(s)/mailto/`/…`/`#…` links
are links. Not rendered: images, nested lists (shown as typed).
## Browsing code (`/code`, `/branch/<name>`, `/commit/<id>` of a repo address)
* **`/code`** = the repo's main branch at its last commit. Main branch = the owner's setting (repo field `branch`), else `main`,
else the first branch. The owner sets it on the code page: "Make main" beside each other branch (only the owner sees it; the
face checks the owner again). The homepage (README / `$docs`) reads the same main branch.
* **`/branch/<name>`** = that branch at its last commit (a name with slashes works: the longest existing branch name wins).
**`/commit/<id>`** = the whole project at that commit (`<id>` = 4–40 hex characters, resolved to the full id).
* After each of them a path: `/code/src/a.txt`, `/branch/feature/x/src`, `/commit/<id>/src`. A folder shows its entries
(folders first, then files, with size), a file its numbered lines (at most 2000 lines, at most 1 MB). Also on the page: the
crumbs, the latest commit, all branches, the latest 20 commits (each links to `/commit/<id>`).
* Read with the `git` binary (`git.hl` `browse`: `for-each-ref`, `log`, `cat-file`, `ls-tree`, argv lists, `--literal-pathspecs`).
Read on the server while the page is built (`browseNow`). `/code`, `/branch/*`, `/commit/*` are three pages sharing
`codebrowser.hl`; a link inside the app is an hl:web navigation. "Make main" (face `gitoriaSetBranch`) answers with the view again.
* **Only UTF-8 text is shown**: a binary file or one that is not valid UTF-8 says so instead (its bytes would break the
page's socket). Paths with `:`, quotes, backslashes or control characters are not browsable (`git.hl` `safePath`).
* Not built: the Markdown read view / API of code, a diff of a commit, images, syntax highlighting, downloading a tree.
## Tickets (`/tickets` of a repo address)
* The tickets are **not stored in gitoria**: they live in tickets.worldapi.org, in a project named `<slug>.<host of GITORIA_PUBLIC_URL>`
(e.g. `myrepo.gitoria.worldapi.org`; a repo slug has no dot, so it never meets another repo's project or the dotted app projects).
Anyone logged in to tickets sees them like any tickets project. `tickets.hl` talks to tickets' public API.
* **List**: `GET <tickets>/api/projects/<project>/tickets` (public), newest update first, at most 200 shown: number, subject (as text), state,
last update; each links to the ticket in tickets (read, comment and change the state there — the list view only is in gitoria).
No project yet (404) → "No ticket yet". Tickets down → "tickets.worldapi.org did not answer". Read when the page is built on
the server (hl:fetch is synchronous): the list is in the first HTML. A ticket opened here shows up live on every open tickets page of that repo.
* **Connect (gitoria#18)**: the repo's **settings** (`/settings`, owner only) has "Tickets: connect". It sends the owner to
`<tickets>/connect?app=gitoria&label=<slug>&return=<repo address>/settings/connected&state=<nonce>` (tickets asks which project they are
admin of); tickets returns `?code&state`; gitoria checks the nonce (parked in the session, bound to the repo), exchanges the code from
the server (`POST /api/connect/exchange`) and stores the key per repo in `repos.db` (`tktKey`, never sent to a page). "Forget the
connection" clears it locally (tickets can also disconnect). Not connected → `/tickets` says so and points to Settings.
`GITORIA_TICKETS_TOKEN` and the auto-created `<slug>.<host>` project are gone (they were the old way).
* **Open a ticket**: any logged-in user with a display name: `POST <api>/tickets` with `Authorization: Bearer <key>` and
`X-Tickets-Identity: <the user's ident id>` — the person is the author in tickets, under the project's roles (they must have logged
in to tickets once).
* **`#N` links both ways**: `#12` in a commit subject (code page, latest commits) or a pull request title links ticket 12. A push
(and the Merge button) runs `notifyPush`: every commit of the last 100 on any branch whose subject names `#N` posts a comment
"Mentioned in commit …" on ticket N (once per commit and ticket, remembered in `ticketlinks.db`), and a **merged** `|||PR` whose
title says `fixes|closes|resolves #N` sets ticket N to state `review` with a comment. Done as the repo's owner. Commits that
exist when the repo is connected are only marked, not announced.
* Not built: showing a ticket's text / comments inside gitoria, editing a ticket from gitoria, a state filter, API of gitoria for tickets.
## Pull requests (`/pulls` of a repo address)
* Nothing is stored: the list is read from git each time (`git.hl` `pulls`). A **pull request is a commit** whose subject starts
`|||PR ` (target = the repo's main branch, i.e. the owner's setting, else `main`) or `|||PR|<branch>] ` (target = `<branch>`).
Anything else (marker not at the start, no space after `]`, an invalid branch name) is a normal commit.
* Title = the text after the marker (empty → the source branch's name). Source = the branch that holds the commit and is not the target
(`for-each-ref --contains`); one request per source branch (its newest marker commit); 50 at most, from the latest 500 commits of all branches.
* State: **merged** when the commit is in the target branch (`merge-base --is-ancestor`), else **open**; a target that does not exist is
shown as "(no such branch)". Merging is done with git itself (push to the target) — no merge button yet.
* Read on the server while the page is built (`pullsNow`).
## Releases (`/releases` of a repo address)
* Nothing is stored: read from git each time (`git.hl` `releases`). A **release is a commit on the repo's main branch** whose subject starts
`|||RL ` (patch +1), `|||RL|med ` (minor +1, patch 0), `|||RL|mj ` (major +1, minor and patch 0) or `|||RL|<version> ` (set by hand,
`1.1.1a`: three numbers, then letters/digits/`.`/`-`). `|||RL` alone counts as `|||RL `. Anything else is a normal commit.
* Versions are counted from 0.0.0 over the main branch's history, oldest to newest (first `|||RL ` = 0.0.1); after a hand-set version the count
goes on from its numbers. A hand-set version already released is not a release. Shown newest first (200 at most, latest marked): version, text
after the marker (else the short commit id; links to `/commit/<id>`), author, date. Read on the server while the page is built (`releasesNow`).
* A release is only that: version + commit. No git tag is written, no archive to download (the concept does not say what else it holds).
## Test
`node tests/browser.mjs` (149 checks; commits real files into the gate's bare repos) — own gitoria + own ident + own tickets (copies without `.env`, mail sink; `tests/ticketskit.mjs`) + headless Chromes; `*.gitoria.test`
is mapped to 127.0.0.1 inside Chrome (`HL_CHROME_ARGS`, `tests/cdp.mjs`). Ports 8700–8709 (gitoria 8700, ident 8701, tickets 8702, Chromes 8703–8709); another range:
`GITORIA_GATE_PORT=8710 GITORIA_GATE_IDENT_PORT=8711 GITORIA_GATE_TICKETS_PORT=8712 GITORIA_GATE_CHROME=8713-8719 node tests/browser.mjs`. Screenshots in `.scratch/gate-*.png`, server log `.scratch/gate-server.log`.
* gitoria#16 block: `firstHtml(path, host)` fetches the FIRST HTML with a Host header (node's fetch cannot set one) — `/` (README), /code,
/code/src, a file, /branch/main, /branch/feature/x, /commit/<id>, /pulls, /releases, /tickets hold their real content and no "Loading";
the main address the repo list; hybriel#43: a code view in a second tab of the session keeps its elements through a login and a
logout of the other tab; every view of an unknown address "No such repository"; `/host.js` 404; in Chrome every view loads
directly, and Readme → Code → Releases → Tickets → Pulls → Readme plus a folder + Back keep `window.__navMarker` (no page load).
* Re-vendor block: `/__hl/app.css` has the token file's `--dark` in `:root` (hybriel#39); a repo description
`</script><b id="xss">…` stays inside the seed (`\u003c`) and shows as text (hybriel#34); `Domain=.gitoria.test` on the cookie (`sessionDomain`).
* Navigation LOGGED IN (the creator saw full page loads in Firefox): a fresh Chrome logs in with the button on a repo address, then
real clicks Readme → Code → Releases → Pulls → Tickets → Code → a folder keep `window.__navMarker`; the same on an empty repo the user
owns, and with the WebSocket closed (POST fallback). The same two runs in a **real Firefox** (`tests/firefox.mjs`: headless
`/usr/bin/firefox` over WebDriver BiDi, no driver/npm; hosts mapped with the pref `network.dns.localDomains`; BiDi port
`GITORIA_GATE_FIREFOX`, default 8699).
* By hand: `curl -s -H 'Host: <slug>.gitoria.test:8720' http://127.0.0.1:8720/code` against a running dev server.
* Lesson: the views are in the FIRST HTML now, so "content is there" no longer means "page is live" — `await hydrated(page)` before
clicking a button with a handler ("Make main" was clicked on the dead SSR page and flaked).
## Deploy
`./deploy.sh` (gates → rsync → restart → 200). First deploy = the architect's: folder, `.env`, wildcard vhost
(`server_name gitoria.worldapi.org *.gitoria.worldapi.org;`, WebSocket upgrade headers), wildcard DNS + certificate, and the app
registered in ident with origin `https://gitoria.worldapi.org`. Container port 45004 (`docker-compose.yml`).

Branches

mainmain branch

Latest commits

fd981932State of 2026-09-27; bin/ no longer tracked (Hybriel commit is in README)mre2026-09-27 12:20
4a2d7125initial commitmre2026-09-25 19:10