gitoria

10.3 KB

1. # How Gitoria uses git — research
3. Research for ticket [gitoria.worldapi.org#5](https://tickets.worldapi.org/projects/gitoria.worldapi.org/5).
4. Blocks #7 (push and pull) and feeds #9 (browse code), #11 (pulls), #12 (releases).
5. Checked 2026-09-24 on Loreana (reworked 2026-09-25 after hybriel#42, stdin write and raw bytes in `hl:proc`, was fixed) with git 2.55.0, the vendored `bin/hybriel` and the plugin sources in
6. `hybriel/plugins/`.
8. ## Verdict
10. **Use the `git` binary. Do not build an `hl:git` plugin on a git library.**
12. - Every hub feature (clone/push over HTTPS and SSH, log, diff, blame, merge, archive, gc, grep) already exists in the
13. binary, is stable and is what every developer's client speaks. A library re-implements a subset.
14. - **libgit2** (installed on Loreana, 1.9) is a client/object library. It has **no server side**: no `upload-pack` /
15. `receive-pack`, so it cannot answer `git clone` / `git push`. That was the same gap as with nodegitserver.
16. Writing our own pack negotiation in Hybriel is a large, security-relevant project with no gain.
17. - Split by job:
18. 1. **Reading for the web UI** (#8 #9 #10 #11 #12): Hybriel spawns `git -C <repo>.git …` with `hl:proc` (argv list,
19. no shell). Works today, see "What hl:proc can and cannot do".
20. 2. **Transport (clone / fetch / push)**: **HTTPS is served by Hybriel itself**: the smart-HTTP endpoints spawn
21. `git upload-pack` / `git receive-pack --stateless-rpc` with `hl:proc` (`stdin = 'pipe'`, `binary = true`) and
22. pipe the request body in and the packfile out through `hl:http1` (byte-safe). **SSH** needs `sshd` (a Hybriel
23. process cannot be an ssh server): a small sshd container whose forced command is `git-upload-pack` /
24. `git-receive-pack`. Hybriel decides *who may*, git moves the bytes.
26. ## What was tested
28. | Claim | Result |
29. |---|---|
30. | (2026-09-24, kept as proof the protocol works) `git http-backend` as CGI behind a Basic-auth check serves clone **and push** to a bare repo (`http.receivepack=true`) | works: clone, push, second clone, wrong password → `Authentifizierung fehlgeschlagen` (test wrapper in Python, 30 lines) |
31. | `hl:http1` request and response bodies are byte-safe (packfiles) | works: 3000 random bytes POSTed and echoed, `cmp` identical |
32. | `hl:proc` output is text lines only (2026-09-24) | **was lossy; fixed by hybriel#42.** Retested 2026-09-25: `run([...'cat-file','blob'…], { binary = true })` returns `ff 00 0d 0a 65 6e 64` for a file `\xff \0 \r\n end` — byte exact, `.data` is a Bytes |
33. | `hl:proc` can write to the child's stdin | **yes now** (`{ stdin = 'pipe' }`, `write()` String or Bytes, `end()`). Retested: `git upload-pack --stateless-rpc r.git` fed a `want <sha>` + `done` request answers `0008NAK` and the packfile as Bytes, exit 0 |
34. | `hl:proc` can set cwd / env | **yes now** (`cwd`, `env` options, hybriel#37) — `git -C <dir>` still fine |
35. | SSH path (sshd `AuthorizedKeysCommand`) | designed, **not run** (no sshd on Loreana for this); standard OpenSSH feature |
37. Hybriel can now pipe a packfile through a child process, so "Hybriel itself answers `git upload-pack` / `receive-pack`"
38. works; no CGI wrapper and no separate HTTP transport container are needed. Only SSH still needs a sshd container.
39. Points found while retesting: a child's `stderr` also arrives as `data` events — keep only `stream == 'stdout'` for the
40. pack; `'\n'` in a string is not a newline (hybriel#44) — write a real newline when building pkt-lines.
42. ## Architecture
44. ```
45. developer ── https ──> nginx (*.gitoria.worldapi.org) ──> Hybriel (one app: web UI, API and git)
46. /repo.git/info/refs?service=… auth check, then hl:proc
47. git upload-pack|receive-pack --stateless-rpc --advertise-refs
48. /repo.git/git-upload-pack request body ─> child stdin, child stdout ─> response
49. /repo.git/git-receive-pack same; repos at repos/<slug>.git
50. developer ── ssh ──> sshd (small container, own port)
51. AuthorizedKeysCommand ─> Hybriel /__git/keys (key → user, may they?)
52. forced command: git-upload-pack / git-receive-pack '<slug>.git' only
53. git hooks in every bare repo: post-receive ─> Hybriel /__git/pushed (refs, old/new ids)
54. Hybriel ── hl:proc ──> git -C repos/<slug>.git log|show|ls-tree|diff-tree|blame|merge-tree|… (reads)
55. ```
57. - One volume `repos/` holds `<slug>.git` bare repositories (slug unique in the whole system, as in the concept).
58. Hybriel and the sshd container mount it; the mpackdb store stays in `storage/mpackdb/`.
59. - **Authorisation lives only in Hybriel** (ident login → users, repo access, keys, tokens). The sshd container
60. asks Hybriel over an internal address with a shared secret; it keeps no user list of its own.
61. - Hybriel reads the Basic credentials and the `service` of the request itself and answers "read" vs "write" before it
62. spawns git. Public repos: read without credentials, write always with.
63. - `post-receive` is how Hybriel learns about pushes: it scans new commits for `|||PR …` and `|||RL …` (#11, #12), fills
64. caches and can create the tickets project. Do this in the hook, not by polling.
65. - Not yet tried: a full clone and push through a running Hybriel HTTP endpoint (the child-process part is tested, the
66. endpoint is built in #7). Watch large pushes: stream the request body into the child, don't hold it whole in memory.
67. - Pass `Git-Protocol` to the child as env `GIT_PROTOCOL` (the `env` option) so clients use protocol v2.
69. ## Credentials — what git accepts today
71. - **HTTPS with user + password/token is not deprecated by git.** Git still uses HTTP Basic through its credential
72. helpers. (GitHub removed *account passwords* for its own service; that is not a git rule.) The dev CLI cannot do
73. an interactive ident login, so Gitoria issues **personal access tokens** (per user, named, revocable, stored hashed)
74. that go in the password field: `git clone https://slug.gitoria.worldapi.org/repo.git` → user name + token, stored by
75. `git credential-store`/`cache`/OS keychain. HTTPS only, never plain HTTP.
76. - **SSH keys**: the user pastes public keys in the settings page; `AuthorizedKeysCommand` looks the key up in Hybriel,
77. so a new key works instantly and no `authorized_keys` file is edited. Supports ed25519/rsa/ecdsa; the forced command
78. refuses shell and any command except the two git ones.
80. ## What hl:proc can and cannot do (for the reading side)
82. Text output comes as lines (`run(...).lines`, `spawn` `line` events); binary output as Bytes (`binary = true`). Fine as
83. lines, because it is text: `log`, `show --stat`, `diff`, `ls-tree`, `for-each-ref`, `rev-list`, `blame
84. --porcelain`, `merge-tree`, `grep`, `shortlog`. Rules for the code that wraps it:
86. - argv list only (`spawnArgs`), never a shell string with user input; slug from a whitelist `[a-z0-9-]`; refs/paths
87. after `--`; refuse names starting with `-`.
88. - Machine-readable formats with a separator that cannot occur (`--format='%H%x1f%an%x1f%at%x1f%s'`), one record per
89. line; take bodies with a separate call. No `-z` (lines only).
90. - **Blobs** (raw file view, images, downloads): `run([...'cat-file','blob', '<ref>:<path>'], { binary = true, cwd })`
91. returns the exact bytes in `.data` (tested, incl. `\xff`, NUL and CRLF). No base64 detour. Text views can use the
92. same call and decode with `.toString()` (a located error on invalid UTF-8: check first, show "binary" instead).
93. - **Archives** (`git archive` zip/tar.gz for releases): the same, `binary = true`, answer the Bytes as the response.
94. - Limit each call (timeout via `kill`, cap on lines) so one huge repo cannot stall the server.
96. ## What else a hub needs (beyond clone/push/browse)
98. Needed for a GitHub/GitLab-like experience, with the git command behind each. **MVP** = needed for #7–#12.
100. | Feature | git mechanism | MVP |
101. |---|---|---|
102. | Create empty repo, default branch | `git init --bare -b <main>`; change via `git symbolic-ref HEAD` | yes |
103. | Rename / delete repo | move / remove directory | yes |
104. | Clone URLs (HTTPS, SSH) on the repo page | — | yes |
105. | Branch and tag lists, default branch | `for-each-ref` | yes |
106. | Commit list per branch / per file, paging | `log --format=… -n -skip <ref> -- <path>` | yes |
107. | Commit page with unified diff + stats | `show`, `diff-tree -p --numstat` | yes |
108. | Tree and file view, raw, README | `ls-tree`, `cat-file` | yes |
109. | Whole project at a commit / branch | `ls-tree -r`, `archive` | yes |
110. | Push hook → PR and release detection | `post-receive` | yes |
111. | Compare two refs, PR diff | `diff a...b`, `rev-list a..b` | yes (#11) |
112. | Merge a PR (fast-forward, merge commit, squash) | `merge-tree --write-tree`, `commit-tree`, `update-ref` in the bare repo, no worktree | later (merging is not decided by the creator) |
113. | Tags for releases (`|||RL`) | annotated tag via `mktag` / `tag -a` with `GIT_*` identity (`env` option) | yes (#12) |
114. | Release archive | `git archive --format=zip\|tar.gz <tag>` | later (content not decided) |
115. | Blame, file history | `blame --porcelain`, `log --follow` | nice |
116. | Code search | `git grep` on a ref | nice |
117. | Contributors / activity | `shortlog -sne`, `rev-list --count` | nice |
118. | SSH keys, access tokens, per-repo collaborators | Hybriel data + transport auth | yes (#7) |
119. | Protected branches / no force push | `update` hook asks Hybriel, or `receive.denyNonFastForwards`, `receive.denyDeletes` | later |
120. | Size limits, garbage collection | `receive.maxInputSize`, scheduled `git gc --auto` | later |
121. | Webhooks / CI triggers | fan-out from `post-receive` | later |
122. | Commit signatures ("Verified") | `%G?` in log format | later |
123. | Forks | `git clone --bare` + alternates | later |
124. | Git LFS | separate protocol | not planned |
126. Nothing needs a working tree on the server: keep bare repos only.
128. ## Issues found (Hybriel)
130. - hybriel#42 (stdin write, raw bytes) and #37 (cwd, env): fixed and merged; the workarounds (shell + base64 for blobs,
131. transport container for HTTPS) are removed from this document.
132. - hybriel#44: `'\n'` is not a newline escape — matters when writing pkt-lines.
134. ## Small choices made here
136. - Clone address: `https://<slug>.gitoria.worldapi.org/repo.git` (path `/repo.git/…` goes to the transport, no clash with
137. `/code`, `/commit`, …); SSH: `[email protected]:<slug>.git` on a separate port.
138. - Repos are stored as `repos/<slug>.git`, bare only.
139. - HTTPS git is served by Hybriel; only SSH gets a container.
140. - Tokens, not account passwords, go in the HTTPS password field.