CI concurrency group cancels other branches' runs #231

New issue

Closed

opened 2026-08-05 20:53:16 +00:00 by robbertbos · 1 comment

robbertbos commented

2026-08-05 20:53:16 +00:00

Owner

.forgejo/workflows/ci.yml:9-11:

concurrency:
  group: ci-${{ forgejo.ref }}
  cancel-in-progress: true

On a pull_request event forgejo.ref does not resolve to a per-PR value, so every open PR shares one concurrency group and each new run cancels whatever is in flight - including runs on unrelated branches.

Measured on PR #224 (all times UTC, from /api/v1/repos/robbertbos/waggle/actions/tasks):

Time	Event
20:48:07	run for `3ca9a4c` (PR #224) starts
20:48:56 - 20:49:05	its `e2e`, `backend-test`, `backend-test-postgres`, both `build` jobs and `pip-audit` are cancelled
20:48:59 - 20:49:11	runs for `4f75c3f4` and `0ba2f3e5` (other branches) start

The cancellation window sits exactly inside the window where the other branches' runs are queued. Same-PR cancellation works correctly and is wanted: the run for f052be07 was cancelled at 20:48:10 when 3ca9a4c was pushed to the same PR.

Effects:

A PR shows failure with Has been cancelled on up to 8 of 14 checks while nothing is actually broken, which blocks merging on a red record that says nothing about the code.
The long jobs are the ones that lose: e2e and backend-test-postgres run 3-5 minutes, so they are almost always the ones still in flight when someone else pushes. The fast checks pass and the expensive ones never finish - the reverse of what you want.
Re-triggering is a coin flip: the re-run cancels somebody else's run in turn.

Fix is to make the group unique per PR, falling back to the ref for branch and tag pushes:

concurrency:
  group: ci-${{ forgejo.event.pull_request.number || forgejo.ref }}
  cancel-in-progress: true

Worth checking .forgejo/workflows/*.yaml for the same pattern in the other workflows while you are in there - test-build and security-scan were cancelled in the same sweep, so they likely share the defect.

Not fixed in #224: that PR is an accessibility change and a CI-config fix wants its own verification (push to two branches at once and confirm both runs survive).

`.forgejo/workflows/ci.yml:9-11`: ```yaml concurrency: group: ci-${{ forgejo.ref }} cancel-in-progress: true ``` On a `pull_request` event `forgejo.ref` does not resolve to a per-PR value, so every open PR shares one concurrency group and each new run cancels whatever is in flight - including runs on unrelated branches. Measured on PR #224 (all times UTC, from `/api/v1/repos/robbertbos/waggle/actions/tasks`): | Time | Event | |---|---| | 20:48:07 | run for `3ca9a4c` (PR #224) starts | | 20:48:56 - 20:49:05 | its `e2e`, `backend-test`, `backend-test-postgres`, both `build` jobs and `pip-audit` are cancelled | | 20:48:59 - 20:49:11 | runs for `4f75c3f4` and `0ba2f3e5` (other branches) start | The cancellation window sits exactly inside the window where the other branches' runs are queued. Same-PR cancellation works correctly and is wanted: the run for `f052be07` was cancelled at 20:48:10 when `3ca9a4c` was pushed to the same PR. Effects: - A PR shows `failure` with `Has been cancelled` on up to 8 of 14 checks while nothing is actually broken, which blocks merging on a red record that says nothing about the code. - The long jobs are the ones that lose: `e2e` and `backend-test-postgres` run 3-5 minutes, so they are almost always the ones still in flight when someone else pushes. The fast checks pass and the expensive ones never finish - the reverse of what you want. - Re-triggering is a coin flip: the re-run cancels somebody else's run in turn. Fix is to make the group unique per PR, falling back to the ref for branch and tag pushes: ```yaml concurrency: group: ci-${{ forgejo.event.pull_request.number || forgejo.ref }} cancel-in-progress: true ``` Worth checking `.forgejo/workflows/*.yaml` for the same pattern in the other workflows while you are in there - `test-build` and `security-scan` were cancelled in the same sweep, so they likely share the defect. Not fixed in #224: that PR is an accessibility change and a CI-config fix wants its own verification (push to two branches at once and confirm both runs survive).

robbertbos commented

2026-08-05 21:09:03 +00:00

Author

Owner

Checked this against the Forgejo source and against the full task history. It is not a defect - closing.

`forgejo.ref` is already per-PR

On a pull_request event Forgejo sets run.Ref = pr.GetGitRefName(), i.e. refs/pull/<n>/head (services/actions/notifier_helper.go:115-116, v15.0.3 - the version this instance runs). services/actions/context.go:59 hands that exact field to the context used to evaluate concurrency.group. So the group already resolves to ci-refs/pull/224/head, unique per PR.

The measurement on #224 was misread

Pulling all 8026 entries from /actions/tasks and reconstructing the window:

Issue says	API says
`0ba2f3e5` is another branch	run 2036, `head_branch` = #224 - the same PR
`4f75c3f4` started inside the cancellation window	run 2033 (PR #228) started 20:48:25, before the cancellations (20:48:56-20:49:05), and ran through to success at 20:53:27

The actual sequence: run 2030 (#224, 3ca9a4cb) is cancelled because a new push lands on the same PR around 20:48:56, whose jobs appear at 20:49:13 as run 2036 (0ba2f3e5). The cancellation timestamp precedes the new run's jobs because Forgejo calls CancelPreviousWithConcurrencyGroup before InsertRun (services/actions/workflows.go:167-171). That is the same mechanism that cancelled f052be07 at 20:48:10 - which the issue itself calls correct and wanted.

Counter-test

If the groups collided, cancel-in-progress would kill every run that is in flight when another branch starts. Over all ci.yml runs whose durations are not polluted by the nightly cleanup sweep:

runs that had a DIFFERENT-branch run start mid-flight:
  survived intact : 17
  cancelled       :  3   <- all three explained by a successor on the same PR

The proposed fix is a no-op

ci-${{ forgejo.event.pull_request.number || forgejo.ref }} changes the group from ci-refs/pull/224/head to ci-224. Equally unique, same behaviour. And test-build.yaml / security-scan.yaml carry no concurrency block at all - they fall back to Forgejo's default <ref>_<workflow>_<event>__auto, which is also per-ref.

One loose end

One cluster I could not fully explain: on 2026-07-25 between 11:50 and 12:30 (the Renovate merge train #183 to #186) runs on main and on several PRs are cancelled within seconds of each other's start, in a chain. That does look like what this issue describes. But it is the last such occurrence in the whole dataset - everything after it (07-26, 08-05) is cleanly same-branch, and the proposed fix would not address it either, since a push-to-main run is among the cancelled ones and that has no PR number. Not reproducible on the current version. Worth a new issue if it comes back.

Checked this against the Forgejo source and against the full task history. It is not a defect - closing. ## `forgejo.ref` is already per-PR On a `pull_request` event Forgejo sets `run.Ref = pr.GetGitRefName()`, i.e. `refs/pull/<n>/head` (`services/actions/notifier_helper.go:115-116`, v15.0.3 - the version this instance runs). `services/actions/context.go:59` hands that exact field to the context used to evaluate `concurrency.group`. So the group already resolves to `ci-refs/pull/224/head`, unique per PR. ## The measurement on #224 was misread Pulling all 8026 entries from `/actions/tasks` and reconstructing the window: | Issue says | API says | |---|---| | `0ba2f3e5` is another branch | run 2036, `head_branch` = **#224** - the same PR | | `4f75c3f4` started inside the cancellation window | run 2033 (PR #228) started **20:48:25**, before the cancellations (20:48:56-20:49:05), and ran through to **success** at 20:53:27 | The actual sequence: run 2030 (#224, `3ca9a4cb`) is cancelled because a new push lands on the *same* PR around 20:48:56, whose jobs appear at 20:49:13 as run 2036 (`0ba2f3e5`). The cancellation timestamp precedes the new run's jobs because Forgejo calls `CancelPreviousWithConcurrencyGroup` *before* `InsertRun` (`services/actions/workflows.go:167-171`). That is the same mechanism that cancelled `f052be07` at 20:48:10 - which the issue itself calls correct and wanted. ## Counter-test If the groups collided, `cancel-in-progress` would kill every run that is in flight when another branch starts. Over all ci.yml runs whose durations are not polluted by the nightly cleanup sweep: ``` runs that had a DIFFERENT-branch run start mid-flight: survived intact : 17 cancelled : 3 <- all three explained by a successor on the same PR ``` ## The proposed fix is a no-op `ci-${{ forgejo.event.pull_request.number || forgejo.ref }}` changes the group from `ci-refs/pull/224/head` to `ci-224`. Equally unique, same behaviour. And `test-build.yaml` / `security-scan.yaml` carry no `concurrency` block at all - they fall back to Forgejo's default `<ref>_<workflow>_<event>__auto`, which is also per-ref. ## One loose end One cluster I could not fully explain: on 2026-07-25 between 11:50 and 12:30 (the Renovate merge train #183 to #186) runs on main and on several PRs are cancelled within seconds of each other's start, in a chain. That does look like what this issue describes. But it is the last such occurrence in the whole dataset - everything after it (07-26, 08-05) is cleanly same-branch, and the proposed fix would not address it either, since a push-to-main run is among the cancelled ones and that has no PR number. Not reproducible on the current version. Worth a new issue if it comes back.

robbertbos closed this issue

2026-08-05 21:09:09 +00:00