Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ jobs:
run: node --test scripts/e2e/stale-stacks.test.mjs

- name: Test the agent release scripts
run: node --test scripts/ops/merge-when-green.test.mjs scripts/ops/board.test.mjs scripts/ops/promote.test.mjs scripts/ops/verify-live.test.mjs
run: node --test scripts/ops/merge-when-green.test.mjs scripts/ops/board.test.mjs scripts/ops/promote.test.mjs scripts/ops/verify-live.test.mjs scripts/ops/incident-report.test.mjs

- name: Replay rate migrations on PostgreSQL 17 (Docker)
run: node scripts/test-migration-replay.mjs
Expand Down
66 changes: 15 additions & 51 deletions .github/workflows/incident-report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@ name: Incident report

# The incident inbox for agents (DOS/Rate#1057). When CI fails on dev or main,
# open an `incident` issue (or comment on the open one) with the run, commit
# and failing jobs; when it passes again, close that issue. Agents treat an
# open `incident` issue as their first priority (AGENTS.md).
# and failing jobs; when every failing job passes again, close that issue
# (scripts/ops/incident-report.mjs). Agents treat an open `incident` issue
# as their first priority (AGENTS.md).
# workflow_run only fires for this file as it is on the default branch (dev).
on:
workflow_run:
Expand Down Expand Up @@ -36,57 +37,20 @@ jobs:
BRANCH: ${{ github.event.workflow_run.head_branch }}
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
RUN_ID: ${{ github.event.workflow_run.id }}
RUN_ATTEMPT: ${{ github.event.workflow_run.run_attempt }}
RUN_URL: ${{ github.event.workflow_run.html_url }}
SHA: ${{ github.event.workflow_run.head_sha }}
COMMIT_TITLE: ${{ github.event.workflow_run.head_commit.message }}
steps:
- name: Open, update or close the incident
run: |
set -euo pipefail
# Events can arrive out of order: only the run of the branch's
# current head speaks for the branch, so a stale failure never
# reopens an incident a newer run has closed.
head="$(gh api "repos/${REPO}/commits/${BRANCH}" --jq .sha)"
if [ "$head" != "$SHA" ]; then
echo "Run is for ${SHA:0:8}, ${BRANCH} is at ${head:0:8} now: skipped."
exit 0
fi

title="incident: ${WORKFLOW} failing on ${BRANCH}"
# Untrusted text, shown as inline code so its @mentions and #refs
# ping or link nothing (backticks removed so it cannot close the span).
subject="\`$(printf '%s' "${COMMIT_TITLE}" | head -n 1 | cut -c1-120 | tr -d '\`')\`"
gh label create incident -R "$REPO" --color B60205 \
--description "Production or CI breakage: first priority for any free agent" 2>/dev/null || true
# Read the list first so an API error fails the step (pipefail does
# not cover process substitution) instead of creating a duplicate.
list="$(gh issue list -R "$REPO" --state open --label incident --limit 50 \
--json number,title --jq '.[] | "\(.number)\t\(.title)"')"
open=""
while IFS=$'\t' read -r num t; do
if [ "$t" = "$title" ]; then open="$num"; break; fi
done <<< "$list"
# The script comes from the default branch (workflow_run runs there),
# never from the run's commit: no untrusted code gets the token.
- name: Check out the incident script
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
sparse-checkout: scripts/ops
persist-credentials: false

if [ "$CONCLUSION" = "success" ]; then
if [ -n "$open" ]; then
gh issue comment "$open" -R "$REPO" --body "Recovered: ${RUN_URL} passed on \`${SHA:0:8}\` (${subject})."
gh issue close "$open" -R "$REPO" --reason completed
fi
exit 0
fi

failing="$(gh run view "$RUN_ID" -R "$REPO" --json jobs \
--jq '[.jobs[] | select(.conclusion == "failure" or .conclusion == "timed_out" or .conclusion == "cancelled") | .name] | join(", ")')"
body="$(printf '%s\n' \
"**${WORKFLOW}** ${CONCLUSION} on \`${BRANCH}\`." \
"" \
"- Run: ${RUN_URL}" \
"- Commit: \`${SHA:0:8}\` ${subject}" \
"- Failing jobs: ${failing:-unknown}" \
"" \
"Agents: this is the first priority (AGENTS.md). Claim it, find the breaking change, fix forward. The issue closes itself when the workflow passes on \`${BRANCH}\` again.")"
if [ -n "$open" ]; then
gh issue comment "$open" -R "$REPO" --body "$body"
else
gh issue create -R "$REPO" --title "$title" --label incident --body "$body"
fi
# Closes only when every job that failed has concluded `success` again:
# a job skipped on a later push proves nothing (DOS/Rate#1134).
- name: Open, update or close the incident
run: node scripts/ops/incident-report.mjs
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ gh issue list -R DOS/Rate-Services --state open
- **Quy tắc Định danh Agent (Agent Nickname) & Tránh trùng lặp:**
- Vì mọi Agent đều chạy dưới tài khoản GitHub của JOY (`@me`), **MỖI AGENT PHẢI TỰ ĐẶT MỘT NICKNAME NGẮN GỌN** theo chuyên môn hoặc phiên làm việc (ví dụ: `Crawl`, `Web`, `Core`, `DB`, `QA`) và gắn cả **Nhãn `agent:<nickname>`** lẫn **Cột `Agent` trên GitHub Project #9**.
- **TUYỆT ĐỐI KHÔNG** tự ý làm các Issue đang có nhãn `status:in-progress`, `status:in-review` hoặc đã có nhãn `agent:<khác>` đang chạy (trừ khi JOY chỉ định rõ).
- **Issue có nhãn `incident` là ưu tiên số 1**, trên mọi việc khác: chúng do `.github/workflows/incident-report.yml` tự mở khi CI đỏ trên `dev`/`main` (DOS/Rate-Services có workflow tương tự cho crawler theo lịch), hoặc do Upptime mở trong `DOS/status` khi rate.vn sập. Nhận issue, tìm thay đổi gây lỗi, sửa tiếp (fix forward). Issue tự đóng khi workflow xanh lại.
- **Issue có nhãn `incident` là ưu tiên số 1**, trên mọi việc khác: chúng do `.github/workflows/incident-report.yml` tự mở khi CI đỏ trên `dev`/`main` (DOS/Rate-Services có workflow tương tự cho crawler theo lịch), hoặc do Upptime mở trong `DOS/status` khi rate.vn sập. Nhận issue, tìm thay đổi gây lỗi, sửa tiếp (fix forward). Issue tự đóng khi mọi job đã lỗi chạy xanh lại (job bị bỏ qua, `skipped`, không tính là đã hồi phục).
- Sau đó ưu tiên chọn các Issue đang có nhãn `status:ready`.
- Nếu yêu cầu của người dùng chưa có Issue tương ứng, **TẠO NGAY ISSUE MỚI BẰNG TIẾNG ANH** và gắn vào GitHub Project #9:
```powershell
Expand Down
Loading
Loading