Run an experience on every PR and pin the report

Problem

You want pull requests to run an experience (a review flow, a doc check, a smoke test) and surface the result where reviewers already are: as a comment on the PR, with a one-click link to the full oe inspect --html run report. And you want it turnkey — no bespoke shell glue copied between repos.

Solution

A reusable composite action ships in this repo at .github/actions/oe-report. It sets up Node 20, installs @openexpertise/cli, runs your experience, captures the run-id, and renders the HTML report — exposing run-id, report-path, and status as outputs. Reference it by tag:

- uses: xingchengxu/OpenExpertise/.github/actions/oe-report@v0.1.5
  with:
    experience-path: examples/review-branch
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

The LLM key is passed by you, the caller, via env — never as an action input. Inputs get echoed into run logs; env does not, and composite steps inherit the calling step's env so oe sees the key. Tool-only experiences (no agent / cli-agent nodes) need no key at all — drop the env: block.

The complete workflow

Drop this in .github/workflows/pr-report.yml. It triggers on pull_request, runs the experience through the oe-report action, uploads the report as an artifact, and posts (then updates) a single sticky PR comment.

name: PR run report

on:
  pull_request:

# Least privilege: read the code, write the PR comment. Nothing else.
permissions:
  contents: read
  pull-requests: write

jobs:
  report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - id: oe
        uses: xingchengxu/OpenExpertise/.github/actions/oe-report@v0.1.5
        with:
          experience-path: examples/review-branch
          args: '{"pr_url": "${{ github.event.pull_request.html_url }}"}'
          llm: anthropic # keyed experiences only — tool-only flows omit BOTH this and env: below
          report-path: oe-report.html
        env:
          # Repo secret → caller env. Omit this whole env: block (and the llm:
          # input above) for tool-only experiences — they run keyless.
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

      - id: upload
        if: always() # still upload the report when the run step failed
        uses: actions/upload-artifact@v4
        with:
          name: oe-report
          path: ${{ steps.oe.outputs.report-path }}
          if-no-files-found: warn # not error — a report-less failed run must still reach the comment step

      - name: Post run summary on the PR
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const runId = ${{ toJSON(steps.oe.outputs['run-id']) }};
            const status = ${{ toJSON(steps.oe.outputs.status) }} || 'failed';
            const artifactUrl = ${{ toJSON(steps.upload.outputs['artifact-url']) }};
            const icon = status === 'success' ? '✅' : '❌';

            const body = [
              '<!-- oe-report -->',
              '## OpenExpertise run report',
              '',
              `${icon} **Status:** \`${status}\``,
              `**Run id:** \`${runId ?? 'n/a'}\``,
              '',
              artifactUrl
                ? `📊 [Download the HTML run report](${artifactUrl}) (artifact \`oe-report\`).`
                : 'Report artifact unavailable for this run.',
              '',
              `<sub>Generated by \`oe inspect --html\` · workflow run [#${context.runId}](${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId})</sub>`,
            ].join('\n');

            const { owner, repo } = context.repo;
            const issue_number = context.issue.number;

            // Update the existing sticky comment if present, else create one.
            // Match on the unique marker alone (not user.type) and paginate
            // fully, so the prior comment is always found regardless of
            // identity or comment volume — no duplicate spam.
            const comments = await github.paginate(
              github.rest.issues.listComments,
              { owner, repo, issue_number, per_page: 100 },
            );
            const existing = comments.find(
              (c) => c.body?.includes('<!-- oe-report -->'),
            );

            if (existing) {
              await github.rest.issues.updateComment({
                owner, repo, comment_id: existing.id, body,
              });
            } else {
              await github.rest.issues.createComment({
                owner, repo, issue_number, body,
              });
            }

Walkthrough

• on: pull_request runs the job in the context of the base repo's default branch workflow but with the head ref checked out, and — crucially — does not expose repo secrets to workflows triggered by forked PRs (see the security note below). For same-repo PRs the secret is available.
• permissions: is scoped to the minimum: contents: read to check out, pull-requests: write so github-script can comment. No id-token, no packages, no broad write-all.
• The oe-report action does the heavy lifting. Internally it runs oe run <path> --args <json> --llm <provider>, then captures the run-id by diffing the experience's .openexpertise/runs/ directory before and after the run (the run-id is logged via pino, not printed as a clean value, and there is no --print-run-id flag), then renders oe inspect <run-id> --experience <dir> --html -o <report-path>. Outputs run-id, report-path, and status flow back to the workflow.
• actions/upload-artifact@v4 persists the self-contained HTML so reviewers can open it after the runner is gone; its artifact-url output is the link we put in the comment. It carries if: always() so the report still uploads when the run step failed, and if-no-files-found: warn (not error) so a genuinely report-less failed run doesn't hard-fail the job — it falls through to the comment step.
• actions/github-script@v7 posts a sticky comment: it paginates all comments, finds the prior one containing the unique <!-- oe-report --> marker, and updates it instead of stacking a new one on every push. if: always() on both this step and the upload step ensures you still get a report comment (with status: failed) when the run fails.

Security: pull_request vs pull_request_target

Use pull_request. Do not reach for pull_request_target to "fix" a missing secret on forked PRs.

• pull_request runs the workflow definition from the base branch but withholds secrets from fork-originated PRs. That is the safety property you want: untrusted contributor code never executes with your ANTHROPIC_API_KEY / OPENAI_API_KEY in scope.
• pull_request_target runs with full access to secrets and (if you naively check out the PR head) executes untrusted code with those secrets in scope — the classic exfiltration footgun. The attacker opens a PR whose code reads $ANTHROPIC_API_KEY and posts it somewhere.

For an experience that needs an LLM key, the right posture is: keep pull_request, and accept that fork PRs will run without the key. Because the action omits --llm by default, oe auto-detects from env and a missing key only surfaces when an agent / cli-agent / skill node actually fires — so the run reaches that node before failing (it does not abort at provider resolution on startup), and the report still uploads with status: failed. If you must run keyed experiences on fork PRs, gate them behind a manual workflow_dispatch / a label + a separate, reviewed workflow — not pull_request_target with a head checkout. Same-repo (branch) PRs get the secret and run end to end.

Variation: no composite action (standalone)

If you'd rather not depend on the oe-report action — or you're on a fork that can't reference it — inline the same steps. This is exactly what the composite action does:

name: PR run report (standalone)

on:
  pull_request:

permissions:
  contents: read
  pull-requests: write

jobs:
  report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - run: npm i -g @openexpertise/cli@0.1.5

      - id: run
        shell: bash
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} # omit for tool-only experiences
          # Pass PR data through env (never into the run: line) so it can't
          # become live shell, then build the JSON args from the shell var.
          PR_URL: ${{ github.event.pull_request.html_url }}
        run: |
          set -euo pipefail
          EXP="examples/review-branch"
          RUNS_DIR="$EXP/.openexpertise/runs"

          # Snapshot run logs so we attribute the right one to THIS run.
          mkdir -p "$RUNS_DIR"
          BEFORE="$(mktemp)"
          ls -1 "$RUNS_DIR"/*.jsonl 2>/dev/null | sort > "$BEFORE" || true

          ARGS="$(jq -nc --arg u "$PR_URL" '{pr_url: $u}')"
          STATUS="success"
          # --llm omitted: oe auto-detects from env and runs tool-only flows
          # keyless. Add --llm anthropic (or openai) only for keyed experiences.
          oe run "$EXP" --args "$ARGS" || STATUS="failed"
          echo "status=$STATUS" >> "$GITHUB_OUTPUT"

          # Newest run log that appeared = this run's id (fallback: newest by mtime).
          AFTER="$(mktemp)"
          ls -1 "$RUNS_DIR"/*.jsonl 2>/dev/null | sort > "$AFTER" || true
          NEW_LOG="$(comm -13 "$BEFORE" "$AFTER" | tail -n 1 || true)"
          [ -z "$NEW_LOG" ] && NEW_LOG="$(ls -1t "$RUNS_DIR"/*.jsonl 2>/dev/null | head -n 1 || true)"
          RUN_ID="$(basename "$NEW_LOG" .jsonl)"
          echo "run-id=$RUN_ID" >> "$GITHUB_OUTPUT"

          oe inspect "$RUN_ID" --experience "$EXP" --html -o oe-report.html

      - if: always() # still upload the report when the run step failed
        uses: actions/upload-artifact@v4
        with:
          name: oe-report
          path: oe-report.html
          if-no-files-found: warn # not error — a report-less failed run must still reach the comment step
      # …then the same actions/github-script@v7 sticky-comment step as above.

Tool-only experiences need no key

If your experience has only tool / dataset / mcp nodes (no agent or cli-agent), there is nothing to authenticate. Delete the env: block — the workflow runs end to end on fork PRs too, since no secret is involved. This is the friendliest setup for community contributions.

See also

• Visualize a graph & share a run report — the oe graph / oe inspect --html commands this recipe drives
• Reference: oe run
• Reference: oe inspect