Pullfrog vs Foreman: Two Ways to Put AI Agents on GitHub Issues and PRs

If you read the eve deep dive from earlier this week, you know the framework by now: filesystem-first, durable by default, sandboxed execution, and a filesystem contract where paths are identity. What that post did not cover is what you can actually build with eve, because the interesting part is not the primitives, it is what people assemble out of them. Two open-source projects currently make the point better than anything else: Vercel's Foreman, which turns eve into a software factory that takes issues and returns reviewed draft pull requests, and Pullfrog, a GitHub bot that runs your favorite coding agent inside GitHub Actions.

They could not be more different in shape, and both work. Foreman is a linear pipeline with four agent stations and a memory that persists between runs. Pullfrog is an event-driven bot: one agent per run, triggered by a mention or a webhook, with GitHub as the source of truth. This post is the hands-on comparison: how each one works under the hood, with real commands and configuration, where the trust boundaries sit, and which one to reach for when an issue lands in your queue.

The short version

• Foreman is a software factory, not a bot. It is a Vercel-hosted eve template that moves each issue through four stations (Classifier, Analyst, Implementer, Reviewer) and ends in a draft PR reviewed against acceptance criteria. People stay on the judgment calls: the ceiling for an unattended run is a draft PR.
• Pullfrog is a GitHub bot that runs your own agent in your repository's Actions. Tag @pullfrog anywhere, or configure automations for issues, PRs, reviews, and CI failures. You bring the model key; the action's MCP server handles git and GitHub operations behind a permission layer.
• The architecture difference is a pipeline with a brain versus an agent per event. Foreman keeps a factory brain, notes about your repository that every run starts from. Pullfrog pulls context from wherever it was triggered and forgets between runs.
• The review story is the real differentiator. Foreman's Reviewer is independent by construction (separate checkout, different model vendor, evidence for every verdict). Pullfrog iterates on human review comments and can gate merges with a requireable approval check.
• Both are MIT and both keep a human gate. Foreman never merges anything. Pullfrog's checks only block merges if you add them to branch protection. Neither replaces the human review; both move the boring parts before it.

What Foreman actually is

Foreman is a public template in vercel-labs/eve-software-factory-template (MIT), which Vercel describes as an eve software factory that puts AI agents on every stage of the development loop and keeps people on the judgment calls. You point it at a repository, connect GitHub and Linear, and it turns issues into reviewed draft pull requests. The docs are refreshingly blunt about what it is: a template, and the expectation is that you change it.

The four stations

• Classifier. Triages the task on a fast model: type, priority, complexity, affected area, and whether it is actionable at all. When it is not actionable, the pipeline stops and asks the requester instead of building the wrong thing.
• Analyst. Turns the item into a plan with acceptance criteria, working from a live checkout of your repository. Those criteria are the exact text the Reviewer will later judge verbatim.
• Implementer. Executes the plan in its own sandboxed checkout on the strongest coding model available, verifies with your repository's own lint, typecheck, and test commands, and pushes a factory/* branch.
• Reviewer. Fetches the pushed branch into a separate checkout, reads the real diff, and judges each criterion with evidence. It approves, requests changes, or rejects, and only an approval opens the draft PR.

Each station is its own eve subagent with its own instructions, sandbox, and tools; the orchestrator routes between them and never does a station's job itself. The independence is by construction, not by instruction: the Reviewer sees only the pushed branch, never the Implementer's reasoning, and it runs on a different model vendor by default, so it does not share the Implementer's idiom or blind spots. Above that sits a hard ceiling. An unattended run tops out at a draft pull request: marking a PR ready waits for a person, and merge tools are not mounted at all. Nothing ships without you.

How work arrives

• Label an issue factory. The pipeline runs on its own, posts progress as stations complete, and ends with a draft PR linked to the issue.
• @mention it. Mentions from owners, members, and collaborators start an interactive session on an issue or PR.
• Delegate in Linear. Linear Agent Sessions run the same pipeline and report progress back in Linear.
• The dev TUI. Hand it a task locally; local runs are treated as untrusted, so changes to GitHub wait for your approval in the TUI.
• Red CI on a factory PR. Foreman diagnoses the failure and pushes a fix to its own branches, never yours.
• Someone opens a pull request. Foreman posts one orienting comment for reviewers: a summary, not a review.

Configuration and local dev

$ vercel link
$ vercel env pull
$ pnpm dev
# hand the TUI a task, watch the four stations fire in order
# local runs are untrusted: changes to GitHub wait for your approval

The Vercel deploy flow wires the GitHub connector, the Linear connector, a Vercel Blob store for the factory brain, and prompts for two environment variables. FACTORY_REPO is required and the build fails without it; FACTORY_LABEL defaults to factory. The full set from .env.example:

FACTORY_SETUP_COMMAND is the one most people get wrong on a monorepo: it runs once inside the sandbox at build time, so if your repository needs pnpm install (or any workspace bootstrap) before anything works, that is where it goes.

What Pullfrog actually is

Pullfrog describes itself as the open-source, model-agnostic BYOK GitHub bot that runs in GitHub Actions, and the landing page leads with bring your favorite coding agent into GitHub. It is generally available, MIT-licensed, and built around a single action, pullfrog/pullfrog@v0, plus a console that manages automations and triggers. There is no hosted agent runtime in the traditional sense: every run happens in your repository's own Actions workflow, on your runners, guarded by your secrets.

Tag, automate, iterate

• Tag @pullfrog. Drop it in a comment anywhere in the repo and it pulls in the relevant context through the action's internal MCP server and performs the task.
• Automated triggers. Issue created, issue labeled, PR created, PR review created, PR review requested, CI failure, merge conflict, and more, each trigger with its own custom prompt instructions.
• Iterate via PR reviews. Leave review comments on a PR Pullfrog created, exactly as you would for a human colleague, and it addresses them and keeps you posted.
• Auto-fix CI failures. It reads the logs of its own PRs and attempts a fix, and it can be configured to do the same on human-created PRs.
• Built-in headless browser. The agent can run end-to-end tests and take screenshots, uploaded to a secure S3 bucket with no extra configuration.

The action and its permission layer

The action installs and spawns the agent, then hands it three tools: a purpose-built MCP server for git and GitHub operations (create PRs, leave reviews, read CI logs, manage issues), an isolated shell subprocess with no access to sensitive environment variables, and the headless browser. All GitHub operations go through Pullfrog's permission layer, so the agent cannot push to protected branches or reach repositories it should not. API keys live in GitHub secrets, the action auto-masks them and passes through only the minimum environment, and every run authenticates with a short-lived GitHub App installation token that is revoked when the run finishes.

Every PR run posts a pullfrog commit-status check (in progress, then success or failure), and an optional pullfrog-approval check reports whether Pullfrog would approve the PR. Both are configured in the console and both can be required as branch-protection status checks. That is the mechanism if you want the bot to gate merges rather than just comment.

Standalone usage

name: Release
on:
  push:
    tags: ['v*']
permissions:
  contents: write
jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
      - name: Generate release notes
        id: notes
        uses: pullfrog/pullfrog@v0
        with:
          prompt: |
            Generate release notes for ${{ github.ref_name }}.
            Compare commits between this tag and the previous tag.
            Format as markdown: summary paragraph, then
            ### Features, ### Fixes, ### Breaking Changes sections.
            Omit empty sections. Be concise.
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - name: Create GitHub release
        run: |
          notesfile="$RUNNER_TEMP/release-notes-$GITHUB_RUN_ID.md"
          printf '%s' "$NOTES" > "$notesfile"
          gh release create ${{ github.ref_name }} \
            --title "${{ github.ref_name }}" \
            --notes-file "$notesfile"
        env:
          GH_TOKEN: ${{ github.token }}
          NOTES: ${{ steps.notes.outputs.result }}

The action exposes a result output that subsequent steps can consume, plus two conveniences worth knowing. prompt_file resolves the prompt from a file committed to the repo (mutually exclusive with prompt, set exactly one), and output_schema forces structured JSON output from a JSON schema you can generate from Zod, so the next step parses a deterministic result instead of prose.

How they differ: the part that matters

A pipeline with a brain versus an agent per event

Foreman is a stateful process: between runs it keeps a factory brain, notes about your repository that every run starts from, so the tenth issue on the same codebase does not re-learn the layout. Pullfrog is deliberately stateless: it pulls the context around the mention or the triggering event, runs, and is done. That makes Pullfrog simpler to reason about and cheaper to operate, and it means every run starts from the same cold start, which is either a feature (reproducible) or a tax (no accumulated context), depending on what you are automating.

Where the code runs and who holds the keys

Foreman executes on Vercel: the eve app, the sandboxes for the Implementer, the connectors for GitHub and Linear, and a Blob store for memory. Model traffic goes through the AI Gateway or provider keys, and the sandbox keeps model-controlled code away from your secrets, the trust boundary we covered in the eve deep dive. Pullfrog executes inside your repository's Actions: your runners, your secrets, your billing, and a short-lived installation token per run. That is a genuinely different trust posture. With Pullfrog, GitHub is both the trigger source and the enforcement point; with Foreman, the pipeline holds the state and GitHub only sees the output branch and the PR.

The review is the difference

Foreman's Reviewer is independent by construction: it never sees the Implementer's reasoning, it works from a separate checkout of the pushed branch, and it judges against the exact acceptance criteria the Analyst wrote, with evidence for each verdict. It also runs on a different model vendor than the Implementer by default, a deliberate hedge against shared blind spots. Pullfrog's review loop is different and just as useful: it creates PRs, then iterates on the comments you leave, the way a junior engineer would, and it reads CI logs to fix its own failures. If you want a gate, you add the approval check to branch protection. One is a reviewer before you look; the other is a collaborator after you look. Most teams want a bit of both.

Hands-on: getting Foreman running

The fastest path is the Vercel deploy flow, which clones the template and wires the connectors. Then link locally to develop against your own repository, and label an issue factory to watch the pipeline run end to end.

$ vercel link
$ vercel env pull
$ pnpm dev

# label an issue with 'factory' to kick off the pipeline
# progress posts as each station completes
# the run ends with a draft PR linked to the issue

Hands-on: wiring Pullfrog without the GitHub App

With the GitHub App installed, the console handles the triggers and you only add the action workflow. The manual path is worth understanding even if you use the App, because it shows exactly what the bot is: a reusable workflow plus a trigger workflow that forwards event payloads as the prompt.

# .github/workflows/pullfrog.yml
name: Pullfrog
on:
  workflow_dispatch:
    inputs:
      prompt:
        type: string
        description: 'Agent prompt'
permissions:
  contents: read
jobs:
  pullfrog:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 1
      - name: Run agent
        uses: pullfrog/pullfrog@v0
        with:
          prompt: ${{ inputs.prompt }}
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

# .github/workflows/triggers.yml
name: Agent Triggers
on:
  issue_comment:
    types: [created]
  issues:
    types: [opened, assigned]
  pull_request_review:
    types: [submitted]
jobs:
  pullfrog:
    if: contains(github.event.comment.body, '@pullfrog') || contains(github.event.issue.body, '@pullfrog')
    permissions:
      id-token: write
      contents: read
    uses: ./.github/workflows/pullfrog.yml
    with:
      prompt: ${{ toJSON(github.event) }}
    secrets: inherit

The trigger workflow forwards the full event payload as the prompt, and the action works out what to do. That is the whole architecture, and it is why the bot is so easy to extend: any event you can express as a workflow trigger becomes an automation, and the standalone step form lets you drop the agent into any pipeline that needs a generated artifact, like release notes or a version bump decision.

Which should you pick?

• Choose Foreman when: you want a repeatable, reviewed pipeline from issue to draft PR, you want the factory brain to learn your repo over time, you like the idea of an independent Reviewer as a pre-review gate, or you are already in the Vercel and eve world. The cost is that it is a template you are expected to change, and it is Vercel-shaped.
• Choose Pullfrog when: you want your favorite agent and your own model keys inside your own Actions, you want the flexibility of event-driven automations and ad-hoc @pullfrog tasks, or you want to drop an agent into an existing workflow step. The cost is that every run is a cold start with no cross-run memory.
• Consider both when: you want Foreman producing the reviewed draft PR while Pullfrog handles the human-facing automations, issue triage, and release notes on the side. They do not conflict; Pullfrog reads the same issues and PRs that Foreman writes.
• Consider neither when: the flow is fully deterministic and a plain CI pipeline with scripts does the job, or you need a human to own the loop end to end with no agent between the issue and the diff. The agent is a layer, not a requirement.

I did not pick a winner here, and neither should you until you have pointed both at a real repository. The experiment is cheap: Foreman is a template deploy, Pullfrog is two YAML files, and both are MIT, so the only thing you are really choosing is which loop you want agents to own: the pipeline before the PR, or the events around it.

Official sources

• Foreman docs: https://ask-foreman.dev/docs
• eve Software Factory Template (GitHub): https://github.com/vercel-labs/eve-software-factory-template
• eve documentation: https://eve.dev/docs/introduction
• Pullfrog on GitHub: https://github.com/pullfrog/pullfrog
• Pullfrog site and pricing: https://pullfrog.com
• Pullfrog docs (PR reviews and status checks): https://docs.pullfrog.dev/pr-reviews
• Our earlier eve deep dive: https://systhoughts.com/posts/vercel-eve-vs-mastra-vs-flue-2-0

Are you running Foreman, Pullfrog, or both? Which loop did you hand to the agents, and where did the pipeline still need a human? Drop it in the comments.

Until next time, keep your systems thoughtful.