Zap Runtime: Convex, Upstash, and Provider Execution

​

Architecture Overview

POST /api/zaps/run
  -> validate Zap.md          (parseZapMarkdown, assertWithinBudget)
  -> create run + steps in Convex
  -> submit provider job       (provider adapter → submit())
  -> enqueue Upstash poll job  (idempotency key written to Redis)
  -> drain endpoint polls provider
  -> update Convex idempotently (step status, asset URL, actualUsd)

​

Components

​

Convex

queued → running → waiting (RLHF pause) → done
                                        → failed

queued → running → done
                 → failed
                 → skipped

​

Upstash Redis

1. Idempotency keys — each provider job submission writes a key to Redis before the API call. If the submission is retried (e.g. after a timeout), the existing requestId is returned instead of creating a duplicate job.
2. Provider poll queues — after submission, a poll job is enqueued in Upstash. The drain endpoint consumes these jobs on a schedule, calls provider.poll(requestId), and updates Convex when the result is ready.

UPSTASH_REDIS_REST_URL=https://your-instance.upstash.io
UPSTASH_REDIS_REST_TOKEN=your_token_here

​

Provider Adapters

interface ProviderAdapter {
  id: "gmi" | "fal" | "mock";
  supports(capability: Capability, model: string): boolean;
  price(req: GenRequest): number;
  submit(req: GenRequest, idemKey: string): Promise<ProviderSubmitResult>;
  poll(requestId: string, secrets?: Record<string, string>): Promise<ProviderPollResult>;
}

type GenRequest = {
  attemptSalt?: string;
  capability: Capability;       // ZapStep["kind"]
  durationS?: number;
  inputs: Record<string, unknown>;
  model: string;
  prompt: string;
  provider?: string;
  runId: string;
  secrets?: Record<string, string>;
  stepId: string;
  webhookUrl?: string;
};

type ProviderPollResult = {
  status: "queued" | "running" | "done" | "failed";
  outputUrl?: string;  // set when status === "done"
  progress?: number;   // 0–100
  actualUsd?: number;  // final billed amount
  error?: string;      // set when status === "failed"
};

type ProviderSubmitResult = {
  provider: string;
  requestId: string;
};

​

Poll Drain Endpoint

1. Dequeues a batch of pending poll jobs from Upstash
2. For each job, calls provider.poll(requestId, secrets)
3. On done: writes the asset to @vercel/blob, records the asset in Convex, marks the step done
4. On failed: records the error on the Convex step, marks the step failed
5. On queued / running: re-enqueues the job with an exponential backoff delay

​

Blob Store

BLOB_READ_WRITE_TOKEN=vercel_blob_rw_your_token

​

BYOK Key Retrieval

1. The client sends a wallet-authenticated Supabase bearer token with the run request
2. The server verifies the JWT and the ZAP_SECRET_REVEAL_TOKEN environment variable
3. Provider keys are fetched from Supabase and injected into the GenRequest.secrets field
4. The provider adapter reads secrets during submit() and poll() calls
5. Plaintext keys are never returned to the browser — they exist only in the server-side request context

ZAP_SECRET_REVEAL_TOKEN=your_reveal_token

​

HyperFrames Stitching

1. Detect stitch.engine: hyperframes in the planned step
2. Generate a temporary HyperFrames project directory with a Zap visual identity (DESIGN.md is written automatically if not present in the recipe root)
3. Validate the project by running the HyperFrames CLI checks in sequence:
   npx hyperframes lint
   npx hyperframes validate
   npx hyperframes inspect
   
4. Render the composition to the output format and quality:
   npx hyperframes render --format mp4 --quality standard
   
5. Persist the rendered file to @vercel/blob and record the asset in Convex
6. On failure: record the error message on the Convex step record and fall back to the first resolved stitch asset — the run is marked done (not failed) with an explanatory step error

​

Live vs Mock Runs

​

Environment Variable Reference