Playground inference
Playground files let you run a prompt template against a real LLM while you develop it, before writing any application code. You can run them from VS Code or from the command line.
Playground files
A playground file pairs a template with a model configuration and a list of sample inputs. It lives next to the template it targets and is named after it: the playground for prompts/Translate.tp is prompts/Translate.play.tp.
<!-- Translate.play.tp -->
<playground for="Translate">
<model>
model: openai/gpt-4o
temperature: 0.7
</model>
<samples>
- name: "Spanish greeting"
input:
text: "Hello, how are you?"
targetLanguage: "Spanish"
- name: "French formal"
input:
text: "Please take a seat."
targetLanguage: "French"
</samples>
</playground>Samples are validated against the template's Input schema at compile time, so they stay in sync as the template evolves. Running a playground renders the prompt for each sample, sends it to the model, and parses the response against the template's Output schema.
Samples can also include history (prior turns) and steps (tool calls made during the current turn) to test multi-turn conversations and tool use. Tools are never executed during inference: when the model requests a tool call, the call itself is the result. To test how the model continues, add the call and its output to the sample's steps.
See the <playground> element reference for the full file structure.
Why <model> is separate from <meta>
Templates often already declare a model in <meta>, which can make the playground's <model> look redundant. They serve different purposes:
<meta>is read by your application. You define its type inmeta.tpand your code interprets its values, so it may use internal aliases (e.g.model: fast) or only reference models your application is wired to call.<model>is read by TypePrompt. It uses a standardprovider/modelidentifier and option set, so TypePrompt can call the model directly.
Keeping them separate lets you try any model on a template (a new release, a cheaper alternative, another provider) without changing the template or adding support for that model to your application first. Once you settle on a model, update <meta> accordingly.
Running from VS Code
Running playgrounds from the editor requires the TypePrompt extension. Search for TypePrompt in the Extensions view, or run:
code --install-extension gepheum.typeprompt-vscodeThen:
- Open a template and click Open Playground (flask icon) in the editor title bar. If the template has no playground yet, one is created with a sample input matching the template's
Inputschema. - In the playground file, click Open Preview (play icon) to open the preview panel.
- Click Run inference to run all samples.
For each sample, the preview shows the parsed output or error, any tool calls requested by the model, latency, and token usage. Runs use the current contents of your editors, including unsaved changes.
Click Save to disk to export the full run (rendered requests, model responses, and a snapshot of the source files) as .run.yaml or .run.json.
Running from the command line
npx @typeprompt/cli run prompts/Translate.play.tpThe command locates the project by searching for typeprompt.yaml in the file's directory and its parents, runs every sample, and prints the inference run as JSON. Use -o to write it to a file instead:
npx @typeprompt/cli run prompts/Translate.play.tp -o runs/translate.jsonThe output has the same structure as runs saved from VS Code. Each entry in samples contains the sample, the rendered request (system message, messages, tools, output schema), and a result whose kind is one of:
kind | Meaning |
|---|---|
output | The model replied. output holds the response parsed against the Output schema, and rawText the unparsed text. |
tool-calls | The model requested tool calls, listed in toolCalls. |
error | The API call failed, or the response did not match the Output schema. |
Results also include telemetry with latency and token usage whenever the model call completed.
Use with coding agents
Because it runs without an editor and produces structured output, typeprompt run lets coding agents iterate on prompts on their own: edit a template, run its playground, read the results, and repeat. For example, in AGENTS.md:
After changing a prompt template in `prompts/`, run its playground with
`npx @typeprompt/cli run prompts/<TemplateName>.play.tp` and check each sample's
`result`. Add samples to the playground file to cover new cases.To have an agent test prompt changes against your running application, with real dependencies and tool integrations, see automating prompt refinement with coding agents.
API keys
VS Code and the CLI both read provider API keys from .typeprompt.secrets.yaml (or .typeprompt.secrets.yml) in the project directory, next to typeprompt.yaml. In VS Code, if no keys are configured, the preview offers to create the file.
# .typeprompt.secrets.yaml
apiKeys:
openai: sk-...
anthropic: sk-ant-...
# Routes requests for any model through Vercel AI Gateway
vercelGateway: ...The supported keys are openai, anthropic, google, xai, mistral, and vercelGateway. All are optional.
WARNING
This file contains secrets. Add it to your .gitignore.
Key resolution
For each inference, TypePrompt picks a key based on the provider/ prefix of the model identifier:
- If the prefix is
openai,anthropic,google,xai, ormistraland that provider's key is set, the model is called directly through the provider's API. - Otherwise, if
vercelGatewayis set, the request is routed through Vercel AI Gateway using the full model identifier. - Otherwise, the inference fails with an error naming the missing key.
A single vercelGateway key is enough to run any model the gateway supports, while provider keys take precedence for the providers they cover.