Open-source red teaming for LLMs and AI agents: thirteen attack strategies, a deliberating council of attacker models, and every turn streamed live.
Every run replays turn by turn: each attacker's proposal, the Borda vote, the chairman's synthesis, the target's response and the judge's verdict.
Runs, Statistics, History and Plugins, plus an assistant. Every run is persisted, searchable and replayable.
Attack, target, behaviours, council. The launch button never leaves the screen.
Proposals, Borda ranking, the prompt sent, the target's answer and the judge's verdict, streamed over WebSocket.
HarmBench is included; download or upload more goal datasets, then filter by source and theme.
Attack Success Rate by attack, target and theme, score distribution and council win rates.
Filter by text, attack, theme, status and outcome. One click reopens the deliberation.
See what installs a usable attack adapter and what only installs an upstream library.
Any provider, any model. It explains attacks and answers questions about your live numbers.
Attacks are plugins with an ARES-compatible contract. Council attacks carry an in-loop judge; the baselines record the raw response.
| Attack | What it does | Scoring |
|---|---|---|
| council | N attacker models propose, cross-critique and vote (Borda Top-1); a chairman synthesises the winner and strikes the target. Each turn is a fresh prompt. | in-loop judge 0-5 |
| council-crescendo | The same pipeline, but the target conversation accumulates across turns: gradual escalation. | in-loop judge 0-5 |
| direct | Sends each behaviour verbatim. The baseline every other attack is compared against. | raw response |
| crescendo | A single helper model escalates a multi-turn conversation toward the goal, exploiting conversational consistency; a judge grades each answer (ARES/PyRIT). Fully black-box. | in-loop judge 0-5 |
| human-jailbreak | Wraps each behaviour in known persona templates (DAN, AIM, dev-mode and others). | raw response |
| obfuscation | Encodes the behaviour with the full ARES/garak encoding family (base64, base32, base16, hex, ascii85, uu, ROT13, Morse, NATO, Braille, binary, leetspeak, reversed) and asks the target to decode and comply. | raw response |
| payload-splitting | Splits the behaviour into innocuous fragments the target is asked to reassemble. | raw response |
| tap | Tree of Attacks with Pruning / PAIR (garak): an attacker LLM iteratively proposes and refines adversarial prompts, a judge scores each answer 1-10, and the best branch is kept and improved. | judge 1-10 |
| promptinject | Goal-hijacking directives that try to override the system prompt and prior context (garak). | raw response |
| grandma | Emotional / nostalgia framing that asks the model to role-play a sympathetic figure reciting the content (garak). | raw response |
| smuggling | Disguises the request as a hypothetical fiction or a function to evaluate, so intent detection misses it (garak). | raw response |
| phrasing | Temporal reframing of the request into past or future tense to distance it from a direct instruction (garak). | raw response |
| latent-injection | Hides the request inside a benign document (CV, report, translation) the model is asked to process: indirect prompt injection (garak). | raw response |
For one behaviour. The colours are the ones the console uses for each role.
"Did the run finish?" and "did the attack work?" never share a vocabulary. Green never means "completed".
Whether the process ran to the end.
The judge's best score for a goal. ASR is the share of goals that reached 5.
Statistics and audit exports include only runs marked Security evaluation by default. Smoke tests and older unreviewed runs stay outside the risk posture until you choose otherwise. Filter by attack, target, dataset and purpose; inspect the target response before treating an automatic judge score as a confirmed finding.
Council, chairman, judge and target each reach their model through an OpenAI-compatible endpoint.
A local, single-operator tool that is careful with your keys and with what the backend can reach.
IBM ARES and NVIDIA garak set the bar for library and CLI red teaming. Myrtia adds autonomous multi-agent attackers, a console and analytics.
| Library / CLI tools | Myrtia | |
|---|---|---|
| Multi-agent attacker council with Borda voting and chairman synthesis | — | Yes |
| Custom agentic attacks (persona, critique style, judge, turns) | — | Yes |
| Classic baselines (direct, human jailbreak, obfuscation, payload splitting) | Yes | Yes |
| Live deliberation UI with replayable history | — | Yes |
| ASR analytics dashboard | partial | Yes |
| Built-in assistant aware of your results | — | Yes |
| Provider-agnostic (OpenAI, Anthropic, vLLM, Ollama) | Yes | Yes |
| Self-hosted, Apache-2.0 | Yes | Yes |
Python 3.11+. A local Ollama with a small model is enough to see a council deliberate, no API key needed.
# install cd backend && pip install -e . && pip install -e vendor/ares-council # run the server (API + console) uvicorn myrtia.api.app:app --port 8000 # open the console open http://127.0.0.1:8000/
Myrtia is open source today. The direction is a hosted service: point your own model and agent endpoints at a deliberating council and get a continuous, reproducible adversarial audit — ASR trends, regressions and an exportable report — without standing up the infrastructure yourself.
On the roadmap.