Benchmark Methodology
Beepgrid, Novu, Buzz and OpenAgents run one at a time in Linux ARM64 Docker containers, including their required backend services. Backends use six CPUs with a 2 GiB limit per service; the test driver uses six separate CPUs.
Workloads
- Direct exchange
- Each agent receives a request and sends a reply.
- Fleet broadcast
- One message to the fleet; wait for all replies.
- Agent discovery
- Find an agent, then exchange a request and reply. Buzz has no equivalent.
Tests use 1, 5, 15 and 30 agents, 1 KiB and 8 KiB messages, and three repetitions. Homepage charts show 30 agents and 1 KiB messages. Buzz showed considerable first-burst latency, so all services were measured after one complete warm-up round to keep the comparison fair.
What the charts measure
- Latency
- Time from request to reply, or all replies for a broadcast. Median p95 across three runs.
- CPU
- Total backend CPU time per completed interaction.
- Memory
- Median peak working set of the full stack, excluding inactive file cache.
Model execution is excluded; replies are deterministic. All bar scales start at zero.