How to Give Your Self-Hosted AI Agent Inbound Phone Calls (OpenClaw + Twilio)

Most self-hosted AI agents live in a chat window or a terminal. They can browse the web, send emails, run code, and manage your calendar — but when someone calls your phone, the agent is nowhere in the loop. Adding inbound phone calls closes that gap and turns your agent into something that can actually be interrupted, reached, and useful in the real world. This guide covers every layer of the stack: the native voice-call plugin that ships with OpenClaw, the community bridges for Deepgram and Twilio, the DIY Twilio Media Streams path for maximum control, and the tool-access question that most write-ups skip but that makes or breaks the whole thing.

To give a self-hosted OpenClaw agent inbound phone calls: enable the native voice-call plugin in ~/.openclaw/openclaw.json, configure an allowFrom caller-ID allowlist, point a Twilio (or Telnyx/Plivo) inbound number at your public webhook URL, and set the greeting and TTS voice per-number. For tool access during realtime voice calls — so the agent can use calendar, memory, and send-message mid-conversation — OpenClaw's native realtime mode now exposes the openclaw_agent_consult tool (shipped in PR #71272). For a different architecture that routes the entire call through the gateway agent turn (maximum tool fidelity, higher latency), see github.com/CODEANDTRUST/clawcall.

Why Your Agent Should Answer the Phone

Chat and API interfaces are great for deliberate interactions. Phone calls are different: they're ambient, interruptive, and represent a class of real-world demand that most agents completely ignore. The use cases that become possible with inbound voice are qualitatively different from what a chat interface can do:

• Voicemail replacement. Your agent answers, takes the message, transcribes it, extracts action items, and adds them to your task list — without you lifting a finger.
• Live scheduling. A caller asks to book a meeting; the agent checks your calendar (via its calendar tool), offers real times, and creates the event — all in one phone call.
• On-call triage. For engineering teams, an on-call phone line where the agent gathers incident details, runs diagnostic commands, and pages the right human only when needed.
• Personal assistant routing. The agent answers, decides based on who's calling and what they want, and either handles it completely or forwards to you with a briefing.

None of this is science fiction — the plumbing exists today. The challenge is wiring it correctly so the agent's tools remain accessible during the call, which is where most integrations have historically failed.

The Options at a Glance

There are three broad approaches to adding inbound calls to an OpenClaw agent. Each trades control for complexity in a different way:

Approach	Setup effort	Tool access during call	Latency	Best for
Native voice-call plugin (realtime mode)	Low — config only	Good via consult tool (PR #71272)	Low (~200–500ms, end-to-end audio)	Conversational feel + tool access; newest path
Community bridges (deepclaw / clawphone)	Medium — deploy a service	Partial	Medium–high	Deepgram STT users, SMS+voice combos
DIY Twilio Media Streams + gateway routing	High — build the pipeline	Full (routed through gateway turn)	Medium (~700ms–1.3s)	Maximum control, auditability, custom pipeline

If you're just getting started and want to prove the concept, the native plugin with realtime mode is the right first step — it now has tool access via the consult tool. If you need every transcript and tool invocation to flow through the full gateway agent turn (for auditability, custom middleware, or precise session control), the DIY path or the clawcall gateway routing layer is the better architecture.

Option 1: The Native voice-call Plugin

OpenClaw ships a voice-call plugin (documented at docs.openclaw.ai/plugins/voice-call) that handles inbound calls from Twilio, Telnyx, and Plivo. It runs inside the gateway process and requires a publicly reachable webhook URL — either a reverse proxy to your local machine (ngrok, Cloudflare Tunnel) or a VPS deployment.

Prerequisites

• A running OpenClaw gateway with a public HTTPS webhook URL
• A Twilio account with at least one purchased phone number
• OpenClaw config at ~/.openclaw/openclaw.json (the standard config location)

Step 1 — Enable the plugin in openclaw.json

The voice-call plugin is disabled by default. Add it to your plugins array and configure the inbound block:

{
  "gateway": {
    "host": "0.0.0.0",
    "port": 3000
  },
  "plugins": [
    {
      "name": "voice-call",
      "enabled": true,
      "config": {
        "provider": "twilio",
        "inbound": {
          "enabled": true,
          "webhookPath": "/voice/inbound",
          "allowPolicy": "allowlist",
          "allowFrom": [
            "+18432965626",
            "+12025551234"
          ]
        },
        "numbers": [
          {
            "number": "+18438846222",
            "greeting": "You've reached the AI assistant. How can I help you today?",
            "persona": "assistant",
            "ttsVoice": "Polly.Joanna"
          }
        ],
        "twilio": {
          "accountSid": "${TWILIO_ACCOUNT_SID}",
          "authToken": "${TWILIO_AUTH_TOKEN}"
        }
      }
    }
  ]
}

Key fields to understand:

• allowPolicy: "allowlist" — only callers in allowFrom can reach the agent. Without this, any caller can trigger the agent. Never deploy inbound calls with allowPolicy: "open" on a production agent.
• numbers[].persona — maps to an OpenClaw persona definition. If omitted, the default persona is used.
• numbers[].ttsVoice — any Amazon Polly voice name, or a Telnyx/Plivo equivalent depending on your provider.

Step 2 — Point Twilio at your webhook

In the Twilio Console, navigate to your phone number's configuration and set the "A call comes in" webhook to:

https://your-gateway-domain.com/voice/inbound

Set the HTTP method to POST. Twilio sends a standard TwiML request body with the caller's number (From), your number (To), and the call SID.

Step 3 — Handle the TwiML response

The voice-call plugin handles TwiML generation automatically. For a custom response shape, you can override the plugin's TwiML handler. A minimal manual example:

<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Say voice="Polly.Joanna">
    You've reached the AI assistant. How can I help?
  </Say>
  <Gather input="speech" action="/voice/process" method="POST"
         speechTimeout="auto" language="en-US">
  </Gather>
</Response>

The <Gather input="speech"> element captures the caller's speech and posts it to /voice/process, where OpenClaw's STT pipeline transcribes it and routes it to the agent.

Step 4 — Restart the gateway and test

# Restart the gateway to pick up config changes
openclaw restart

# Tail the gateway log to watch inbound call events
openclaw logs --follow --filter voice-call

Call your Twilio number from an allowlisted caller. You should see the call event in the log, hear the greeting, and be able to speak to the agent.

Option 2: Community Bridges

Two community projects extend OpenClaw's voice capability beyond what the native plugin provides:

deepclaw (github.com/deepgram/deepclaw) is a Deepgram-backed bridge that replaces the plugin's default STT with Deepgram's streaming transcription API. This matters for latency-sensitive applications: Deepgram's streaming model returns partial transcripts in real time, enabling barge-in (the caller can interrupt the agent mid-sentence) rather than waiting for end-of-speech detection. The tradeoff is a Deepgram API dependency and a separate service to deploy.

clawphone (github.com/ranacseruet/clawphone) is a Twilio Voice + SMS bridge that wraps both channels. It uses Twilio's <Say> verb for TTS (no external TTS provider needed) and adds SMS fallback — if the call fails, the agent can follow up by text. Useful for notification-heavy workflows where voice and SMS are complementary.

Neither bridge fully replaces the native plugin's tool-access capabilities or the full-turn routing of a custom Media Streams pipeline. They improve STT quality and add channel coverage but do not inherently route through OpenClaw's gateway agent turn where tools are available.

Option 3: DIY Twilio Media Streams

Twilio Media Streams gives you a raw WebSocket audio feed from the call — 8kHz mulaw PCM, bidirectional. You build everything above the transport layer: STT, agent routing, TTS, and response injection. Maximum control, maximum complexity.

This is the right choice when you need:

• Sub-500ms end-to-end latency (requires streaming STT + streaming TTS)
• Full access to all gateway agent tools mid-call (calendar, memory, send-message, etc.), with every turn going through the full agent runtime
• Custom barge-in behavior
• Multi-party call routing
• A complete audit trail of every STT → agent turn → TTS exchange

The TwiML to open a Media Stream

<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://your-gateway-domain.com/voice/stream">
      <Parameter name="callSid" value="{{ CallSid }}" />
      <Parameter name="from" value="{{ From }}" />
    </Stream>
  </Connect>
</Response>

The WebSocket audio handler (Node.js)

// voice-stream-handler.js
// Handles Twilio Media Stream WebSocket — pipes audio to Deepgram STT,
// routes transcript to OpenClaw gateway agent, injects TTS audio response.

import WebSocket from 'ws'
import { createClient } from '@deepgram/sdk'
import ElevenLabs from 'elevenlabs-node'
import fetch from 'node-fetch'

const deepgram = createClient(process.env.DEEPGRAM_API_KEY)
const elevenlabs = new ElevenLabs({ apiKey: process.env.ELEVENLABS_API_KEY })

// OpenClaw gateway base URL — the same instance your agent runs on
const OPENCLAW_GATEWAY = process.env.OPENCLAW_GATEWAY_URL || 'http://localhost:3000'

export function handleVoiceStream(ws) {
  let streamSid = null
  let callSid = null
  let dgSocket = null

  // Open a Deepgram streaming transcription connection
  const startDeepgram = () => {
    dgSocket = deepgram.listen.live({
      model: 'nova-2-phonecall',
      encoding: 'mulaw',
      sample_rate: 8000,
      language: 'en-US',
      interim_results: true,
      endpointing: 300,          // ms of silence = end of utterance
      utterance_end_ms: 1000,    // fallback utterance boundary
    })

    dgSocket.on('Results', async (data) => {
      const transcript = data.channel?.alternatives?.[0]?.transcript
      const isFinal = data.is_final

      if (!transcript || !isFinal) return

      // Route the transcript through the OpenClaw gateway agent turn
      // so that calendar, memory, and send-message tools are available
      const agentResponse = await routeToOpenClawAgent(transcript, callSid)

      // Convert the agent's text response to audio and inject it
      const audio = await elevenlabs.generate({
        voice: 'Rachel',
        text: agentResponse,
        model_id: 'eleven_turbo_v2',   // lowest-latency ElevenLabs model
      })
      injectAudio(ws, streamSid, audio)
    })
  }

  ws.on('message', (raw) => {
    const msg = JSON.parse(raw)

    switch (msg.event) {
      case 'start':
        streamSid = msg.start.streamSid
        callSid = msg.start.callSid
        startDeepgram()
        break

      case 'media':
        // Twilio sends base64-encoded mulaw audio chunks
        if (dgSocket?.getReadyState() === WebSocket.OPEN) {
          dgSocket.send(Buffer.from(msg.media.payload, 'base64'))
        }
        break

      case 'stop':
        dgSocket?.finish()
        break
    }
  })
}

// Route a transcript through the OpenClaw gateway's /agent/turn endpoint.
// This is the key step — hitting the gateway turn (not a standalone LLM call)
// means all of the agent's registered tools are available during inference.
async function routeToOpenClawAgent(transcript, callSid) {
  const res = await fetch(`${OPENCLAW_GATEWAY}/agent/turn`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      message: transcript,
      sessionId: `voice-${callSid}`,   // scoped session per call
      channel: 'voice',
    }),
  })
  const data = await res.json()
  return data.response
}

// Inject synthesized audio back into the Twilio stream
function injectAudio(ws, streamSid, audioBuffer) {
  const b64 = audioBuffer.toString('base64')
  ws.send(JSON.stringify({
    event: 'media',
    streamSid,
    media: { payload: b64 },
  }))
}

Latency notes for this pipeline:

• Deepgram nova-2-phonecall: ~200–300ms for final transcript after end of utterance
• OpenClaw gateway agent turn: 300–600ms depending on model and tool calls
• ElevenLabs eleven_turbo_v2: ~200–400ms to first audio byte (streaming)
• Total perceived latency: 700ms–1.3s, competitive with managed platforms like Vapi

Tool Access During Calls — Two Architectures (Issue #71262, Resolved)

This is the part most voice integration guides skip, and it's the most important design decision you'll make.

OpenClaw's gateway maintains a tool registry — all the capabilities your agent has: calendar access, memory read/write, send-message, browser control, code execution, and whatever custom tools you've added. These tools are available during a normal agent turn (chat, API call, cron trigger). The question is: are they available during a voice call?

This was an open problem captured in GitHub issue #71262: the native voice-call plugin's realtime mode (end-to-end audio via OpenAI Realtime / Gemini Live) ran as a self-contained session that could not invoke gateway-registered tools — registerToolHandler was wired but never populated at runtime, so every realtime tool call returned { error: "Tool not available" }.

This is fixed. PR #71272 ("feat(voice-call): share realtime agent consult tool") merged and closed/locked issue #71262. The fix ships the openclaw_agent_consult tool into the voice-call realtime runtime — the same consult tool already used by the Google Meet and browser Talk integrations. During a realtime voice session, when the model needs to perform an action (calendar query, memory lookup, send message), it calls openclaw_agent_consult, which runs an embedded agent turn against the gateway's tool surface and returns the result to the realtime LLM. The realtime session then speaks the result back to the caller.

You control which tools the consult agent can reach via the realtime.toolPolicy config key:

• "none" — no tool access (pre-PR behavior)
• "safe-read-only" — default; memory search, web search/fetch, calendar read
• "owner" — full tool surface (use with care on a public phone line)

The related issues that tracked this problem:

• #7200 + #8088 — requests for real-time bidirectional voice with full tool access. Both are open but unscheduled.
• #5147 — SIP/VoIP support. Closed as "not planned" — the team is focused on PSTN via Twilio/Telnyx rather than SIP.
• #71262 — tool-access gap in realtime voice mode. Closed — fixed by PR #71272 (merged, commit e2f13959d4).

Native Realtime + Consult Tool vs. Full Gateway-Turn Routing — Trade-offs

Now that OpenClaw ships the consult tool natively, there are two legitimate architectures for tool-capable voice calls. Understanding the trade-offs helps you pick the right one:

Architecture	How tools are reached	Conversational latency	Tool-call fidelity	Control surface
Native realtime + openclaw_agent_consult	Realtime LLM decides to call openclaw_agent_consult; embedded agent turn runs against gateway tools	Very low — end-to-end audio, sub-500ms conversational feel	Good — consult tool reaches the same tool surface; one extra hop vs. direct turn	toolPolicy config; consult runs as a nested agent turn
DIY Media Streams + /agent/turn (e.g. clawcall)	Every transcript POST hits the gateway agent turn directly — STT → chat.send → full agent run with tools → TTS	Medium — add STT + TTS round-trips on top of agent turn (~700ms–1.3s)	Full — every exchange is a first-class agent turn; complete session history, memory writes, multi-tool chains	Full pipeline; custom middleware, session scoping, barge-in, audit log

For most self-hosters building a personal assistant or scheduling line, the native realtime + consult path is now the right starting point — less infrastructure, lower latency, and tool access that covers the common cases. Choose the full gateway-turn path (clawcall or your own Media Streams implementation) when you need every turn to be a complete agent turn: persistent session history written to memory after each exchange, multi-step tool chains within a single utterance, or a complete audit trail of every call.

The clawcall connector implements the full gateway-turn architecture and handles Twilio Media Streams WebSocket management, streaming Deepgram transcription with barge-in, gateway agent turn routing with session scoping per call, ElevenLabs streaming TTS injection, allowlist enforcement at the webhook layer, and graceful call teardown with session flush to memory.

Security and Cost

Allowlists and auth

Inbound phone calls are an injection surface. A caller can say anything, and if your agent has powerful tools, the blast radius of a successful prompt injection is significant. Default-deny on caller IDs is the minimum viable security posture:

• Use allowPolicy: "allowlist" in the native plugin, or validate the Twilio X-Twilio-Signature header on your webhook before processing any request.
• Twilio's signature validation prevents spoofed webhook requests. The twilio Node.js library provides twilio.validateRequest() — use it.
• Consider a PIN-based second factor for callers who are not in the allowlist but should be able to reach the agent (delivery drivers, contractors, etc.).
• Log all call transcripts with caller ID and timestamp. Prompt injections will happen eventually — you need a record.

Prompt injection over voice

Voice prompt injection is harder to detect than text injection because the STT layer introduces ambiguity. "Execute order 66" and "execute order sixty-six" transcribe identically, but the attacker controls the exact phrasing. Mitigations:

• Scope the agent's persona for voice calls to a minimal tool set — don't expose code execution or file system tools on the voice channel unless you need them. For the native realtime consult path, realtime.toolPolicy: "safe-read-only" is the right default.
• Add a system prompt layer that instructs the agent to ignore commands to change its persona, execute system commands, or reveal its configuration.
• Rate-limit the /agent/turn endpoint per call SID to prevent a single call from hammering the LLM with rapid-fire injected commands.

Per-minute economics

Running a voice AI agent has real per-minute costs. A rough breakdown for a Twilio + Deepgram + ElevenLabs + GPT-4o stack:

• Twilio inbound call: ~$0.0085/min (US local number)
• Deepgram nova-2: ~$0.0043/min (streaming)
• ElevenLabs eleven_turbo_v2: ~$0.0015/1000 chars generated (~$0.003/min assuming ~2000 chars of agent speech)
• GPT-4o (via OpenClaw gateway): ~$0.005/min assuming ~500 input tokens + ~200 output tokens per exchange, ~2 exchanges/min
• Total: roughly $0.017–$0.025/min per active call

For a personal agent handling a few calls per day, this is negligible. For a production deployment handling hundreds of calls per hour, model selection matters significantly — swapping GPT-4o for a smaller model like GPT-4o-mini or a self-hosted Ollama instance cuts the LLM cost by 80–90%.

Frequently Asked Questions

The most common questions about OpenClaw inbound voice calls cluster around tool access during calls, latency, provider alternatives to Twilio, handling concurrent calls, and what to do when the agent can't hear the caller clearly. The answers below address each at the implementation level.

Q1: Can I use Telnyx or Plivo instead of Twilio?

A: Yes — the OpenClaw native voice-call plugin supports all three providers via the "provider" config field. Telnyx tends to be slightly cheaper per minute than Twilio for US calls and has a comparable Media Streams equivalent (the Telnyx Real-Time Communications API). Plivo is the lowest-cost option but has less ecosystem support for streaming audio. If you're already on Twilio, stay there — the documentation, webhook debugging tools, and community support are strongest. If cost is the primary concern, Telnyx is the best alternative.

Q2: My agent works perfectly in chat but is useless on a call. Why?

A: If you're running an older version of OpenClaw (before PR #71272 landed), the native voice-call plugin's realtime mode did not wire gateway tools into the realtime session — the toolHandlers map was always empty, so every tool call returned an error. The fix: upgrade to a release that includes PR #71272, then set realtime.toolPolicy in your plugin config ("safe-read-only" is the right default). If you're on a current version and still seeing no tool access, check that realtime.enabled: true and that realtime.toolPolicy is not "none".

For a fully deterministic alternative where every call exchange flows through the full gateway agent turn, the clawcall connector implements that pipeline — it bypasses the plugin's realtime LLM entirely and posts each transcript to /agent/turn directly.

Q3: How do I handle concurrent inbound calls?

A: Each call gets its own WebSocket connection and a distinct sessionId (scoped to the call SID). The OpenClaw gateway handles concurrent agent turns — they're stateless at the LLM layer and serialized per session. The constraint is your gateway host's compute and your LLM API's rate limits. For high-concurrency deployments, run multiple gateway instances behind a load balancer and partition sessions by hash of call SID.

Q4: The agent mishears the caller constantly. What helps?

A: Switch to Deepgram nova-2-phonecall (the model optimized for telephone audio quality — 8kHz mulaw) if you're not already using it. Enable interim_results: true to see partial transcripts in your logs and identify if the issue is STT accuracy or end-of-speech detection. For strong accents or technical vocabulary, Deepgram supports custom vocabulary boosting via the keywords parameter. If the problem is background noise, Deepgram's noise suppression is enabled by default on nova-2.

Q5: Can I make the agent initiate outbound calls?

A: Yes — Twilio's Calls API lets you POST to /2010-04-01/Accounts/{AccountSid}/Calls to initiate an outbound call with a TwiML webhook URL. The OpenClaw voice-call plugin does not natively support outbound-initiated calls, but you can trigger them from any of the agent's tool slots. Wire a custom make-phone-call tool that hits the Twilio Calls API, then your agent can initiate calls as part of any workflow — including from chat, cron triggers, or email events.

Q6: Is there a managed alternative if I don't want to host the WebSocket layer myself?

A: Yes — Vapi, Retell, and Bland are managed platforms that handle the telephony, STT, and TTS layers and expose an LLM turn endpoint you configure. The tradeoff is that your agent's tool execution stays on their platform unless you build a custom tool-call webhook. For OpenClaw specifically, you'd need to proxy their tool-call webhook to your OpenClaw gateway's /agent/turn to get full tool access. Managed platforms make sense for teams that don't want to operate WebSocket infrastructure.

Q7: How do I test inbound calls without a real Twilio number?

A: Use ngrok http 3000 (or Cloudflare Tunnel) to expose your local gateway, then use Twilio's "Test" phone numbers for outbound-to-your-webhook testing. The Twilio Studio debugger lets you replay webhook requests against your local URL. For unit testing the STT→agent→TTS pipeline without live calls, send raw mulaw audio files to your WebSocket handler using a local WebSocket client — no Twilio required for the audio processing logic.

Build a Voice-Capable Agent with Code and Trust

Adding inbound phone calls to a self-hosted AI agent is a solvable engineering problem, but the implementation surface is wider than it looks: provider plumbing, streaming audio, STT model tuning, gateway routing, tool-access architecture, security hardening, and cost optimization are all real concerns that take time to get right.

If you're evaluating whether to build this in-house or work with a team that has done it before, the Code and Trust AI Audit is a practical starting point. We map your existing workflows and agent capabilities against what a voice integration would unlock — and tell you honestly whether it's worth the build effort for your use case.

For organizations that want to move faster, our AI implementation practice covers the full stack: agent architecture, voice integration, tool wiring, and production deployment — with the operational experience to avoid the gaps that most first implementations hit.

---

Related Guides

• Giving Your AI Agent a Phone Number: Twilio vs Vapi vs Retell vs Self-Hosted (2026) — comparison of every telephony option with a cost/latency/control table.
• How to Add SMS to Your Self-Hosted AI Agent (Twilio + OpenClaw) — extend your voice setup with two-way SMS, opt-in compliance, and cross-channel session management.