How Developers Build and Host API-Based Server Bots on Poe to Deploy Custom Models

Poe Server Bots allow developers to connect external computing infrastructure, custom machine learning models, and proprietary logic directly to Quora's Poe platform using a standardized HTTP protocol based on Server-Sent Events (SSE). Unlike standard prompt bots that rely on Poe's hosted commercial models with custom system prompts, server bots give engineering teams complete control over backend inference engines, private weights, retrieval-augmented generation (RAG) pipelines, and external API integrations. By wrapping custom models in the lightweight fastapi_poe Python library and deploying the service to a publicly reachable endpoint, developers can instantly distribute proprietary AI tools to Poe's global user base across web, iOS, Android, and desktop apps.

Architecture and the Poe Server Bot Protocol

The communication between Poe and your custom backend follows an asynchronous client-server model over HTTPS. When a user sends a message to your server bot on Poe, the Poe infrastructure forwards the message context as an HTTP POST request to your designated server endpoint. Your server processes the incoming payload and streams the generated response back to Poe using Server-Sent Events (SSE). This architecture ensures low latency and provides real-time token streaming to the end user.

The protocol specification standardizes several event and request types:

• QueryRequest: Contains the full conversational context, including past messages in the thread (query), the unique conversation identifier (conversation_id), the user identifier (user_id), message metadata, and client capabilities.
• PartialResponse: The atomic streaming chunk emitted by your server. Each partial response yields incremental text, data payloads, or state updates to the client.
• SettingsRequest / SettingsResponse: Allows your server to negotiate bot-level configurations with Poe, such as introduction greetings, file attachment permissions, suggested replies, and dependency declarations for bot-to-bot querying.
• ReportFeedbackRequest and ReportErrorRequest: Provide telemetry and feedback callbacks from Poe to your server when users rate messages or when client-side transmission errors occur.

To authenticate requests and protect your backend from unauthorized traffic, Poe issues an Access Key. Your server verifies this key against the Authorization: Bearer <access_key> header sent with every incoming request.

Setting Up the Development Environment

The official fastapi_poe library provides the fastest route to implementing a compliant server bot. It abstracts HTTP parsing, event serialization, and authentication into an extensible asynchronous interface.

Begin by installing fastapi-poe along with an ASGI server such as Uvicorn:

pip install fastapi-poe uvicorn

A minimal server bot subclasses PoeBot and implements the get_response generator:

import fastapi_poe as fp
from typing import AsyncIterable

class EchoBot(fp.PoeBot):
    async def get_response(
        self, request: fp.QueryRequest
    ) -> AsyncIterable[fp.PartialResponse]:
        last_message = request.query[-1].content
        yield fp.PartialResponse(text=f"Received: {last_message}")

    async def get_settings(
        self, setting: fp.SettingsRequest
    ) -> fp.SettingsResponse:
        return fp.SettingsResponse(
            introduction_message="Welcome! This bot is running on a custom backend.",
            allow_attachments=False
        )

bot = EchoBot()
app = fp.make_app(bot, access_key="YOUR_SECRET_ACCESS_KEY")

if __name__ == "__main__":
    fp.run(bot, access_key="YOUR_SECRET_ACCESS_KEY", port=8080)

During local prototyping, you can expose your local port (e.g., 8080) to the public internet using a tunneling tool like ngrok or Cloudflare Tunnels:

ngrok http 8080

This generates a temporary public HTTPS URL (e.g., https://abcdef123.ngrok-free.app) that you can feed into Poe's bot registration interface for immediate testing.

Deploying Custom Models and Inference Pipelines

Connecting a server bot to a custom model involves bridging incoming QueryRequest messages to an inference engine and streaming the generated tokens out as PartialResponse objects.

Integrating with a vLLM or Hugging Face Inference Backend

When running open-weights models (such as Llama 3, Mistral, or Qwen) on dedicated GPU hardware, serving them through an optimized engine like vLLM or Hugging Face Text Generation Inference (TGI) delivers high throughput and continuous batching.

The following example illustrates a Poe server bot forwarding requests to an OpenAI-compatible vLLM inference server:

import os
import fastapi_poe as fp
from openai import AsyncOpenAI
from typing import AsyncIterable

# Connect to local or remote vLLM endpoint
client = AsyncOpenAI(
    base_url=os.environ.get("VLLM_BASE_URL", "http://localhost:8000/v1"),
    api_key=os.environ.get("VLLM_API_KEY", "EMPTY"),
)

class CustomModelBot(fp.PoeBot):
    async def get_response(
        self, request: fp.QueryRequest
    ) -> AsyncIterable[fp.PartialResponse]:
        # Convert Poe conversation history to chat messages
        messages = []
        for message in request.query:
            role = "user" if message.role == "user" else "assistant"
            messages.append({"role": role, "content": message.content})

        try:
            stream = await client.chat.completions.create(
                model="meta-llama/Meta-Llama-3-8B-Instruct",
                messages=messages,
                stream=True,
                temperature=0.7,
                max_tokens=1024,
            )

            async for chunk in stream:
                token = chunk.choices[0].delta.content
                if token:
                    yield fp.PartialResponse(text=token)

        except Exception as e:
            yield fp.ErrorResponse(
                text=f"An error occurred during inference: {str(e)}",
                allow_retry=True
            )

    async def get_settings(
        self, setting: fp.SettingsRequest
    ) -> fp.SettingsResponse:
        return fp.SettingsResponse(
            introduction_message="Powered by a custom open-source model running on dedicated infrastructure.",
            enable_image_comprehension=False,
            allow_attachments=False
        )

bot = CustomModelBot()
app = fp.make_app(bot, access_key=os.environ.get("POE_ACCESS_KEY"))

Handling Multi-Turn Context and Attachments

Poe sends the full dialogue history in the request.query array, ordered chronologically. If your custom model has a limited context window, your server bot logic should prune or summarize earlier turns before tokenization.

When your bot enables allow_attachments=True in get_settings, file uploads (images, PDFs, text documents) appear in request.query[-1].attachments. Each attachment object contains a download URL and MIME type, allowing your backend to download the asset, perform OCR, run vision encoding, or embed the document into a vector store before generating a response.

Hosting Options for Server Bots

Because Poe requires an uninterrupted HTTPS endpoint capable of streaming SSE responses, selecting the right hosting platform depends on your model size, compute requirements, and budget.

Serverless GPU Deployment with Modal

Modal provides first-class support for Poe server bots, allowing developers to define GPU infrastructure as code and pay only for actual execution time. Modal automatically scales container instances from zero and provides an HTTPS URL.

A production-ready Modal deployment script:

import os
import modal
import fastapi_poe as fp

app = modal.App("poe-custom-model-bot")

# Define container environment with dependencies
image = (
    modal.Image.debian_slim()
    .pip_install("fastapi-poe", "openai")
)

@app.function(
    image=image,
    secrets=[modal.Secret.from_name("poe-custom-model-secrets")],
    keep_warm=1,  # Keeps one instance ready to prevent cold starts
)
@modal.asgi_app()
def fastapi_app():
    from bot_implementation import CustomModelBot
    bot = CustomModelBot()
    return fp.make_app(bot, access_key=os.environ["POE_ACCESS_KEY"])

Deploying this service with modal deploy app.py outputs a permanent URL such as: https://<organization>--poe-custom-model-bot-fastapi-app.modal.run.

Containerized Hosting on Cloud Providers (AWS, GCP, Render, Fly.io)

For teams running proprietary inference clusters or leveraging managed container platforms:

1. Dockerize the Application: Create a lightweight Dockerfile running uvicorn app:app --host 0.0.0.0 --port 8080 --workers 4.
2. Reverse Proxy and SSL: Deploy behind an ingress controller (such as AWS ALB, Cloudflare, or Nginx) that supports long-lived HTTP connections and streaming chunked responses. Ensure proxy timeouts (e.g., proxy_read_timeout) are configured for at least 300 seconds to prevent mid-stream disconnections.
3. Environment Secrets: Inject POE_ACCESS_KEY and inference credentials securely through container environment variables.

Registering and Configuring the Bot on Poe

Once your server is deployed and accessible via public HTTPS, complete the registration on the Poe platform:

1. Navigate to poe.com/create_bot?server=1.
2. Bot Handle: Choose a unique alphanumeric identifier (e.g., @MyCustomModel). This forms the public link for your bot.
3. Bot Type: Ensure "Server bot" is selected.
4. Server URL: Enter your full public endpoint (e.g., https://your-domain.com/ or your Modal deployment URL).
5. Access Key: Paste the secret token configured in your server's fp.make_app() call. Poe will validate this token with every test request.
6. Profile Details: Set your display name, description, avatar, and system instructions visible to users.
7. Visibility: Select whether the bot should remain private (accessible only to you or designated team members) or public (searchable in the Poe Explore directory).

Click Create Bot. Poe immediately verifies connectivity by issuing a SettingsRequest to your endpoint. If the server returns a valid response, the bot is activated.

Monetization and Compute Point Economics

Poe provides native creator monetization for server bots, allowing developers to earn revenue as users interact with their tools.

Per-Message Pricing and the Bot Monetization API

Creators can set a fixed compute point cost per message in the bot settings, or utilize the Bot Monetization API for dynamic, usage-based billing:

• Subscription Revenue Share: When new users subscribe to Poe after interacting with your bot, you receive an affiliate payout.
• Compute Points Consumption: When existing Poe subscribers or free users spend compute points on your bot, you earn revenue proportionate to the points consumed.
• Variable Pricing: If your model performs tasks with unpredictable computational costs (e.g., high-resolution image rendering or long-context document analysis), you can configure variable pricing using authorize_cost and capture_cost endpoints. Users are prompted with a clear rate_card or cost_label before high-cost operations execute.

# Setting dynamic rate cards in SettingsResponse
async def get_settings(self, setting: fp.SettingsRequest) -> fp.SettingsResponse:
    return fp.SettingsResponse(
        introduction_message="Professional analysis bot with dynamic compute points.",
        cost_label="50-200 pts",
        rate_card="* Base Query: 50 points\n* Extended Analysis: 150-200 points"
    )

Production Best Practices

Deploying a reliable server bot requires addressing network volatility, latency budgets, and operational telemetry:

• Implement Immediate Streaming: Users perceive low latency when the first token appears quickly. Avoid buffering entire model generations in memory before yielding. Start emitting tokens as soon as the first chunk arrives from your inference backend.
• Graceful Error Recovery: Wrap model calls in structured exception handlers. If an upstream GPU worker fails, return an fp.ErrorResponse with a clear explanation and set allow_retry=True so the user can re-trigger the query without reloading the app.
• Suggested Follow-Ups: Enhance engagement by appending suggested replies via fp.MetaResponse(suggested_replies=["Explain further", "Provide code sample"]) at the conclusion of a response.
• Monitor Telemetry: Implement the on_feedback method on your PoeBot class to log positive (thumbs up) and negative (thumbs down) ratings directly into your observability platform (e.g., Datadog, Prometheus, or Grafana).