MCP Toolbox - Analytics & Monitoring Dashboard

Overview

Table of Contents

• Features
• Quick Start
• Configuration
• Dashboard Overview
• Analytics & Metrics
• Debugging
• Troubleshooting
• Security

Features

Real-time Monitoring

• Live Tool Invocations: Watch MCP tools being called in real-time
• Performance Metrics: Track execution times, success rates, and error rates
• Resource Utilization: Monitor memory, CPU, and network usage
• Active Connections: See which clients are connected to the server

Analytics

• Usage Patterns: Identify most/least used tools and resources
• Performance Trends: Historical data with time-series graphs
• Error Analysis: Track error types, frequencies, and patterns
• Client Statistics: Per-client usage and performance metrics

Debugging

• Request/Response Inspection: View full request and response payloads
• Stack Traces: Detailed error stack traces with context
• Parameter Validation: See parameter validation errors
• Execution Timeline: Step-by-step execution flow visualization

Alerting (Optional)

• Performance Alerts: Get notified when response times exceed thresholds
• Error Rate Alerts: Alert on high error rates
• Custom Alerts: Define custom alerting rules

Quick Start

Prerequisites

• Docker and Docker Compose installed
• UniFi API key from unifi.ui.com
• Agnost Organization ID from app.agnost.ai

Setup Steps

# 1. Clone the repository (if not already done)
git clone https://github.com/enuno/unifi-mcp-server.git
cd unifi-mcp-server

# 2. Copy environment template
cp .env.docker.example .env

# 3. Edit .env file with your credentials
nano .env  # or use your preferred editor

# UniFi API Configuration
UNIFI_API_KEY=your-unifi-api-key-here

# Agnost Configuration (optional)
AGNOST_ENABLED=true
AGNOST_ORG_ID=your-organization-id-here

# Toolbox telemetry export (optional; off when unset)
TOOLBOX_TELEMETRY_OTLP=https://api.agnost.ai/v1/otel/your-organization-id-here

# 4. Start all services
docker-compose up -d

# 5. Verify services are running
docker-compose ps

# 6. Access the Toolbox dashboard
open http://localhost:8080

Stopping Services

# Stop all services
docker-compose down

# Stop and remove volumes (includes Redis data)
docker-compose down -v

Configuration

Environment Variables

Basic Configuration

Toolbox Configuration

Privacy Controls

Port Configuration

# .env file
TOOLBOX_PORT=9090

Authentication

# .env file
TOOLBOX_AUTH_ENABLED=true
TOOLBOX_USERNAME=admin
TOOLBOX_PASSWORD=your-secure-password-here

Dashboard Overview

Home Page

• Server Status: Uptime, version, configuration
• Tool Statistics: Total tools, most used, success rate
• Resource Statistics: Resource types, access counts
• Recent Activity: Last 10 tool invocations with timing

Tools Tab

• Tool List: All available tools with descriptions
• Usage Statistics: Call count, avg response time, error rate
• Performance Graphs: Response time distribution, success/failure trends
• Tool Details: Click any tool to see detailed analytics

Resources Tab

• Resource URIs: All available resource endpoints
• Access Patterns: Most accessed resources, access frequency
• Performance: Response times, cache hit rates
• Resource Details: Click to see access history and performance

Errors Tab

• Error List: Recent errors with timestamps
• Error Types: Group by error type or tool
• Stack Traces: Full stack traces with code context
• Error Trends: Error frequency over time
• Quick Filters: Filter by tool, error type, or time range

Analytics Tab

• Performance Metrics:
  • Average response time by tool
  • P50, P95, P99 percentiles
  • Throughput (requests per second)
  • Error rates and types
• Usage Metrics:
  • Tool popularity (most/least used)
  • Peak usage times
  • Client distribution
  • Request patterns
• Time-series Graphs:
  • Response time trends
  • Request volume over time
  • Error rate trends
  • Resource utilization

Analytics & Metrics

Key Performance Indicators (KPIs)

Response Time Metrics

• Average Response Time: Mean execution time across all tools
• Median (P50): 50th percentile response time
• P95: 95th percentile - 95% of requests faster than this
• P99: 99th percentile - 99% of requests faster than this

• < 100ms: Excellent
• 100-500ms: Good
• 500ms-1s: Acceptable
• > 1s: Needs optimization

Success Rate

• Overall Success Rate: Percentage of successful tool invocations
• Per-Tool Success Rate: Success rate for individual tools

• > 99%: Excellent
• 95-99%: Good
• 90-95%: Needs attention
• < 90%: Critical

Throughput

• Requests Per Second (RPS): Number of tool invocations per second
• Requests Per Minute (RPM): Number of tool invocations per minute

Tool-Specific Metrics

1. Invocation Count: Total number of times called
2. Avg Response Time: Average execution time
3. Success Rate: Percentage of successful calls
4. Error Count: Number of failed calls
5. Last Used: Timestamp of most recent invocation

Resource Metrics

1. Access Count: Number of times accessed
2. Cache Hit Rate: Percentage of cached responses
3. Avg Response Size: Average size of responses
4. Response Time: Average time to fetch resource

Debugging

Request/Response Inspection

• Tool name
• Input parameters (if tracking enabled)
• Timestamp
• Client information
• Headers (if available)

• Output data (if tracking enabled)
• Response time
• Success/failure status
• Error message (if failed)
• Stack trace (if error occurred)

Error Analysis

1. Error Grouping: Errors grouped by type or tool
2. Stack Traces: Full Python stack traces with line numbers
3. Context: Request parameters that caused the error
4. Frequency: How often each error occurs
5. Recent Occurrences: Timeline of when errors happened

Performance Profiling

1. Slowest Tools: List of tools with highest avg response time
2. Performance Timeline: Visual timeline of execution
3. Bottleneck Identification: Pinpoint slow operations
4. Comparison: Compare performance across time ranges

Live Monitoring

• Live Feed: Stream of tool invocations as they happen
• Active Requests: Currently executing tools
• Connection Status: Connected clients and their status
• Resource Usage: Live CPU, memory, and network stats

Troubleshooting

Toolbox Dashboard Not Accessible

1. Check if Toolbox container is running:
   docker-compose ps mcp-toolbox
   
2. Check Toolbox logs:
   docker-compose logs mcp-toolbox
   
3. Verify port is not in use:
   lsof -i :8080  # macOS/Linux
   netstat -ano | findstr :8080  # Windows
   
4. Try a different port:
   TOOLBOX_PORT=9090
   

No Data Showing in Dashboard

1. Verify agnost tracking is enabled:
   AGNOST_ENABLED=true
   AGNOST_ORG_ID=your-org-id-here
   
2. Check MCP server logs for tracking errors:
   docker-compose logs unifi-mcp | grep -i agnost
   
3. Verify MCP server is making tool calls:
   docker-compose logs unifi-mcp | grep -i "tool"
   
4. Check Organization ID is correct in agnost.ai dashboard

Authentication Issues

1. Verify credentials in .env:
   TOOLBOX_AUTH_ENABLED=true
   TOOLBOX_USERNAME=admin
   TOOLBOX_PASSWORD=your-password
   
2. Restart Toolbox container:
   docker-compose restart mcp-toolbox
   
3. Disable authentication temporarily for testing:
   TOOLBOX_AUTH_ENABLED=false
   

High Memory Usage

1. Reduce data retention period (configure in agnost.ai dashboard)
2. Enable input/output tracking controls:
   AGNOST_DISABLE_INPUT=true
   AGNOST_DISABLE_OUTPUT=true
   
3. Add memory limits to docker-compose.yml:
   mcp-toolbox:
     mem_limit: 512m
   

Security

Best Practices

1. Enable Authentication
   TOOLBOX_AUTH_ENABLED=true
   TOOLBOX_PASSWORD=strong-random-password
   
2. Use Strong Passwords
   • Minimum 16 characters
   • Mix of uppercase, lowercase, numbers, and symbols
   • Use a password manager
3. Restrict Network Access
   # docker-compose.yml
   mcp-toolbox:
     ports:
       - "127.0.0.1:8080:8080"  # Only localhost
   
4. Enable HTTPS (for production)
   • Use a reverse proxy (nginx, Caddy, Traefik)
   • Configure SSL certificates
   • Redirect HTTP to HTTPS
5. Control Data Tracking
   AGNOST_DISABLE_INPUT=true   # Don't track input params
   AGNOST_DISABLE_OUTPUT=true  # Don't track output data
   

Data Privacy

• Tool names and execution times (always)
• Input parameters (if AGNOST_DISABLE_INPUT=false)
• Output results (if AGNOST_DISABLE_OUTPUT=false)
• Error messages and stack traces (always)

• UniFi API keys (always masked)
• Passwords (always masked)
• Authentication tokens (always masked)
• Personal data (unless explicitly in parameters)

• Default: 30 days
• Configurable in agnost.ai dashboard
• Can be deleted manually at any time

Network Security

# Allow Toolbox only from local network
iptables -A INPUT -p tcp --dport 8080 -s 192.168.1.0/24 -j ACCEPT
iptables -A INPUT -p tcp --dport 8080 -j DROP

• Use VPN for remote access
• Deploy behind a firewall
• Use reverse proxy with SSL
• Enable rate limiting
• Monitor access logs

Advanced Configuration

Custom Agnost Endpoint

AGNOST_ENDPOINT=https://your-agnost-instance.com

Docker Compose Overrides

version: '3.8'

services:
  mcp-toolbox:
    environment:
      - CUSTOM_VAR=value
    volumes:
      - ./custom-config:/config
    deploy:
      resources:
        limits:
          memory: 512M
          cpus: '0.5'

Health Checks

# Check all services
docker-compose ps

# Check specific service health
docker inspect --format='{{.State.Health.Status}}' mcp-toolbox

# View health check logs
docker inspect --format='{{json .State.Health}}' mcp-toolbox | jq

Support

Resources

• Agnost.ai Documentation: docs.agnost.ai
• MCP Toolbox GitHub: googleapis/genai-toolbox
• UniFi MCP Server: enuno/unifi-mcp-server

Getting Help

• GitHub Issues: Report bugs or request features
• GitHub Discussions: Ask questions and share ideas
• Agnost Support: support@agnost.ai

FAQ

docker-compose pull mcp-toolbox
docker-compose up -d mcp-toolbox