Monitoring & Maintenance

This guide covers monitoring the bot's health and performing regular maintenance tasks.

Health Monitoring

Real-time Status Monitoring

Using PM2

If deployed with PM2, use built-in monitoring:

# View process status
pm2 status

# Real-time monitoring dashboard
pm2 monit

# View logs
pm2 logs somnia-bot --lines 100

# View detailed process info
pm2 describe somnia-bot

Using Systemd

For systemd deployments:

# Check service status
sudo systemctl status somnia-bot

# View recent logs
sudo journalctl -u somnia-bot -n 100

# Follow logs in real-time
sudo journalctl -u somnia-bot -f

Key Metrics to Monitor

1. Process Health

  • Uptime: Should be continuously running
  • Restarts: Frequent restarts indicate issues
  • Memory Usage: Should stay under 1GB
  • CPU Usage: Should be minimal except during checks

2. Bot Responsiveness

  • Command response time < 1 second
  • Monitoring cycle completion time
  • Queue size (if using Redis)

3. Database Health

  • Database file size
  • Query execution time
  • Number of active subscriptions

4. External Services

  • Telegram API connectivity
  • Blockchain RPC availability
  • Redis connection (if used)

Monitoring Tools

Application Logs

The bot uses Winston logger with daily rotation:

# View today's logs
tail -f logs/bot-$(date +%Y-%m-%d).log

# Search for errors
grep ERROR logs/bot-*.log

# Monitor specific user
grep "userId:123456789" logs/bot-*.log

Performance Monitoring

Memory Usage Tracking

# Create monitoring script
cat > monitor-memory.sh << 'EOF'
#!/bin/bash
while true; do
  echo "$(date): $(ps aux | grep "node.*bot.js" | awk '{print $6}') KB"
  sleep 60
done
EOF

chmod +x monitor-memory.sh
./monitor-memory.sh > memory-usage.log &

Response Time Monitoring

# Extract response times from logs
grep "Response time:" logs/bot-*.log | \
  awk '{print $NF}' | \
  awk '{sum+=$1; count++} END {print "Average:", sum/count, "ms"}'

Database Monitoring

# Check database size
du -h somnia_validator_bot.db

# Monitor growth
watch -n 60 'du -h somnia_validator_bot.db'

# Check table sizes
sqlite3 somnia_validator_bot.db << EOF
SELECT name, COUNT(*) as rows FROM subscriptions;
SELECT name, COUNT(*) as rows FROM validator_states;
SELECT name, COUNT(*) as rows FROM banned_users;
EOF

# Check slow queries
sqlite3 somnia_validator_bot.db << EOF
.timer on
SELECT * FROM subscriptions WHERE chatId = 123456789;
.timer off
EOF

Regular Maintenance Tasks

Daily Tasks

1. Log Review

# Check for errors
grep -c ERROR logs/bot-$(date +%Y-%m-%d).log

# Check for warnings
grep -c WARN logs/bot-$(date +%Y-%m-%d).log

# Review admin actions
grep "Admin action" logs/bot-$(date +%Y-%m-%d).log

2. Performance Check

  • Monitor command response times
  • Check memory usage trends
  • Verify monitoring is running

Weekly Tasks

1. Database Optimization

# Backup database
cp somnia_validator_bot.db backups/bot-$(date +%Y%m%d).db

# Optimize database
sqlite3 somnia_validator_bot.db "VACUUM;"
sqlite3 somnia_validator_bot.db "ANALYZE;"

# Check integrity
sqlite3 somnia_validator_bot.db "PRAGMA integrity_check;"

2. Log Cleanup

# Compress old logs
find logs -name "*.log" -mtime +7 -exec gzip {} \;

# Remove very old logs
find logs -name "*.gz" -mtime +30 -delete

3. Cache Cleanup

# Clear application cache
rm -rf .cache/*

# Clear PM2 logs if too large
pm2 flush

Monthly Tasks

1. Security Review

  • Review admin access list
  • Check for unusual activity patterns
  • Update dependencies

2. Performance Analysis

# Generate monthly report
cat > monthly-report.sh << 'EOF'
#!/bin/bash
echo "=== Monthly Bot Report ==="
echo "Date: $(date)"
echo ""
echo "=== User Statistics ==="
sqlite3 somnia_validator_bot.db "SELECT COUNT(DISTINCT chatId) as 'Total Users' FROM subscriptions;"
echo ""
echo "=== Subscription Statistics ==="
sqlite3 somnia_validator_bot.db "SELECT COUNT(*) as 'Total Subscriptions' FROM subscriptions;"
echo ""
echo "=== Error Summary ==="
grep -c ERROR logs/bot-*.log | tail -30
EOF

chmod +x monthly-report.sh
./monthly-report.sh > reports/monthly-$(date +%Y%m).txt

3. Backup Verification

# Test backup restoration
cp backups/latest.db test-restore.db
sqlite3 test-restore.db "SELECT COUNT(*) FROM subscriptions;"
rm test-restore.db

Automated Monitoring

Health Check Script

Create health-check.sh:

#!/bin/bash

# Configuration
BOT_NAME="somnia-bot"
ALERT_EMAIL="admin@example.com"
DB_PATH="./somnia_validator_bot.db"

# Check if bot is running
if pm2 status | grep -q "$BOT_NAME.*online"; then
    echo "✓ Bot is running"
else
    echo "✗ Bot is not running!"
    # Send alert
    echo "Bot is down!" | mail -s "Somnia Bot Alert" $ALERT_EMAIL
    # Try to restart
    pm2 restart $BOT_NAME
fi

# Check database accessibility
if sqlite3 $DB_PATH "SELECT 1;" > /dev/null 2>&1; then
    echo "✓ Database is accessible"
else
    echo "✗ Database error!"
fi

# Check memory usage
MEMORY=$(pm2 describe $BOT_NAME | grep "memory" | awk '{print $4}')
if [ ${MEMORY%M} -gt 800 ]; then
    echo "⚠ High memory usage: $MEMORY"
fi

# Check recent errors
ERROR_COUNT=$(grep -c ERROR logs/bot-$(date +%Y-%m-%d).log)
if [ $ERROR_COUNT -gt 10 ]; then
    echo "⚠ High error count: $ERROR_COUNT errors today"
fi

Cron Jobs

Add to crontab:

# Health check every 5 minutes
*/5 * * * * /opt/somnia-bot/health-check.sh >> /opt/somnia-bot/logs/health.log 2>&1

# Daily database backup
0 2 * * * /opt/somnia-bot/backup-db.sh

# Weekly optimization
0 3 * * 0 sqlite3 /opt/somnia-bot/somnia_validator_bot.db "VACUUM;"

# Monthly report
0 0 1 * * /opt/somnia-bot/monthly-report.sh

Monitoring Best Practices

1. Set Up Alerts

  • High memory usage (>80%)
  • Bot offline/crashed
  • High error rate
  • Database growth

2. Regular Reviews

  • Daily: Check logs for errors
  • Weekly: Review performance metrics
  • Monthly: Analyze usage patterns

3. Proactive Maintenance

  • Keep logs rotated
  • Optimize database regularly
  • Monitor disk space
  • Update dependencies

4. Documentation

  • Keep incident log
  • Document configuration changes
  • Track performance baselines
  • Note recurring issues

Troubleshooting Performance Issues

High Memory Usage

  1. Check for memory leaks:

    # Monitor memory over time
pm2 describe somnia-bot | grep memory

  2. Reduce cache size:

    CACHE_TTL=60  # Reduce from default

  3. Restart periodically:

    # Add to cron for weekly restart
0 4 * * 0 pm2 restart somnia-bot

Slow Response Times

  1. Check RPC latency:

    time curl -X POST $SOMNIA_RPC_URL \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

  2. Analyze database queries:

    .timer on
SELECT * FROM validator_states WHERE nodeAddress = '0x...';

  3. Review monitoring intervals:

    • Increase intervals if too frequent
    • Use batch processing
Previous
Configuration
Next
Links & Tools