📊 System Overview
Platform: Multi-tenant SaaS platform for monitoring AI-generated answer visibility across major AI platforms (Google AI Overviews, Bing Copilot, Perplexity, Brave Summarizer, Arc Browse For Me)
Architecture: Hybrid serverless + containerized microservices on AWS, with frontend on Netlify and backend services on Render
Status: ✓ Production Ready
🌐 Frontend Infrastructure
📱 Static Site Hosting
Netlify
Service:
Static Site
Framework:
Next.js 14.2.33 (SSG)
Build Command:
npm run build
Publish Dir:
frontend/out
📝 Note: Configured in
netlify.toml. Deployment via GitLab integration or manual upload.
🔧 Configuration
Node Version:
18
Region:
Auto (Global CDN)
API Proxy:
→ Render Backend
🔑 Environment Variables:
- NEXT_PUBLIC_API_URL
- NEXT_PUBLIC_APP_ENV
⚙️ Backend Services (Render.com)
🚀 Analytics API
Render FastAPI
Service Name:
ai-visibility-analytics-api
Runtime:
Python 3.11
Build Command:
pip install -r requirements.txt
Start Command:
uvicorn main:app --host 0.0.0.0
Plan:
Starter
🔑 Environment Variables:
- ENVIRONMENT=production
- DATABASE_URL (from DB)
- JWT_SECRET (auto-generated)
- EVIDENCE_BUCKET_URL
- CORS_ORIGINS
🧠 Scoring Engine
Render Background Worker
Service Name:
ai-visibility-scoring-engine
Runtime:
Python 3.11
ML Models:
RoBERTa, SpaCy
Plan:
Starter
🔑 Environment Variables:
- DATABASE_URL
- SCORE_QUEUE_URL (AWS SQS)
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_DEFAULT_REGION=us-east-1
- LOG_LEVEL=INFO
🗄️ PostgreSQL Database
Render Managed Database
Database Name:
ai-visibility-db
Database:
ai_visibility_platform
User:
aivis_admin
Extensions:
TimescaleDB, uuid-ossp
Plan:
Starter
📝 Features:
- Multi-tenant schema isolation
- Time-series data with hypertables
- Automated backups
- Connection string auto-injected
☁️ AWS Infrastructure
Lambda Functions (Serverless)
| Function Name | Purpose | Memory | Timeout | Trigger |
|---|---|---|---|---|
| google-aio-worker | Fetch Google AI Overviews via SerpApi | 1024 MB | 900s | SQS (Fetch Queue) |
| perplexity-worker | Fetch Perplexity AI responses | 512 MB | 300s | SQS (Fetch Queue) |
| bing-copilot-worker | Fetch Bing Copilot answers via SerpApi | 1024 MB | 900s | SQS (Fetch Queue) |
| brave-worker | Fetch Brave Search summaries | 512 MB | 300s | SQS (Fetch Queue) |
| arc-worker | Browser automation for Arc Browse For Me | 2048 MB | 900s | SQS (Fetch Queue) |
| data-parser | Normalize and parse collected data | 1024 MB | 900s | SQS (Parse Queue) |
| gsc-connector | Sync Google Search Console data | 512 MB | 300s | Schedule (6 hours) |
| ga4-connector | Sync Google Analytics 4 data | 512 MB | 300s | Schedule (6 hours) |
SQS Queues (Message Broker)
📬 Fetch Queue
AWS SQS
Queue Name:
ai-visibility-fetch-{env}
Visibility Timeout:
15 minutes
Retention:
14 days
DLQ:
ai-visibility-fetch-dlq
📬 Parse Queue
AWS SQS
Queue Name:
ai-visibility-parse-{env}
Visibility Timeout:
10 minutes
DLQ:
ai-visibility-parse-dlq
📬 Score Queue
AWS SQS
Queue Name:
ai-visibility-score-{env}
Visibility Timeout:
5 minutes
Consumer:
Scoring Engine (Render)
Storage & CDN
🪣 Evidence Bucket
AWS S3
Bucket Name:
ai-visibility-evidence-{env}-{account}
Encryption:
AES-256 (Server-side)
Lifecycle:
Standard → IA (30d) → Glacier (90d)
Retention:
7 years
📝 Stored: Screenshots, HTML snapshots, JSON responses
🐳 Container Registry
AWS ECR
Repositories:
2
- ai-visibility-analytics-api
- ai-visibility-scoring-engine
📝 Note: Lifecycle policy deletes untagged images after 1 day
Orchestration & Networking
🔄 Step Functions
AWS Step Functions
State Machine:
ingestion-orchestrator
Schedule:
Every 4 hours
Purpose:
Coordinate data collection workflow
🌐 VPC & Networking
AWS VPC
VPC CIDR:
10.0.0.0/16
Subnets:
2 Public (Multi-AZ)
Load Balancer:
Application LB
📊 Monitoring
CloudWatch
Log Groups:
10+ (per service)
Retention:
30 days
Metrics:
Custom + AWS
🔌 External Services & APIs
🔍 SerpApi
External API
Purpose:
Google & Bing search data
Used By:
Google AIO, Bing Copilot workers
🔑 Required: SERP_API_KEY
🤖 Perplexity API
External API
Purpose:
AI answer generation
Model:
llama-3.1-sonar-large-128k-online
🔑 Required: PERPLEXITY_API_KEY
🦁 Brave Search API
External API
Purpose:
Search + AI summarization
Endpoints:
Search, Summarizer
🔑 Required: BRAVE_API_KEY
🎯 2Captcha
External Service
Purpose:
CAPTCHA solving for browser automation
Used By:
Arc Browse For Me worker
🔑 Optional: 2CAPTCHA_API_KEY
🌐 Proxy Services
External Service
Providers:
Bright Data, Oxylabs, Smartproxy
Type:
Residential proxies
📝 Purpose: Anti-detection for browser automation
📊 Google APIs
External API
Services:
Search Console, Analytics 4
Auth:
OAuth 2.0
🔑 Required: OAuth client credentials, refresh tokens
🔄 Data Flow Architecture
Step Functions Orchestrator (AWS - Every 4 hours)
↓
Reads tracked queries from PostgreSQL (Render)
↓
Dispatches jobs to SQS Fetch Queue (AWS)
↓
5 Lambda Workers process in parallel (AWS)
Google AIO | Perplexity | Bing Copilot | Brave | Arc
↓
Store raw data in S3 Evidence Bucket (AWS)
↓
Send to SQS Parse Queue (AWS)
↓
Data Parser Lambda normalizes data (AWS)
↓
Write to PostgreSQL (Render)
↓
Send to SQS Score Queue (AWS)
↓
Scoring Engine applies ML models (Render)
Sentiment Analysis (RoBERTa) | NER (SpaCy) | AQI Calculation
↓
Update scores in PostgreSQL (Render)
↓
Analytics API serves dashboard (Render)
↓
Frontend displays insights (Netlify)
🔐 Required Credentials & Configuration
AWS Credentials
- AWS_ACCESS_KEY_ID - IAM user with permissions for Lambda, SQS, S3, Step Functions
- AWS_SECRET_ACCESS_KEY - Corresponding secret key
- AWS_DEFAULT_REGION - us-east-1 (default)
- AWS_ACCOUNT_ID - Your AWS account ID
Database Credentials
- DATABASE_URL - PostgreSQL connection string (auto-provided by Render)
- DB_ENDPOINT - Database host
- DB_USERNAME - aivis_admin
- DB_PASSWORD - Secure password
- DB_NAME - ai_visibility_platform
API Keys (External Services)
- SERP_API_KEY - SerpApi for Google/Bing searches
- PERPLEXITY_API_KEY - Perplexity AI Search API
- BRAVE_API_KEY - Brave Search API
- 2CAPTCHA_API_KEY (Optional) - CAPTCHA solving
- PROXY_USERNAME (Optional) - Residential proxy credentials
- PROXY_PASSWORD (Optional) - Residential proxy password
Application Secrets
- JWT_SECRET - Secret for JWT token signing (auto-generated by Render or manual)
- FRONTEND_URL - Netlify frontend URL for CORS
OAuth Credentials (Google APIs)
- GOOGLE_CLIENT_ID - OAuth 2.0 client ID
- GOOGLE_CLIENT_SECRET - OAuth 2.0 client secret
- GOOGLE_REDIRECT_URI - OAuth callback URL
💰 Estimated Monthly Costs
Netlify (Frontend)
- Free Tier: $0/month (300 build minutes, 100GB bandwidth)
- Pro: $19/month (if needed for higher traffic)
Render.com (Backend)
- Analytics API (Web Service): $7/month (Starter)
- Scoring Engine (Background Worker): $7/month (Starter)
- PostgreSQL Database: $7/month (Starter, 1GB)
- Total Render: ~$21/month
AWS Services
- Lambda: $10-50/month (depends on execution count)
- SQS: $1-5/month (first 1M requests free)
- S3: $5-20/month (storage + requests)
- Step Functions: $5-15/month
- CloudWatch: $5-10/month (logs + metrics)
- Total AWS: ~$26-100/month (depending on scale)
External APIs
- SerpApi: $50-200/month (depends on query volume)
- Perplexity API: Variable (per query)
- Brave Search: $5-50/month
- 2Captcha: $3-10/month (if used)
- Proxy Services: $50-200/month (if needed)
- Total External: ~$108-460/month
💵 Grand Total: ~$155-581/month
Estimate varies based on query volume, data retention, and feature usage. Development/staging environments can use lower-cost tiers.
🚀 Deployment Status
| Component | Vendor | Status | Notes |
|---|---|---|---|
| Frontend (Next.js) | Netlify | ⏳ Ready to Deploy | Built, configured, awaiting upload |
| Analytics API | Render | ⏳ Ready to Deploy | Configuration in render.yaml |
| Scoring Engine | Render | ⏳ Ready to Deploy | Configuration in render.yaml |
| PostgreSQL Database | Render | ⏳ Ready to Deploy | Configuration in render.yaml |
| AWS Lambda Functions | AWS | ⏳ Ready to Deploy | SAM template ready |
| AWS Infrastructure | AWS | ⏳ Ready to Deploy | template.yaml configured |