{"id":62,"date":"2025-08-14T17:12:00","date_gmt":"2025-08-14T17:12:00","guid":{"rendered":"https:\/\/balamurali.in\/blog\/?p=62"},"modified":"2026-02-23T14:27:09","modified_gmt":"2026-02-23T14:27:09","slug":"building-an-ai-powered-competitive-intelligence-platform-a-deep-technical-dive","status":"publish","type":"post","link":"https:\/\/balamurali.in\/blog\/tech-posts\/building-an-ai-powered-competitive-intelligence-platform-a-deep-technical-dive\/","title":{"rendered":"Building an AI-Powered Competitive Intelligence Platform: A Deep Technical Dive"},"content":{"rendered":"\n

How I built a fully functional competitor monitoring system that automatically crawls websites, detects changes, and provides AI-powered business insights<\/em><\/p>\n\n\n\n

\n\n\n\n

Introduction<\/h2>\n\n\n\n

In today’s fast-moving business landscape, keeping track of what your competitors are doing is crucial. When they change their pricing, launch new features, or pivot their strategy, you need to know\u2014ideally before your customers do.<\/p>\n\n\n\n

I built Competition Monitoring System<\/strong>\u2014a comprehensive platform that automatically tracks competitor websites, detects meaningful changes, and uses AI to explain what those changes mean in a business context. While the market got saturated before I could launch it commercially, this project represents a fully-functional, production-ready system that demonstrates sophisticated engineering across multiple domains.<\/p>\n\n\n\n

In this deep-dive, I’ll walk you through the architecture, technical challenges, and solutions that make this system work.<\/p>\n\n\n\n

\n\n\n\n

The Problem: Why Manual Competitor Tracking Fails<\/h2>\n\n\n\n

Every product manager, growth marketer, and competitive analyst knows the pain:<\/p>\n\n\n\n

    \n
  1. Manual checking doesn’t scale<\/strong> – You can’t manually check 10+ competitor websites daily<\/li>\n\n\n\n
  2. Changes slip through<\/strong> – A competitor’s pricing page changes on Friday night, and you don’t notice until Monday’s customer call<\/li>\n\n\n\n
  3. No historical context<\/strong> – Even if you catch a change, you often don’t know what it was before<\/li>\n\n\n\n
  4. Signal vs. noise<\/strong> – Most website changes are irrelevant (footer updates, minor copy tweaks). The important ones get lost<\/li>\n<\/ol>\n\n\n\n

    The Solution: An Intelligent Monitoring Engine<\/h2>\n\n\n\n

    I built a system that:<\/p>\n\n\n\n

      \n
    • Automatically crawls competitor websites<\/strong> on a configurable schedule<\/li>\n\n\n\n
    • Detects ALL changes<\/strong> using dual-hashing (raw HTML + extracted text)<\/li>\n\n\n\n
    • Filters out noise<\/strong> through intelligent feed detection and change validation<\/li>\n\n\n\n
    • Provides AI-powered analysis<\/strong> explaining what each change means for your business<\/li>\n\n\n\n
    • Stores everything efficiently<\/strong> for historical comparison and dashboard visualization<\/li>\n<\/ul>\n\n\n\n
      \n\n\n\n

      System Architecture Overview<\/h2>\n\n\n\n

      The system is built with a clear separation of concerns:<\/p>\n\n\n\n

      \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502                          Competition Monitoring System                \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502                                                                       \u2502\n\u2502  \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510     \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510     \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510    \u2502\n\u2502  \u2502   Frontend  \u2502\u2500\u2500\u2500\u2500\u25b6\u2502   REST API  \u2502\u2500\u2500\u2500\u2500\u25b6\u2502  Backend Job Engine \u2502    \u2502\n\u2502  \u2502  (React.js) \u2502     \u2502  (FastAPI)  \u2502     \u2502     (Python)        \u2502    \u2502\n\u2502  \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518     \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518     \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518    \u2502\n\u2502                                                      \u2502               \u2502\n\u2502                                          \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510   \u2502\n\u2502                                          \u2502                       \u2502   \u2502\n\u2502                                    \u250c\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2510          \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2510\u2502\n\u2502                                    \u2502  Supabase \u2502          \u2502   S3    \u2502\u2502\n\u2502                                    \u2502 (Metadata)\u2502          \u2502(Content)\u2502\u2502\n\u2502                                    \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518          \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\u2502\n\u2502                                                                       \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/code><\/pre>\n\n\n\n
      Technology Stack<\/h3>\n\n\n\n
      Layer<\/th>Technology<\/th>Purpose<\/th><\/tr><\/thead>
      Web Crawling<\/strong><\/td>Playwright + BeautifulSoup<\/td>JavaScript-enabled headless browsing<\/td><\/tr>
      AI Analysis<\/strong><\/td>Google Gemini API<\/td>Page summarization, change analysis<\/td><\/tr>
      URL Discovery<\/strong><\/td>Firecrawl API<\/td>Comprehensive site mapping<\/td><\/tr>
      Database<\/strong><\/td>Supabase (PostgreSQL)<\/td>Metadata, sessions, change records<\/td><\/tr>
      Object Storage<\/strong><\/td>S3-compatible (iDrive e2)<\/td>HTML snapshots, screenshots<\/td><\/tr>
      API<\/strong><\/td>FastAPI<\/td>RESTful endpoints<\/td><\/tr>
      Frontend<\/strong><\/td>React\/Next.js<\/td>Dashboard visualization<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
      \n\n\n\n
      Deep Dive: The Crawling Engine<\/h2>\n\n\n\n
      The heart of the system is the OptimizedWebCrawler<\/code>\u2014a sophisticated Python class that handles everything from URL normalization to JavaScript rendering.<\/p>\n\n\n\n
      Concurrent Architecture<\/h3>\n\n\n\n
      Traditional web crawlers are slow because they process one URL at a time. My crawler uses an isolated worker architecture<\/strong> where multiple browser instances work in parallel:<\/p>\n\n\n\n
      class OptimizedWebCrawler:\n    def __init__(self, base_url: str, max_pages: int = 100, \n                 concurrent_limit: int = 5):\n        self.concurrent_limit = concurrent_limit\n        self.url_queue = RobustURLQueue()\n        # Each worker gets its own browser instance\n        # Complete isolation prevents context conflicts<\/code><\/pre>\n\n\n\n
      Key insight<\/strong>: Sharing browser contexts between workers causes “Target page, context or browser has been closed” errors. By giving each worker its own browser instance, I achieved a 96.7% success rate<\/strong> (up from 45% with shared contexts).<\/p>\n\n\n\n
      Robust URL Queue with Retry Logic<\/h3>\n\n\n\n
      URLs fail for many reasons\u2014network timeouts, rate limiting, temporary server errors. The RobustURLQueue<\/code> class handles this gracefully:<\/p>\n\n\n\n
      @dataclass\nclass URLTask:\n    url: str\n    retry_count: int = 0\n    max_retries: int = 3\n    last_error: Optional[str] = None\n\nclass RobustURLQueue:\n    def __init__(self):\n        self.main_queue = asyncio.Queue()\n        self.retry_queue = asyncio.Queue()  # Priority for retries\n        self.failed_urls = []\n\n    async def get(self, timeout: float = 2.0) -> Optional[URLTask]:\n        # Prioritize retry queue over main queue\n        if not self.retry_queue.empty():\n            return await self.retry_queue.get()\n        return await self.main_queue.get()<\/code><\/pre>\n\n\n\n
      Enhanced JavaScript Rendering<\/h3>\n\n\n\n
      Modern websites are JavaScript-heavy. A naive crawler that just fetches HTML will miss 80% of the content. My solution implements multi-stage loading detection<\/strong>:<\/p>\n\n\n\n
      async def _wait_for_javascript_content(self, page, url: str):\n    # Step 1: Quick check if already loaded\n    ready_state = await page.evaluate('() => document.readyState')\n    if ready_state == 'complete':\n        await page.wait_for_timeout(500)\n        return\n\n    # Step 2: Wait for meaningful content\n    await page.wait_for_function('''() => {\n        return document.body &&\n               document.body.innerText &&\n               document.body.innerText.length > 100;\n    }''', timeout=5000)\n\n    # Step 3: Special SPA handling (React\/Vue\/Angular)\n    if any(fw in url.lower() for fw in ['app.', 'console.', 'dashboard.']):\n        await page.wait_for_function('''() => {\n            if (window.React || window.Vue || window.ng) {\n                return document.readyState === 'complete';\n            }\n            return true;\n        }''', timeout=3000)<\/code><\/pre>\n\n\n\n
      Result<\/strong>: 80-90% faster JavaScript content waiting while maintaining reliability.<\/p>\n\n\n\n
      Smart Navigation Strategy<\/h3>\n\n\n\n
      A critical discovery: using wait_until=\"networkidle\"<\/code> causes hangs on sites with continuous background requests (analytics, tracking pixels). The fix was simple but crucial:<\/p>\n\n\n\n
      # Before (problematic - would timeout on modern sites)\nresponse = await page.goto(url, wait_until=\"networkidle\", timeout=180000)\n\n# After (reliable - works with background activity)\nresponse = await page.goto(url, wait_until=\"domcontentloaded\", timeout=60000)<\/code><\/pre>\n\n\n\n
      This single change reduced crawl times from 180+ seconds (timeout) to 8-10 seconds for sites like factors.ai.<\/p>\n\n\n\n
      \n\n\n\n
      Intelligent Feed Detection<\/h2>\n\n\n\n
      One of the most sophisticated features is the Feed Detection & Optimization System<\/strong>. Feed pages (blogs, news, resources) contain hundreds of child URLs that follow similar patterns. Crawling them all is wasteful.<\/p>\n\n\n\n
      The Problem<\/h3>\n\n\n\n
      Without feed detection:<\/p>\n\n\n\n
        \n
      1. Crawl \/blog<\/code> page<\/li>\n\n\n\n
      2. Discover 200 blog post URLs<\/li>\n\n\n\n
      3. Crawl all 200 posts (expensive!)<\/li>\n\n\n\n
      4. Next crawl: same 200 posts + 1 new one<\/li>\n\n\n\n
      5. Re-crawl everything again<\/li>\n<\/ol>\n\n\n\n
        The Solution: AI-Powered Feed Discovery<\/h3>\n\n\n\n
        On first crawl, the system uses Firecrawl API<\/strong> to discover ALL URLs on a domain, then Gemini AI<\/strong> to identify feed patterns:<\/p>\n\n\n\n
        # Firecrawl discovers comprehensive URL structure\nfirecrawl_result = await firecrawl_service.discover_urls(\n    domain='competitor.com',\n    include_subdomains=True\n)\n\n# AI analyzes URLs for feed patterns\nai_analysis = await llm_service.analyze_feed_patterns(\n    urls=firecrawl_result['urls'],\n    paths=firecrawl_result['paths'],\n    domain='competitor.com'\n)\n\n# Save domain-specific patterns to Supabase\ndomain_config_service.save_domain_config(\n    domain='competitor.com',\n    feed_paths=ai_analysis['feed_paths'],  # e.g., ['\/blog', '\/insights', '\/news']\n    ai_analysis=ai_analysis\n)<\/code><\/pre>\n\n\n\n
        Multi-Session Feed Processing<\/h3>\n\n\n\n
        The magic happens across crawl sessions:<\/p>\n\n\n\n
        First Crawl (Discovery)<\/strong>:<\/p>\n\n\n\n
          \n
        • Detect \/blog<\/code> as a feed page<\/li>\n\n\n\n
        • Discover 50 child URLs<\/li>\n\n\n\n
        • Store URLs WITHOUT processing<\/strong> (no AI, no screenshots)<\/li>\n\n\n\n
        • Establish baseline for comparison<\/li>\n<\/ul>\n\n\n\n
          Second Crawl (Detection)<\/strong>:<\/p>\n\n\n\n
            \n
          • Re-crawl \/blog<\/code><\/li>\n\n\n\n
          • Find 52 URLs (2 new posts)<\/li>\n\n\n\n
          • Only process the 2 new URLs<\/strong> with AI<\/li>\n\n\n\n
          • Include existing URLs in session data (prevents false “removed” alerts)<\/li>\n<\/ul>\n\n\n\n
            Result<\/strong>: 60-80% reduction in redundant processing.<\/p>\n\n\n\n
            Feed Pattern Detection<\/h3>\n\n\n\n
            The system recognizes these URL patterns as feeds:<\/p>\n\n\n\n
            FEED_PATTERNS = [\n    r'\/news\/?$', r'\/blog\/?$', r'\/articles\/?$',\n    r'\/insights\/?$', r'\/resources\/?$', r'\/whitepapers\/?$',\n    r'\/awards\/?$', r'\/events\/?$', r'\/webinars\/?$',\n    r'\/docs\/?$', r'\/help\/?$', r'\/careers\/?$'\n]<\/code><\/pre>\n\n\n\n
            Combined with AI-discovered domain-specific patterns stored in Supabase, this catches even unusual feed structures.<\/p>\n\n\n\n
            \n\n\n\n
            Dual Hashing: Catching Every Change<\/h2>\n\n\n\n
            Change detection uses a two-tier hashing approach<\/strong>:<\/p>\n\n\n\n
            Tier 1: Raw HTML Hash<\/h3>\n\n\n\n
            def calculate_html_hash(self, html: str) -> str:\n    return hashlib.md5(html.encode('utf-8')).hexdigest()<\/code><\/pre>\n\n\n\n
            Captures ALL changes<\/strong>: script updates, CSS modifications, A\/B tests, tracking pixels.<\/p>\n\n\n\n
            Tier 2: Text Content Hash<\/h3>\n\n\n\n
            def calculate_content_hash(self, text: str) -> str:\n    # Extract text, remove scripts\/styles\/metadata\n    soup = BeautifulSoup(html, 'lxml')\n    for script in soup([\"script\", \"style\", \"meta\", \"link\"]):\n        script.decompose()\n    text = soup.get_text()\n    return hashlib.md5(text.encode('utf-8')).hexdigest()<\/code><\/pre>\n\n\n\n
            Focuses on meaningful content changes<\/strong> for AI analysis.<\/p>\n\n\n\n
            Why Both?<\/h3>\n\n\n\n
              \n
            • HTML hash changes but text hash doesn’t<\/strong> \u2192 Technical change (A\/B test, script update)<\/li>\n\n\n\n
            • Both hashes change<\/strong> \u2192 Content change worth analyzing with AI<\/li>\n\n\n\n
            • Backward compatibility<\/strong> \u2192 Old data without HTML hash still works via text hash fallback<\/li>\n<\/ul>\n\n\n\n
              \n\n\n\n
              Precise Word-Level Change Analysis<\/h2>\n\n\n\n
              When content changes, we need to know exactly what<\/em> changed. The naive approach (line-by-line diff) produces terrible results:<\/p>\n\n\n\n
              Bad Output<\/strong> (line-based):<\/p>\n\n\n\n
              REMOVED: \"Home Products Pricing About Blog Contact\"\nADDED: \"Home Products Pricing About Blog Contact Login\"<\/code><\/pre>\n\n\n\n
              Good Output<\/strong> (word-level):<\/p>\n\n\n\n
              ADDED: \"Login\"<\/code><\/pre>\n\n\n\n
              My implementation uses word tokenization with sequence matching:<\/p>\n\n\n\n
              def _analyze_text_changes(self, old_text: str, new_text: str) -> Dict:\n    # Tokenize to words\n    old_words = self._tokenize_text(old_text)\n    new_words = self._tokenize_text(new_text)\n\n    # Find precise word-level differences\n    matcher = SequenceMatcher(None, old_words, new_words)\n    opcodes = list(matcher.get_opcodes())\n\n    added_segments = []\n    removed_segments = []\n\n    for tag, i1, i2, j1, j2 in opcodes:\n        if tag == 'insert':\n            added_segments.append(' '.join(new_words[j1:j2]))\n        elif tag == 'delete':\n            removed_segments.append(' '.join(old_words[i1:i2]))\n\n    return {\n        'added': added_segments,\n        'removed': removed_segments,\n        'change_ratio': 1 - matcher.ratio()\n    }<\/code><\/pre>\n\n\n\n
              \n\n\n\n
              Change Validation: Eliminating False Positives<\/h2>\n\n\n\n
              Not all detected changes are real. Dynamic content, A\/B tests, and JavaScript timing issues create false positives<\/strong>. The ChangeValidator<\/code> service handles this:<\/p>\n\n\n\n
              class ChangeValidator:\n    async def validate_changes(self, changes: List[Change]) -> List[Change]:\n        validated = []\n\n        for change in changes:\n            # Re-scrape twice with delay\n            scrape1 = await self._scrape_url_once(change.url, 1)\n            await asyncio.sleep(2)\n            scrape2 = await self._scrape_url_once(change.url, 2)\n\n            # Compare consistency\n            if scrape1['content_hash'] == scrape2['content_hash']:\n                validated.append(change)  # Consistent = real change\n            else:\n                logger.info(f\"Invalidated {change.url}: inconsistent content\")\n\n        return validated<\/code><\/pre>\n\n\n\n
              Only changes that consistently reproduce<\/strong> proceed to AI analysis.<\/p>\n\n\n\n
              \n\n\n\n
              AI-Powered Business Context<\/h2>\n\n\n\n
              Detecting changes is only half the battle. The real value is understanding what they mean<\/strong>. The ChangeAnalyzer<\/code> service uses Google’s Gemini API to provide business context.<\/p>\n\n\n\n
              Page Summarization During Crawling<\/h3>\n\n\n\n
              Every page is summarized in real-time:<\/p>\n\n\n\n
              async def generate_page_summary(self, text_content: str, url: str) -> PageSummary:\n    prompt = f\"\"\"Analyze the following webpage content from URL: {url}\n\n    Provide:\n    1. A concise 2-3 sentence summary\n    2. The page type (pricing, product, blog, etc.)\n    3. 5-10 relevant keywords\n    4. Key entities mentioned (products, people, companies)\n    \"\"\"\n\n    response = await self._call_with_backoff(prompt)\n    return self._parse_summary_response(response)<\/code><\/pre>\n\n\n\n
              Change Analysis with Severity Scoring<\/h3>\n\n\n\n
              When changes are detected, AI evaluates their business significance:<\/p>\n\n\n\n
              @dataclass\nclass ChangeAnalysis:\n    url: str\n    change_type: str        # pricing_update, feature_addition, etc.\n    severity: int           # 1-10 scale\n    change_definition: str  # 3-line business analysis\n    recommended_pages: List[str]  # Related pages for context<\/code><\/pre>\n\n\n\n
              Severity Guidelines<\/strong>:<\/p>\n\n\n\n
                \n
              • 1-3<\/strong>: Minor (typo fixes, date updates)<\/li>\n\n\n\n
              • 4-6<\/strong>: Moderate (new blog post, team change)<\/li>\n\n\n\n
              • 7-9<\/strong>: Significant (pricing change, new feature)<\/li>\n\n\n\n
              • 10<\/strong>: Critical (acquisition, major pivot)<\/li>\n<\/ul>\n\n\n\n
                Exponential Backoff for API Reliability<\/h3>\n\n\n\n
                Gemini API has rate limits. The service implements robust retry logic:<\/p>\n\n\n\n
                async def _call_with_backoff(self, prompt: str) -> str:\n    for attempt in range(self.max_retries):  # 10 retries\n        try:\n            response = await asyncio.to_thread(\n                self.model.generate_content, prompt\n            )\n            return response.text\n        except Exception as e:\n            if 'rate' in str(e).lower() or '429' in str(e):\n                delay = min(self.base_delay * (2 ** attempt), 60)\n                jitter = random.uniform(0, delay * 0.1)\n                await asyncio.sleep(delay + jitter)\n            else:\n                raise<\/code><\/pre>\n\n\n\n
                \n\n\n\n
                Data Architecture: Hybrid Storage Strategy<\/h2>\n\n\n\n
                The system uses a hybrid storage approach<\/strong> optimized for different access patterns:<\/p>\n\n\n\n
                Supabase (PostgreSQL) – Fast Queries<\/h3>\n\n\n\n
                Normalized tables for dashboard queries:<\/p>\n\n\n\n
                -- Sessions table\nCREATE TABLE competition_analysis_sessions (\n    id UUID PRIMARY KEY,\n    domain TEXT NOT NULL,\n    session_id TEXT NOT NULL,\n    comparison_type TEXT,\n    total_changes INTEGER,\n    created_at TIMESTAMPTZ\n);\n\n-- Changes table with indexes\nCREATE TABLE competition_analysis_changes (\n    id UUID PRIMARY KEY,\n    session_analysis_id UUID REFERENCES competition_analysis_sessions(id),\n    url TEXT NOT NULL,\n    change_type TEXT NOT NULL,\n    severity INTEGER CHECK (severity >= 1 AND severity <= 10),\n    page_type TEXT,\n    change_definition TEXT,\n    recommended_pages JSONB,\n    created_at TIMESTAMPTZ\n);\n\n-- Indexes for fast filtering\nCREATE INDEX idx_severity ON competition_analysis_changes(severity);\nCREATE INDEX idx_change_type ON competition_analysis_changes(change_type);<\/code><\/pre>\n\n\n\n
                S3 (iDrive e2) – Complete Backups<\/h3>\n\n\n\n
                Object storage for full content:<\/p>\n\n\n\n
                competitor.com\/\n\u251c\u2500\u2500 sessions\/\n\u2502   \u251c\u2500\u2500 20250115_103000.json.gz   # Complete session data\n\u2502   \u2514\u2500\u2500 20250114_103000.json.gz\n\u251c\u2500\u2500 master\/\n\u2502   \u2514\u2500\u2500 state.json.gz             # Master state tracking\n\u2514\u2500\u2500 screenshots\/\n    \u2514\u2500\u2500 20250115_103000\/\n        \u251c\u2500\u2500 homepage.png\n        \u2514\u2500\u2500 pricing.png<\/code><\/pre>\n\n\n\n
                Master State: The Source of Truth<\/h3>\n\n\n\n
                The master state tracks all pages across sessions:<\/p>\n\n\n\n
                {\n  \"domain\": \"competitor.com\",\n  \"pages\": {\n    \"https:\/\/competitor.com\/pricing\": {\n      \"html_hash\": \"x9y8z7...\",\n      \"content_hash\": \"a1b2c3...\",\n      \"title\": \"Pricing Plans\",\n      \"ai_summary\": \"...\",\n      \"last_session\": \"20250115_103000\"\n    }\n  },\n  \"feed_state\": {\n    \"feeds\": {\n      \"https:\/\/competitor.com\/blog\": {\n        \"feed_type\": \"blog\",\n        \"discovered_urls\": [\"...\", \"...\"],\n        \"last_url_count\": 50\n      }\n    }\n  },\n  \"sessions\": [\n    {\"id\": \"20250115_103000\", \"pages_count\": 45},\n    {\"id\": \"20250114_103000\", \"pages_count\": 44}\n  ]\n}<\/code><\/pre>\n\n\n\n
                \n\n\n\n
                URL Normalization: Preventing Duplicate Crawls<\/h2>\n\n\n\n
                A subtle but critical feature: URLs with minor variations must be treated as identical:<\/p>\n\n\n\n
                https://factors.ai\/pricing\nhttps:\/\/www.factors.ai\/pricing      # www subdomain\nhttps:\/\/factors.ai\/pricing\/         # trailing slash\nhttps:\/\/FACTORS.AI\/pricing          # case difference\nhttps:\/\/factors.ai\/pricing?ref=nav  # query params<\/code><\/pre>\n\n\n\n
                The normalization function handles all these cases:<\/p>\n\n\n\n
                def normalize_url(self, url: str) -> str:\n    parsed = urlparse(url)\n\n    # 1. Remove www subdomain\n    netloc = parsed.netloc\n    if netloc.startswith('www.'):\n        netloc = netloc[4:]\n\n    # 2. Lowercase hostname\n    netloc = netloc.lower()\n\n    # 3. Remove trailing slash\n    path = parsed.path.rstrip('\/') or '\/'\n\n    # 4. Sort query parameters\n    if parsed.query:\n        params = sorted(parse_qsl(parsed.query))\n        query = urlencode(params)\n\n    # 5. Remove fragments\n    # 6. Normalize index files\n\n    return urlunparse((parsed.scheme, netloc, path, '', query, ''))<\/code><\/pre>\n\n\n\n
                \n\n\n\n
                Pagination URL Filtering<\/h2>\n\n\n\n
                An interesting edge case: pagination URLs were causing exponential crawling:<\/p>\n\n\n\n
                \/blog \u2192 discovers \/blog?page=1, \/blog?page=2, ...\n\/blog?page=1 \u2192 treated as new feed \u2192 discovers more pagination URLs\n\/blog?page=2 \u2192 same problem\n... (exponential explosion)<\/code><\/pre>\n\n\n\n
                Solution<\/strong>: Filter pagination URLs before they enter the queue:<\/p>\n\n\n\n
                pagination_patterns = [\n    'page=', '_page=', 'p=', 'offset=', 'start=',\n    'pagenum=', 'pagenumber=', 'pageindex=', 'paged='\n]\n\ndef should_crawl(self, url: str) -> bool:\n    if parsed_url.query:\n        query_lower = parsed_url.query.lower()\n        if any(pattern in query_lower for pattern in pagination_patterns):\n            return False  # Skip pagination URLs\n    return True<\/code><\/pre>\n\n\n\n
                \n\n\n\n
                Key Technical Decisions & Trade-offs<\/h2>\n\n\n\n
                1. Playwright vs. Puppeteer vs. Selenium<\/h3>\n\n\n\n
                Choice<\/strong>: Playwright<\/p>\n\n\n\n
                Why<\/strong>:<\/p>\n\n\n\n
                  \n
                • Best JavaScript rendering support<\/li>\n\n\n\n
                • Async API for concurrent crawling<\/li>\n\n\n\n
                • Cross-browser testing if needed<\/li>\n\n\n\n
                • Active development and great docs<\/li>\n<\/ul>\n\n\n\n
                  2. Gemini vs. GPT-4 vs. Claude<\/h3>\n\n\n\n
                  Choice<\/strong>: Google Gemini<\/p>\n\n\n\n
                  Why<\/strong>:<\/p>\n\n\n\n
                    \n
                  • Cost-effective for high-volume summarization<\/li>\n\n\n\n
                  • Fast response times<\/li>\n\n\n\n
                  • Sufficient quality for page analysis<\/li>\n\n\n\n
                  • Easy to switch via LLMFactory<\/code> abstraction<\/li>\n<\/ul>\n\n\n\n
                    3. Supabase vs. Self-hosted PostgreSQL<\/h3>\n\n\n\n
                    Choice<\/strong>: Supabase<\/p>\n\n\n\n
                    Why<\/strong>:<\/p>\n\n\n\n
                      \n
                    • Managed PostgreSQL with row-level security<\/li>\n\n\n\n
                    • Real-time subscriptions for dashboard<\/li>\n\n\n\n
                    • Built-in auth for multi-tenant future<\/li>\n\n\n\n
                    • Edge functions for serverless compute<\/li>\n<\/ul>\n\n\n\n
                      4. S3 vs. Database for HTML Storage<\/h3>\n\n\n\n
                      Choice<\/strong>: Hybrid (both)<\/p>\n\n\n\n
                      Why<\/strong>:<\/p>\n\n\n\n
                        \n
                      • S3: Cheap storage for large HTML\/screenshots<\/li>\n\n\n\n
                      • Supabase: Fast queries for dashboard data<\/li>\n\n\n\n
                      • Master state in S3: Single source of truth for crawl history<\/li>\n<\/ul>\n\n\n\n
                        \n\n\n\n
                        Performance Characteristics<\/h2>\n\n\n\n
                        Metric<\/th>Value<\/th><\/tr><\/thead>
                        Concurrent Workers<\/strong><\/td>5 (configurable to 10)<\/td><\/tr>
                        URLs per Minute<\/strong><\/td>~50-100 (depending on site speed)<\/td><\/tr>
                        JavaScript Wait Time<\/strong><\/td>0.5s (fast) to 5s (SPAs)<\/td><\/tr>
                        Success Rate<\/strong><\/td>96.7%<\/td><\/tr>
                        AI Summaries per Minute<\/strong><\/td>~20 (with rate limiting)<\/td><\/tr>
                        Storage per Crawl<\/strong><\/td>~2-10 MB compressed<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
                        \n\n\n\n
                        Lessons Learned<\/h2>\n\n\n\n
                        1. Browser Context Isolation is Critical<\/h3>\n\n\n\n
                        Sharing browser contexts between concurrent workers causes race conditions and cryptic errors. Each worker needs its own isolated browser instance.<\/p>\n\n\n\n
                        2. NetworkIdle is a Lie<\/h3>\n\n\n\n
                        Modern websites never reach “network idle” due to analytics, tracking, and WebSocket connections. Use domcontentloaded<\/code> instead.<\/p>\n\n\n\n
                        3. Change Detection Needs Two Hashes<\/h3>\n\n\n\n
                        Raw HTML hash catches technical changes. Text content hash catches meaningful changes. You need both for complete coverage.<\/p>\n\n\n\n
                        4. Feed Detection is Essential for Scale<\/h3>\n\n\n\n
                        Without intelligent feed handling, you’ll waste 80% of your crawl budget on blog posts and news articles that haven’t changed.<\/p>\n\n\n\n
                        5. Validation Eliminates False Positives<\/h3>\n\n\n\n
                        A\/B tests, personalization, and dynamic content create fake “changes”. Re-scraping twice with a delay catches these.<\/p>\n\n\n\n
                        \n\n\n\n
                        What I Would Do Differently<\/h2>\n\n\n\n
                          \n
                        1. Start with headless Chrome service<\/strong>: Instead of managing Playwright browsers in-process, use a dedicated browser pool service (Browserless, etc.)<\/li>\n\n\n\n
                        2. Event-driven architecture from day one<\/strong>: Use message queues (Redis, SQS) between crawl and analysis phases for better scaling<\/li>\n\n\n\n
                        3. More aggressive caching<\/strong>: Cache AI summaries longer\u2014they rarely need regeneration if content hasn’t changed<\/li>\n\n\n\n
                        4. Visual diffing earlier<\/strong>: Screenshot comparison could catch changes that text analysis misses<\/li>\n<\/ol>\n\n\n\n
                          \n\n\n\n
                          The Market Reality<\/h2>\n\n\n\n
                          I built this system over several months, iterating through dozens of technical challenges. By the time it was production-ready, several well-funded competitors had entered the space:<\/p>\n\n\n\n
                            \n
                          • Klue<\/li>\n\n\n\n
                          • Crayon<\/li>\n\n\n\n
                          • Kompyte<\/li>\n\n\n\n
                          • Similarweb<\/li>\n<\/ul>\n\n\n\n
                            The market went from “interesting opportunity” to “saturated” faster than expected. But the technical work remains valuable\u2014both as a portfolio piece and as a foundation for future projects.<\/p>\n\n\n\n
                            \n\n\n\n
                            Conclusion<\/h2>\n\n\n\n
                            Building a competitive intelligence platform touches nearly every aspect of modern software engineering:<\/p>\n\n\n\n
                              \n
                            • Distributed systems<\/strong>: Concurrent crawling, queue management<\/li>\n\n\n\n
                            • Web scraping<\/strong>: JavaScript rendering, anti-bot evasion<\/li>\n\n\n\n
                            • AI\/ML<\/strong>: LLM integration, prompt engineering<\/li>\n\n\n\n
                            • Data engineering<\/strong>: Hybrid storage, change detection<\/li>\n\n\n\n
                            • Database design<\/strong>: Normalized schemas, efficient indexing<\/li>\n\n\n\n
                            • API design<\/strong>: RESTful endpoints, real-time updates<\/li>\n<\/ul>\n\n\n\n
                              While the market timing didn’t work out for commercial launch, this project demonstrates that complex, production-grade systems can be built by a small team (or solo developer) using modern tools and cloud services.<\/p>\n\n\n\n
                              The code is functional, the architecture is sound, and the technical challenges were genuinely interesting to solve. Sometimes that’s the real value of a project\u2014not the business outcome, but what you learn along the way.<\/p>\n\n\n\n
                              \n\n\n\n
                              If you’re interested in the technical details or want to discuss competitive intelligence systems, feel free to reach out. The lessons learned here apply to many other domains: price monitoring, content aggregation, research automation, and more.<\/em><\/p>\n\n\n\n
                              \n\n\n\n
                              Technical Stack Summary<\/h2>\n\n\n\n
                              Component<\/th>Technology<\/th><\/tr><\/thead>
                              Language<\/td>Python 3.12<\/td><\/tr>
                              Web Crawling<\/td>Playwright, BeautifulSoup, lxml<\/td><\/tr>
                              AI<\/td>Google Gemini API, LangChain<\/td><\/tr>
                              URL Discovery<\/td>Firecrawl API<\/td><\/tr>
                              Database<\/td>Supabase (PostgreSQL)<\/td><\/tr>
                              Object Storage<\/td>S3-compatible (iDrive e2)<\/td><\/tr>
                              API Framework<\/td>FastAPI, Uvicorn<\/td><\/tr>
                              Job Scheduling<\/td>Celery, Redis, APScheduler<\/td><\/tr>
                              Testing<\/td>pytest, pytest-asyncio<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
                              \n\n\n\n
                              This project represents approximately 15,000+ lines of production-quality Python code, comprehensive documentation, and battle-tested solutions to real-world web scraping challenges.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"
                              How I built a fully functional competitor monitoring system that automatically crawls websites, detects changes, and provides AI-powered business insights Introduction In today’s fast-moving business landscape, keeping track of what…<\/p>\n","protected":false},"author":1,"featured_media":154,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[8,4],"tags":[],"class_list":["post-62","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-learn-with-me","category-tech-posts"],"jetpack_featured_media_url":"https:\/\/balamurali.in\/blog\/wp-content\/uploads\/2026\/02\/competitive_intelligence.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/posts\/62","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/comments?post=62"}],"version-history":[{"count":1,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/posts\/62\/revisions"}],"predecessor-version":[{"id":64,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/posts\/62\/revisions\/64"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/media\/154"}],"wp:attachment":[{"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/media?parent=62"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/categories?post=62"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/tags?post=62"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}