{"id":65,"date":"2025-11-14T17:21:00","date_gmt":"2025-11-14T17:21:00","guid":{"rendered":"https:\/\/balamurali.in\/blog\/?p=65"},"modified":"2026-02-23T14:26:50","modified_gmt":"2026-02-23T14:26:50","slug":"building-an-internal-analytics-dashboard-why-we-needed-it-and-how-it-came-together","status":"publish","type":"post","link":"https:\/\/balamurali.in\/blog\/tech-posts\/building-an-internal-analytics-dashboard-why-we-needed-it-and-how-it-came-together\/","title":{"rendered":"Building an Internal Analytics Dashboard: Why We Needed It and How It Came Together"},"content":{"rendered":"\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p>At Factors.ai, we manage over 4,000+ customer projects. Each project generates a wealth of data \u2014 events, user sessions, integrations, CRM syncs, background jobs, and more. While our platform provides customers with great insights, I realized we lacked a unified internal view to monitor the health of all these projects at scale.<\/p>\n\n\n\n<p>Our Customer Success and Engineering teams needed answers to questions like:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Which projects are experiencing data collection failures?<\/li>\n\n\n\n<li>How many accounts are being identified daily across our customer base?<\/li>\n\n\n\n<li>Are CRM integrations syncing properly?<\/li>\n\n\n\n<li>Which background jobs are failing and why?<\/li>\n<\/ul>\n\n\n\n<p>Previously, this meant jumping between multiple tools, manually querying databases, and piecing together information. I decided to build a single source of truth \u2014 and that&#8217;s how the <strong>Factors.ai Project Health Dashboard<\/strong> was born.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">The Solution: A Modern Analytics Dashboard<\/h2>\n\n\n\n<p>I built a comprehensive internal dashboard that consolidates project health metrics, user activity, system status, and integration health into one place.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The Architecture<\/h3>\n\n\n\n<p>The system is composed of three main components:<\/p>\n\n\n\n<p><strong>1. React Frontend (TypeScript)<\/strong><br>A modern, responsive web app built with React 19, TypeScript, and styled-components. It features:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Google Authentication restricted to <code>@factors.ai<\/code> emails<\/li>\n\n\n\n<li>Interactive charts powered by Chart.js<\/li>\n\n\n\n<li>Real-time KPI cards with trend indicators<\/li>\n\n\n\n<li>Dark-themed UI for comfortable all-day use<\/li>\n<\/ul>\n\n\n\n<p><strong>2. Node.js\/Express Backend (API Server)<\/strong><br>A backend API deployed on Google Cloud Run that:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Aggregates data from BigQuery and the live Factors.ai Admin API<\/li>\n\n\n\n<li>Handles authentication and session management<\/li>\n\n\n\n<li>Provides RESTful endpoints with Swagger documentation<\/li>\n\n\n\n<li>Implements proper rate limiting, logging, and error handling<\/li>\n<\/ul>\n\n\n\n<p><strong>3. Python Analytics Collector (Cloud Run Job)<\/strong><br>An automated data pipeline that:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Runs on a schedule via Cloud Scheduler<\/li>\n\n\n\n<li>Fetches analytics data from the Factors.ai API for all tracked projects<\/li>\n\n\n\n<li>Stores processed data in BigQuery for historical analysis<\/li>\n\n\n\n<li>Handles rate limiting, retries, and error recovery gracefully<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Cloud Run Job   \u2502    \u2502   Factors.ai     \u2502    \u2502    BigQuery     \u2502\n\u2502   (Scheduler)   \u2502\u2500\u2500\u2500\u25b6\u2502      API         \u2502\u2500\u2500\u2500\u25b6\u2502   (Analytics)   \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518    \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518    \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Key Technical Decisions<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Live Data + Historical Data<\/h3>\n\n\n\n<p>I adopted a hybrid approach: the dashboard prioritizes <strong>live data from the Factors.ai Admin API<\/strong> for current metrics while using <strong>BigQuery for historical trends<\/strong>. This ensures users always see up-to-date information while still being able to analyze 30-day trends.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">BigQuery for Analytics Storage<\/h3>\n\n\n\n<p>BigQuery was the natural choice for storing analytics data \u2014 it scales effortlessly with 4,000+ projects and allows complex aggregations on millions of rows without infrastructure headaches.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Cloud Run for Everything<\/h3>\n\n\n\n<p>Both the frontend and backend are containerized and deployed on Cloud Run. This gave me:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Zero infrastructure management<\/li>\n\n\n\n<li>Automatic scaling based on traffic<\/li>\n\n\n\n<li>Pay-per-use pricing<\/li>\n\n\n\n<li>Easy CI\/CD via GitHub Actions<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Domain-Restricted Authentication<\/h3>\n\n\n\n<p>Security was a priority. I integrated Google OAuth but restricted it to <code>@factors.ai<\/code> email addresses only. New team members are automatically provisioned when they first sign in.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Why This Matters<\/h2>\n\n\n\n<p>This dashboard has transformed how our teams operate:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Faster Incident Response<\/strong> \u2014 When a project has data collection issues, the team knows immediately rather than waiting for a customer complaint.<\/li>\n\n\n\n<li><strong>Proactive Customer Success<\/strong> \u2014 CS teams can spot projects with declining engagement before it becomes churn.<\/li>\n\n\n\n<li><strong>Engineering Visibility<\/strong> \u2014 Engineers can identify systemic issues (like a failing integration type) across all projects at once.<\/li>\n\n\n\n<li><strong>Data-Driven Decisions<\/strong> \u2014 With 30-day trend data, the team can track the impact of platform changes on customer projects.<\/li>\n<\/ol>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Lessons Learned<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Start with the questions, not the data.<\/strong> I mapped out exactly what questions each team needed answered before designing schemas or UIs.<\/li>\n\n\n\n<li><strong>Hybrid data sourcing works.<\/strong> Mixing live API data with batch-collected BigQuery data gave me the best of both worlds.<\/li>\n\n\n\n<li><strong>Invest in error handling.<\/strong> When you&#8217;re processing 4,000+ projects, graceful failure handling and retry logic aren&#8217;t optional \u2014 they&#8217;re essential.<\/li>\n\n\n\n<li><strong>Internal tools deserve good UX too.<\/strong> A well-designed interface means higher adoption across teams.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">What&#8217;s Next<\/h2>\n\n\n\n<p>I continue to iterate on the dashboard based on team feedback. Recent additions include job duration tracking, CRM sync backlog visualization, and enhanced failure alerting. The modular architecture makes it easy to add new data sources and visualizations as needs evolve.<\/p>\n\n\n\n<p>Building internal tools might not be as glamorous as building customer-facing features, but the productivity gains they unlock are invaluable. This dashboard is now an essential part of daily operations at Factors.ai.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p><em>Built with React, Node.js, Python, BigQuery, and Google Cloud Run.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>At Factors.ai, we manage over 4,000+ customer projects. Each project generates a wealth of data \u2014 events, user sessions, integrations, CRM syncs, background jobs, and more. While our platform provides&#8230;<\/p>\n","protected":false},"author":1,"featured_media":152,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[8,4],"tags":[],"class_list":["post-65","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-learn-with-me","category-tech-posts"],"jetpack_featured_media_url":"https:\/\/balamurali.in\/blog\/wp-content\/uploads\/2026\/02\/analytics_dashboard.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/posts\/65","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/comments?post=65"}],"version-history":[{"count":1,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/posts\/65\/revisions"}],"predecessor-version":[{"id":67,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/posts\/65\/revisions\/67"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/media\/152"}],"wp:attachment":[{"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/media?parent=65"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/categories?post=65"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/tags?post=65"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}