A multi-tenant log-analytics SaaS
AWS-validated reference — full details available to AWS Partner Validation or on request. (Customer name held on file with VeUP.)
For the customer — a multi-tenant log-analytics SaaS ingesting ~4-5 TB/day per customer — VeUP delivered a Well-Architected modernization: Kinesis Firehose ingestion into an S3 log store, hundreds of GraalVM Lambdas for burst search, and a FinOps target of a 30-40% per-search Lambda cost cut.
The challenge
the customer (bronto.io / Logchat) operates a multi-tenant log-visualization-and-analytics SaaS that ingests, stores, and queries customer log data in real time at ~4-5 TB/day per customer at scale. Its ingestion ran on a Lambda log-forwarder path that coupled cost and reliability to a function fleet and lacked managed buffering; burst search over the data is spiky and embarrassingly parallel, so per-search Lambda cost was a growing margin driver (tracked >$2,000/month within ~$467K/yr AWS spend); and a Well-Architected gap analysis was needed to clear a Foundational Technical Review (FTR).
The solution
A Well-Architected modernization of a TB/day log-analytics pipeline: Amazon Kinesis Data Firehose ingestion (buffered, flow-controlled) into an Amazon S3 log store (per-tenant partitioning), over which hundreds of parallel AWS Lambda functions — GraalVM-compiled Java in container images to minimize cold starts at high fan-out — run burst searches feeding real-time visualization, with Amazon MSK (managed Kafka, IAM-auth resolved with AWS Support) in the ingest path and a proposed Amazon Bedrock serving layer for AI-driven log analytics. FinOps layered a hybrid baseline (EC2-hosted Lambda + Savings Plans) plus on-demand peak design and a Graviton re-evaluation. S3-over-warehouse, Lambda-over-cluster, and Firehose-over-log-forwarder selections are documented with their cost tuning.
Architecture
AWS Well-Architected view of the ingestion-and-search estate — from the pre-modernization Lambda log-forwarder path through to the production Kinesis Firehose + Amazon MSK ingestion, the Amazon S3 log store, and the Lambda burst-search fan-out, with the Well-Architected remediation and FinOps plane.
Production outcomes
| KPI | Result |
|---|---|
| Production outcomes | Ingestion modernized from a Lambda log-forwarder path to Amazon Kinesis Data Firehose (buffered, flow-controlled, billing-increment-aware at multi-TB/day); a Well-Architected remediation roadmap closing four high-risk items (fault isolation/bulkhead, undefined RPO/RTO, two cost-optimization findings); and a per-search FinOps target of a 30-40% Lambda cost reduction with no P95-latency regression. Realized per-search cost delta is held in the customer's Cost Explorer and provided via the customer-reference channel. |
| Engagement window | 2026-03 (Art of the Possible / alignment session 2026-03-13) → 2026-04-27 (Well-Architected Review Closed-Won); ongoing |
| Cost / TCO posture | Per-search unit-economics is the platform-margin lever: target a 30-40% reduction in per-search Lambda cost (baseline >$2,000/month, within ~$467K/yr total AWS spend) with a P95-latency guardrail, via a hybrid EC2-hosted-Lambda + Savings Plans baseline, on-demand peak design, and a Graviton re-evaluation; Firehose flow-control aligns records to the 5 KB billing increment to avoid inflated ingestion cost at TB/day. |
| Lessons & continuation | At TB/day, S3 over a warehouse and Lambda fan-out over an always-on cluster are the cost-defensible choices (pay-per-search vs idle capacity); Firehose ingestion must be flow-controlled to the 5 KB billing increment or cost inflates; the per-search cost-down needs a P95-latency guardrail so margin gains can't degrade the product; the managed-Kafka/MSK IAM-auth path needed AWS Support to harden. |
Amazon Kinesis Data Firehose · Amazon S3 · AWS Lambda (GraalVM / Graviton) · Amazon MSK · Amazon Bedrock (proposed serving layer) · AWS Cost Explorer / Savings Plans · AWS Well-Architected Tool