A client-side application security and compliance vendor
AWS-validated reference — full details available to AWS Partner Validation or on request. (Customer name held on file with VeUP.)
VeUP verified the architecture-of-record for the customer ~3-billion-events/month client-side threat-detection pipeline (API Gateway → Lambda → Kinesis → OpenSearch, with Firehose → S3) and delivered a costed, success-criteria-bound modernization plan mapping each documented reliability and cost driver to a specific AWS-native remediation.
The challenge
the customer platform detects client-side threats by ingesting browser-side telemetry at scale — up to roughly 3 billion events per month, with individual customer projects able to spike to ~0.5 billion events in a month. That real-time detection plane has to stay reliable and fresh under bursty, highly uneven traffic, and its cost has to scale sub-linearly with event volume. In discovery, the streaming-ingest pipeline showed classic high-volume failure modes: hot Kinesis shards driven by an uneven partition key, OpenSearch write-amplification from frequent in-place updates, and overage traffic only rejected after it had already been ingested and paid for. the customer needed a partner with deep AWS streaming and FinOps expertise to harden the pipeline for reliability at peak and to bend its cost curve.
The solution
VeUP ran a discovery of the live pipeline and produced an engineering Art-of-the-Possible proposal with a scoped proof-of-concept plan. Architecture-of-record verified in discovery: Amazon API Gateway → AWS Lambda → Amazon Kinesis Data Streams (ingest plus an enriched stream) → ~15–20 AWS Lambda consumers → Amazon OpenSearch Service for hourly aggregates, with Amazon Kinesis Data Firehose → Amazon S3 for raw-event backup and replay. Target patterns mapped to each verified failure mode: (1) de-skew the Kinesis partition key to eliminate per-project hot shards; (2) left-shift entitlements and throttling to the edge so the platform stops paying to ingest and then reject overage; (3) move to an append-only write pattern with materialized aggregates and OpenSearch index lifecycle management to remove update-churn write-amplification; (4) per-ARN / per-resource FinOps instrumentation across Kinesis, Lambda, and OpenSearch. The plan scoped an optional real-time OLAP pilot alongside OpenSearch and defined measurable success criteria: near-zero throttled/dropped events at peak, near-real-time freshness (sub-60–90s) plus hourly rollups, and a lower cost-per-ingested-event.
Production outcomes
| KPI | Result |
|---|---|
| Production outcomes | A verified architecture-of-record for the ~3-billion-events/month detection pipeline and a costed, success-criteria-bound modernization plan that maps each documented reliability and cost driver to a specific AWS-native remediation — de-skewed Kinesis sharding, edge-enforced entitlements, an append-only OpenSearch pattern with index lifecycle management, and per-resource FinOps instrumentation. The engagement also advanced the customer AWS Marketplace listing and established an ongoing SA-on-demand advisory relationship, with a Q1 alignment review on file. |
| Engagement window | 2024-11-25 (AWS referral / resell onboarding); Art-of-the-Possible 2025-09-30 → Architecture-of-record + modernization plan delivered; SA-on-demand advisory ongoing |
| Cost / TCO posture | FinOps was load-bearing: discovery identified Kinesis and Lambda (read and write) as the largest cost drivers, with OpenSearch cost rising from update churn. The plan targets a lower cost-per-ingested-event via edge-enforced entitlements (stop paying to ingest-then-reject overage), append-only OpenSearch with index lifecycle management, and per-ARN cost attribution so dominant drivers stay visible as volume grows. ~$458K/yr AWS spend at scale. |
| Lessons & continuation | On high-volume streaming-ingest, the dominant reliability and cost risks are structural — an uneven partition key (hot shards), update-churn write-amplification, and post-ingest overage rejection. Attributing cost per ARN across Kinesis/Lambda/OpenSearch is what keeps the savings durable. Realized post-PoC deltas should be captured from the customer before any reduction percentage is published. |
Architecture
The Well-Architected view maps the verified architecture-of-record — Amazon API Gateway → AWS Lambda → Amazon Kinesis Data Streams → Amazon OpenSearch Service, with Kinesis Data Firehose → Amazon S3 for raw-event backup — to the four target remediations scoped in the Art-of-the-Possible plan: a de-skewed Kinesis partition key, edge-enforced entitlements, an append-only OpenSearch pattern with index lifecycle management, and per-ARN FinOps instrumentation.
Amazon API Gateway · AWS Lambda · Amazon Kinesis Data Streams · Amazon Kinesis Data Firehose · Amazon OpenSearch Service · Amazon S3