Summary

Aiven provides a fully managed platform that allows organizations to deploy and scale open-source technologies  like PostgreSQL, Kafka, and OpenSearch across all major clouds. By partnering with Alvin, Aiven sought to solve a complex BigQuery cost optimization challenge within their dbt transformation layer. Through Alvin’s autonomous query routing and reservation management, Aiven successfully eliminated the financial trade-offs of static billing models, achieving a more efficient and predictable data environment.

‍

Challenge

After investigating a move to capacity-based pricing, the Aiven team concluded that making the switch from the on-demand model could actually be more expensive. Along with the additional complexity involved with managing reservations, the Aiven team decided to remain with the on-demand model.

The core issue lay in the heterogenous characteristics of their queries; some were scan-heavy (making on-demand expensive), while others were compute-heavy (making capacity reservations inefficient). Without a way to treat queries individually, Aiven was faced with a choice between two suboptimal "all-or-nothing" pricing models.

‍

Solution

Aiven implemented Alvin’s autonomous agent to resolve their dilemma, attracted by the platform's ability to deliver immediate value with minimal engineering effort. The integration targeted Aiven’s dbt transformation layer, which required a level of granularity that standard BigQuery pricing models could not provide. Alvin sits between Aiven’s dbt pipelines and the BigQuery API, intercepting and optimizing every request in single-digit milliseconds.

• Dynamic Query Routing: Alvin fingerprints every dbt query to analyze historical slot usage and bytes processed. The agent then dynamically assigns each query to the most cost-effective billing model—either on-demand or capacity—ensuring Aiven can reap the full benefits of the different pricing models.
• Automated Reservation Management: To maximize efficiency, the agent manages Aiven’s BigQuery reservations in real-time via API calls. This functionality scales capacity to meet workload demand, ensuring all routed queries stay within Aiven’s SLAs while minimizing spend on underutilized slots, crucial to realising the theoretical cost differential between the two models.
• Performance-Based Savings: Alvin operates on a "no cure, no pay" model, charging a percentage of the realized savings. This ensures a guaranteed positive ROI for Aiven, as costs are only incurred when Alvin reduces their BigQuery spend below the original baseline.

‍

Results

• 45% reduction in BigQuery compute spend for dbt workloads.
• Optimal workload distribution with a balanced cost split between on-demand and capacity models.
• Automated SLA protection through intelligent reservation scaling.
• Zero code changes required to maintain or update the optimization logic.

“Setup was surprisingly easy: straightforward, hands-off, and we saw results quickly. Exactly what you want when you’re running a lean data platform team.”

Stijn Zanders, Senior Manager, Enterprise IT & Data Platform at Aiven

‍

Billing Model Optimization

Alvin analyzes the characteristics of every query executed by Aiven to calculate the most cost-effective path. For scan-heavy queries, Alvin routes them to the capacity reservation to avoid high per-byte charges. For compute-heavy queries, Alvin keeps them on on-demand billing, ensuring Aiven only pays for the data scanned rather than the high slot-seconds required for complex processing.

‍

Automated Reservation Management

Beyond routing, Alvin manages Aiven’s BigQuery reservations in real-time via API calls. This functionality reduces periods of wasted slot consumption by scaling resources up or down to meet the demand of the routed queries at the second level. This ensures that even as Aiven’s transformation needs fluctuate, the effective saving rate remains high without degrading performance.

‍

Ready to Optimize Your Data Spend?

Discover how Alvin can transform your cloud data costs. Schedule a personalized demo or run a free savings analysis to quantify your potential outcomes and see immediate value.

‍