Sort providers by cost, latency, or throughput on AI Gateway | Endigest
Vercel
|AIGet the latest tech trends every morning
Receive daily AI-curated summaries of engineering articles from top tech companies worldwide.
AI Gateway now allows sorting providers by cost, latency, or throughput to optimize request routing based on specific needs.
- •Three sorting options available: 'cost' for lowest price, 'ttft' for lowest latency, 'tps' for highest throughput
- •Sorting is computed at request time, automatically reflecting provider changes, price updates, and latency shifts without code changes
- •Compatible with other gateway routing options like Zero Data Retention (ZDR) for combined filtering and sorting
- •Providers are tried in sort order with fallback to next provider only when current one is unavailable
- •Every response includes routing metadata showing providers considered, metric values, execution order, and deprioritized providers
This summary was automatically generated by AI based on the original article and may not be fully accurate.