Data Eng Articles

This post covers Databricks' Predictive Optimization (PO) in Unity Catalog, which became the default platform behavior in 2025 for autonomous lakehouse table maintenance.

Platform

Product

Figma

241 min read

Data Engineering•2026-02-18

Redefining impact as a data scientist

This post explores how data science work at Figma's Billing infrastructure differs from traditional product analytics, offering five lessons on expanding impact in complex, correctness-driven domains.

281 min read

Data Engineering•2026-02-17

Drastically Reducing Out-of-Memory Errors in Apache Spark at Pinterest

This post describes Pinterest's Auto Memory Retries feature for Apache Spark, which automatically retries OOM-failed tasks on larger executors to reduce failures and resource waste.

Data Engineering•2026-02-10

Hydra joins Supabase

The Hydra team, maintainers of the pg_duckdb extension, is joining Supabase to advance Postgres-native analytics capabilities.

21 min read

Data Engineering•2026-02-05

Next Generation DB Ingestion at Pinterest

Pinterest describes its next-generation database ingestion framework built on CDC, Kafka, Flink, Spark, and Iceberg to replace legacy batch-based pipelines.

Data Engineering•2026-01-06

Lyft’s Feature Store: Architecture, Optimization, and Evolution

This article describes the architecture, optimization, and evolution of Lyft's Feature Store, a core ML infrastructure platform serving 60+ use cases across the rideshare stack.

Data Engineering•2025-12-18

Announcing support for GROUP BY, SUM, and other aggregation queries in R2 SQL

Cloudflare announces support for GROUP BY, SUM, and other aggregation queries in R2 SQL, its serverless analytics query engine over R2 Data Catalog.

Data Engineering•2025-12-18

How Grab is accelerating growth with real-time personalization using Customer Data Platform scenarios

Grab built 'Scenarios' in their CDP to enable real-time personalization beyond daily batch updates.

Data Engineering•2025-12-08

Introducing iceberg-js: A JavaScript Client for Apache Iceberg

This post introduces iceberg-js, a minimal JavaScript client for the Apache Iceberg REST Catalog API targeting JavaScript and TypeScript developers.

AWS

21 min read

Data Engineering•2025-12-02

Announcing replication support and Intelligent-Tiering for Amazon S3 Tables

Amazon S3 Tables announces Intelligent-Tiering storage and cross-region replication support for Apache Iceberg tables.

Data Engineering•2025-11-26

Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic test

This post introduces Coban, Grab's platform for real-time Kafka stream data quality monitoring using user-defined data contracts with syntactic and semantic test rules.

Data Engineering•2024-09-30

Enhancing Data Quality Using Better Designed ETLs

This article presents an ETL design document template used at Square to improve data quality, team consistency, and documentation practices.

Data Science