open-source Articles

281 min read

Data Engineering•2026-02-17

Drastically Reducing Out-of-Memory Errors in Apache Spark at Pinterest

This post describes Pinterest's Auto Memory Retries feature for Apache Spark, which automatically retries OOM-failed tasks on larger executors to reduce failures and resource waste.

engineering

data

apache-spark

open-source

open-source Articles

Related Tags

Drastically Reducing Out-of-Memory Errors in Apache Spark at Pinterest

On the (re)-prioritization of open-source AI

Viaduct, Five Years On: Modernizing the Data-Oriented Service Mesh

Seamless Istio Upgrades at Scale