Run Interactive Workloads on Amazon EMR Serverless with Spark Connect

Amazon Web Services has launched interactive session support for Amazon EMR Serverless through Spark Connect, allowing developers to run Apache Spark applications interactively from managed notebooks in Amazon SageMaker Unified Studio as well as popular development environments like Jupyter and Visual Studio Code. The new capability provides a persistent Spark context that spans across cells and scripts, enabling developers to blend local Python code execution with remote Spark operations in a unified environment. The implementation leverages Spark Connect's client-server architecture to decouple application clients from the Spark driver, allowing developers to maintain their preferred development tools while Spark infrastructure runs independently on EMR Serverless. This architecture enables workflows including ad hoc data exploration, iterative debugging, and incremental PySpark job development before production deployment. The service includes comprehensive observability features with real-time session monitoring through the Spark UI, history tracking via the Spark History Server, and session management through the EMR console or API/CLI/SDK. Spark Connect on Amazon EMR Serverless is available with EMR release 7.13 across all AWS regions where EMR Serverless operates, while the SageMaker Unified Studio integration is available in supported regions. The enhancement addresses a key developer workflow challenge by providing seamless integration between local development environments and scalable cloud-based Spark processing infrastructure.