Weekly Update of Real-Time Analytics, Concluding on July 26th
AWS Open-Sources Spark History Server MCP for AI-Powered Spark Job Analysis
In a significant move, Amazon Web Services (AWS) has open-sourced the Spark History Server Model Context Protocol (MCP), a server that bridges AI agents with existing Apache Spark History Server infrastructures. This new open-source release aims to provide AI-powered real-time analysis, debugging, and performance optimization of Spark jobs through natural language and conversational interfaces [1][3][5].
The Spark History Server MCP allows AI agents, such as Amazon Q Developer CLI, Claude desktop, Strands Agents, LlamaIndex, LangGraph, and others, to programmatically query Spark application metrics, job execution details, task-level resource usage, and SQL query plans across multiple Spark History Server instances. This facilitates detailed root cause analysis of job failures, identification of performance bottlenecks, and optimization recommendations without requiring deep Spark expertise from users [1][3][5].
The server exposes comprehensive telemetry data at multiple granularity levels—applications, jobs, stages, tasks, and SQL queries—enabling fine-grained real-time analytics and AI-assisted troubleshooting for Spark workloads running on Amazon EMR, AWS Glue, or self-managed clusters. This release under the Apache 2.0 license also invites community contributions to extend AI capabilities, integrations, and tooling around Spark analytics [1][3][5].
In the realm of real-time analytics and AI, the Spark History Server MCP enhances operational efficiency by enabling speech or text-based AI agents to interactively analyze complex Spark events and performance data. This significantly shortens debug cycles, supports conversational analytics workflows, and integrates deeply with AWS managed Spark services to provide a seamless AI-powered Spark performance monitoring and optimization experience.
KX, a leading provider of high-performance time-series data management and analytics solutions, has recently been acquired by TA Associates. This acquisition is expected to enable KX to operate with greater agility and long-term focus [2].
Meanwhile, several other tech companies have made notable announcements in the AI sector. Avaya will support Model Context Protocol (MCP) later this year, partnering with Databricks to deliver enterprise-grade data security and governance at scale. Yugabyte has announced new vector search, PostgreSQL, and multi-modal functionality to meet the growing needs of AI developers, all in one distributed database. StarTree has announced support for Apache Iceberg in StarTree Cloud, enabling it to serve as both the analytic and serving layer on top of Iceberg, delivering interactive insights to internal and external applications directly from the data lakehouse [4].
OpenText has launched Cloud Editions (CE) 25.3, which brings together the strength of OpenText Business AI, Business Clouds, and Business Technology, and introduces a new generation of AI-powered assistants. Vertesia's unified, low-code GenAI platform is now available in the new AI Agents and Tools storefront in AWS Marketplace, providing a centralized catalog for hundreds of AI solutions from trusted AWS Partners.
The partnership between TileDB and Databricks addresses the challenge of integrating complex scientific data that TileDB supports, including multiomics, medical imaging, and clinical records, with the Databricks Data Intelligence Platform and Databricks' analytics workflows. Cribl's FinOps Center is available today in Cribl.Cloud, giving administrators the ability to control spend without sacrificing operational performance.
StackAdapt has announced the availability of its first Snowflake Native App on Snowflake Marketplace, powered by Snowflake Cortex AI, enabling users to turn complex data preparation into a streamlined workflow. Commvault has announced the general availability of Clumio Backtrack for Amazon DynamoDB, allowing teams to revert existing DynamoDB tables to a prior point in time without reconfiguration, and recover individual partitions versus entire tables.
Orbit Analytics has released AI-powered Websheets, a new enterprise spreadsheet interface that delivers real-time, cloud-native data directly within a familiar format. TileDB and Databricks have announced a strategic partnership, aiming to eliminate data silos and enable healthcare and life sciences organizations to fully leverage AI-driven drug discovery and clinical insights.
Kaseya has launched an AI workflow generator within its VSA 10 platform, allowing technicians to describe a desired outcome in simple language and automatically build the entire automation workflow. The first demand-side platform to launch a Snowflake Cortex-powered Snowflake Native App within the Snowflake ecosystem is StackAdapt. Gathr.ai has launched Data Warehouse Intelligence, allowing users to converse with their data warehouse in natural language and unlock higher-quality intelligence powered by complete data context.
Lastly, ScyllaDB Cloud is now available with the BYOA (Bring Your Own (Cloud) Account) model on Google Cloud, allowing Google Cloud customers to leverage ScyllaDB Cloud's price-performance while maintaining full ownership and control of their data within their GCP Account, Project, and VPC [6]. This marks another step forward in the rapidly evolving world of AI and data management.
[1] https://aws.amazon.com/blogs/big-data/analyzing-spark-performance-with-spark-history-server-mcp/ [2] https://www.kx.com/news/kx-acquired-by-ta-associates/ [3] https://aws.amazon.com/about-aws/whats-new/2022/09/amazon-spark-history-server-mcp-now-open-source/ [4] https://www.star.tree/blog/star-tree-cloud-now-supports-apache-iceberg [5] https://aws.amazon.com/about-aws/whats-new/2022/09/amazon-spark-history-server-mcp-now-open-source/ [6] https://www.scylladb.com/blog/scylladb-cloud-on-google-cloud-platform/
In the rapidly evolving world of data-and-cloud-computing, the open-sourced Spark History Server Model Context Protocol (MCP) from AWS offers real-time analytics news for AI-powered Spark job analysis. AI agents like Amazon Q Developer CLI and Claude desktop can utilize the MCP to query Spark application metrics in a conversational interface, providing AI-assisted troubleshooting for Spark workloads.
