Today we are open sourcing the Xonai Dashboard, a Grafana-based application to assist Big Data infrastructure optimization initiatives where Spark applications are a dominant cost driver.
Why We Built It
Data-driven organizations rely on Spark jobs running on large clusters to power their offerings. As growing cloud costs prompt infrastructure optimization initiatives, engineers often find themselves ill-equipped with generic monitoring tools disconnected from the bottom line: the actual applications driving resource utilization. This disconnect makes it difficult to pinpoint where inefficiencies originate from and waste valuable engineering time.
We built the Xonai Dashboard to facilitate infrastructure optimization of both Spark applications and their execution environment.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
This is a placeholder image.
Why We Built It
Data-driven organizations rely on Spark jobs running on large clusters to power their offerings. As growing cloud costs prompt infrastructure optimization initiatives, engineers often find themselves ill-equipped with generic monitoring tools disconnected from the bottom line: the actual applications driving resource utilization. This disconnect makes it difficult to pinpoint where inefficiencies originate from and waste valuable engineering time.
We built the Xonai Dashboard to facilitate infrastructure optimization of both Spark applications and their execution environment.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
This is a placeholder image.
Why We Built It
Data-driven organizations rely on Spark jobs running on large clusters to power their offerings. As growing cloud costs prompt infrastructure optimization initiatives, engineers often find themselves ill-equipped with generic monitoring tools disconnected from the bottom line: the actual applications driving resource utilization. This disconnect makes it difficult to pinpoint where inefficiencies originate from and waste valuable engineering time.
We built the Xonai Dashboard to facilitate infrastructure optimization of both Spark applications and their execution environment.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
This is a placeholder image.
How It Works
The Xonai Dashboard aggregates Spark execution metrics and cloud cost estimates for entire clusters and down to each individual application with the goal of exposing optimization opportunities.
In a nutshell, it combines the features of applications such as Ganglia, Spark history server, EMR console metrics, into one centralized application tailored for observability and optimization of Spark clusters.
As this initial release exclusively supports Amazon EMR and Databricks on AWS, the entry point is a dashboard providing either a top-down overview of active and terminated EMR clusters or Databricks Job Clusters for a selected time range – not too different from the EMR console – but with additional aggregated hardware utilization metrics such as CPU and Memory and red flagging if these are underutilized. In addition, costs are also estimated for each cluster based on EC2 instance and data platform prices and assuming no savings or commitment plans.
Clicking on any cluster link in the overview table will open the Cluster Overview dashboard, breaking down costs, Spark applications launched and detailed utilization metrics underneath.
Metrics from each individual node (e.g. EC2 instances) can also be obtained by clicking on instance ID links on the "Instances" table.
Last, clicking any Spark application ID link in the Cluster Overview dashboard will open an Application Overview dashboard, presenting Spark execution metrics presented in a more intuitive manner compared to the Spark history server, as well as a cost estimate for the application in particular.
Deploying the Xonai Dashboard is straightforward: All components are installed by executing a single script and clusters are monitored after adding a bootstrap action that activates metric collector daemons; no code changes are required.
Find the prerequisites for activation and all installation steps in our GitHub repository.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
This is a placeholder image.
How It Works
The Xonai Dashboard aggregates Spark execution metrics and cloud cost estimates for entire clusters and down to each individual application with the goal of exposing optimization opportunities.
In a nutshell, it combines the features of applications such as Ganglia, Spark history server, EMR console metrics, into one centralized application tailored for observability and optimization of Spark clusters.
As this initial release exclusively supports Amazon EMR and Databricks on AWS, the entry point is a dashboard providing either a top-down overview of active and terminated EMR clusters or Databricks Job Clusters for a selected time range – not too different from the EMR console – but with additional aggregated hardware utilization metrics such as CPU and Memory and red flagging if these are underutilized. In addition, costs are also estimated for each cluster based on EC2 instance and data platform prices and assuming no savings or commitment plans.
Clicking on any cluster link in the overview table will open the Cluster Overview dashboard, breaking down costs, Spark applications launched and detailed utilization metrics underneath.
Metrics from each individual node (e.g. EC2 instances) can also be obtained by clicking on instance ID links on the "Instances" table.
Last, clicking any Spark application ID link in the Cluster Overview dashboard will open an Application Overview dashboard, presenting Spark execution metrics presented in a more intuitive manner compared to the Spark history server, as well as a cost estimate for the application in particular.
Deploying the Xonai Dashboard is straightforward: All components are installed by executing a single script and clusters are monitored after adding a bootstrap action that activates metric collector daemons; no code changes are required.
Find the prerequisites for activation and all installation steps in our GitHub repository.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
This is a placeholder image.
How It Works
The Xonai Dashboard aggregates Spark execution metrics and cloud cost estimates for entire clusters and down to each individual application with the goal of exposing optimization opportunities.
In a nutshell, it combines the features of applications such as Ganglia, Spark history server, EMR console metrics, into one centralized application tailored for observability and optimization of Spark clusters.
As this initial release exclusively supports Amazon EMR and Databricks on AWS, the entry point is a dashboard providing either a top-down overview of active and terminated EMR clusters or Databricks Job Clusters for a selected time range – not too different from the EMR console – but with additional aggregated hardware utilization metrics such as CPU and Memory and red flagging if these are underutilized. In addition, costs are also estimated for each cluster based on EC2 instance and data platform prices and assuming no savings or commitment plans.
Clicking on any cluster link in the overview table will open the Cluster Overview dashboard, breaking down costs, Spark applications launched and detailed utilization metrics underneath.
Metrics from each individual node (e.g. EC2 instances) can also be obtained by clicking on instance ID links on the "Instances" table.
Last, clicking any Spark application ID link in the Cluster Overview dashboard will open an Application Overview dashboard, presenting Spark execution metrics presented in a more intuitive manner compared to the Spark history server, as well as a cost estimate for the application in particular.
Deploying the Xonai Dashboard is straightforward: All components are installed by executing a single script and clusters are monitored after adding a bootstrap action that activates metric collector daemons; no code changes are required.
Find the prerequisites for activation and all installation steps in our GitHub repository.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
This is a placeholder image.
Contributing
The Xonai Dashboard was just released. We plan to extend support for other cloud providers and platforms such as Azure Databricks and Databricks for Google Cloud, as well as facilitating installation on private cloud environments and making it better at pinpointing issues in monitored clusters.
Contributing with bugfixes, improvements and suggestions for new features from the community are very welcome.
Don’t forget to star it!
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
This is a placeholder image.
Contributing
The Xonai Dashboard was just released. We plan to extend support for other cloud providers and platforms such as Azure Databricks and Databricks for Google Cloud, as well as facilitating installation on private cloud environments and making it better at pinpointing issues in monitored clusters.
Contributing with bugfixes, improvements and suggestions for new features from the community are very welcome.
Don’t forget to star it!
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
This is a placeholder image.
Contributing
The Xonai Dashboard was just released. We plan to extend support for other cloud providers and platforms such as Azure Databricks and Databricks for Google Cloud, as well as facilitating installation on private cloud environments and making it better at pinpointing issues in monitored clusters.
Contributing with bugfixes, improvements and suggestions for new features from the community are very welcome.
Don’t forget to star it!
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
The rich text element allows you to create and format headings,
Sample text is being used as a placeholder for real text that is normally present. Sample text helps you understand how real text may look on your website. Sample text is being used as a placeholder for real text.
This is a placeholder image.
Sign up to our newsletter so you can be the first to find out the latest news.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Featured Posts
Sorry, we couldn't find articles in the same category.
