As Microsoft says, “Monitoring is the act of collecting and analyzing data to determine the performance, health, and availability of your business application and the resources it depends on”.
An effective monitoring strategy will help you understand the detailed operation of the different components of your application and increase your up time by proactively notifying you of critical issues so that you can resolve them before they become problems. Also, an effective strategy requires that the set-up and maintenance of the monitoring is automated, lightweight and does not require significant manual resources.
About Event hub
Azure Event Hubs is a big data streaming platform and event ingestion service. It can receive and process millions of events per second. Data sent to an event hub can be transformed and stored by using any real-time analytics provider or batching/storage adapters.
The following scenarios are some of the scenarios where you can use Event Hubs:
- Anomaly detection (fraud/outliers)
- Application logging
- Analytics pipelines, such as clickstreams
- Live dashboarding
- Archiving data
- Transaction processing
- User telemetry processing
- Device telemetry streaming
Why use Event Hubs?
Data is valuable only when there is an easy way to process and get timely insights from data sources. Event Hubs provides a distributed stream processing platform with low latency and seamless integration, with data and analytics services inside and outside Azure to build your complete big data pipeline.
Event Hubs represents the "front door" for an event pipeline, often called an event ingestor in solution architectures. An event ingestor is a component or service that sits between event publishers and event consumers to decouple the production of an event stream from the consumption of those events. Event Hubs provides a unified streaming platform with time retention buffer, decoupling event producers from event consumers.
Source: Microsoft
Why Monitor Azure Event Hub?
As described by Microsoft, Event Hub is used in several important use cases processing massive events to empower users with efficient processing of data, analytics and insights.
It should be needless to say that the availability and performance of Event Hub is key for users leveraging Event Hub in applications or processes their company rely on. Event Hub is a Microsoft managed service, but you still need to take responsibility and control of your actions and code which could impact performance and costs but also potentially ensure you get insight (that is not necessarily related to performance degradation) that could by valuable business insight to your organization.
Monitoring Azure with Azure's tooling is difficult
Monitoring in Azure is primarily provided by Azure Monitor which provides common stores for storing monitoring data, multiple data sources for collecting data from the different tiers supporting your application and features for analyzing and responding to collected data.
Monitoring Azure with Microsoft's built in tooling is difficult because:
1. The data is all over the place
2. Setting up monitoring and alerting is a manual job
3. Maintenance is time consuming
4. Relies mostly on traditional reactive monitoring
5. Does not efficiently solve silo challenges across cloud and on premise systems
As a consequence monitoring with the built in Microsoft tools tends to fall into the same traps as other antiquated tools that sets your organization up for failure.
You can read more about monitoring of Azure with Microsoft's tools here.
Why you should consider using AIMS to Monitor Azure Event Hub?
AIMS is built on the fundamental value proposition of automating monitoring and insight into increasingly complex enterprise IT systems with a focus on the applications or business processes supported. With Azure, the number of applications – or services – explodes taking the growth in complexity from linear to exponential.
Each of the services or applications requires monitoring of relevant performance metrics. Taking the number of services, systems or applications from tens to thousands with cloud means that you should expand the metrics you monitor by a similar factor – that is probably about 100-fold. That cannot be done by adding manual monitoring or human resources.
Normal behavior
Monitoring of this complexity needs to be done by using machine learning and algorithms. AIMS’ monitoring for Azure fetches all relevant performance metric for each IaaS or PaaS in Azure. For each service we fetch all available metrics in real time and AIMS build normal behavior patterns for each metric. This normal behavior metric represents the cyclicality of your business by different time resolutions – minutes a day, hour per day or day per week. This means that AIMS builds a digital DNA of how your business lives – represented through these normal behavior patterns. (see image below for example)
Metrics
Now, with this digital DNA that consist of thousands of normal behavior patterns AIMS will monitor in real time the current performance vs the patterns build while also dynamically updating the normal behavior patterns with the new performance data fetched. The effect is that all available metrics are monitoring in real time vs the normal behavior pattern.
For Event Hub AIMS fetch and build normal behavior patterns for the following metrics:
-
Failed Requests
-
Size
-
Archive backlog messages
-
Incoming Messages
-
Outgoing Messages
-
Captured Bytes.
-
Connections Opened.
-
Connections Closed.
-
Active Connections
-
Successful Requests
-
Incoming Requests
-
Throttled Requests.
-
Archive message throughput
-
Archive messages
-
Captured Messages.
-
Capture Backlog.
-
Outgoing bytes
-
Incoming Bytes.
-
Quota Exceeded Errors.
-
Internal Server Errors
-
Server Busy Errors
-
User Errors.
-
Server Errors.
-
Other Errors
Anomalies
Using normal behavior patterns, AIMS identifies anomalies on each single metric vs normal behavior and correlates the anomalies across the rest of the metrics to create Anomaly Warnings.
Anomaly Warnings provide early notification of trouble with Azure Services or across cloud / hybrid environments.
Auto-detection
You can also use the “Activity & changes” module in the Analytics tab to create reports & dashboards to identify new resources & applications in Azure, which allows early identification of new resources and applications in Azure to ease insight and transparency.
AIMS will do all this without you needing to manually intervene, and there are no static thresholds needed!
AIMS Event Hub reports & dashboard
All the data collected by AIMS is available to users in the Analytics section that allows you to access default reports or to create custom reports and dashboards. When creating your first report, you will discover that the AIMS Analytics consists of a wide range of possibilities. You can create public or private reports, set a report as a dashboard and define which users will receive dashboard reports.
AIMS stores performance data, making it a valuable data source for later use – such as for root-cause analysis. More about reports, dashboards and analytics.
So, summing up: with AIMS you can automate monitoring of your Azure IaaS and PaaS through a easy configuration that can be done in minutes. This will allow you to free up valuable time spent monitoring using other tools or manual monitoring. The consequence is that you will have better monitoring and more time to deliver projects for your business.
Learn more about how to use AIMS to monitor Event Hub or sign up for the free Community Edition and test it out for yourself.
Topics from this blog: Blog
