Microsoft Integration tools, monitoring requirements and challenges

Overview on Integration

Integration has been defined as connecting different applications or different systems together. With the growing need to communicate between different systems data has to be encrypted or it has to follow certain standards. Electronic Data Interchange (EDI) has also come into picture certain rules for how to create a document.

Now with the advent of digital transformation, organizations will look to leverage some of the features in the cloud. We might face a situation where we need to migrate some of our on-premise systems to cloud and we need communicate between on-premise and cloud platforms.

Gartner predicts that 65% of large organizations will have implemented a Hybrid Integration Platform (HIP) to power their digital transformation by 2022 (Hybrid Integration From Gartner). So, integration will play a pivotal role in many areas such as B2B, Applications, Data Migration, IOT etc.

I will be discussing some of the Microsoft Integration Tools below and how to leverage some of them.

Microsoft BizTalk Server

For about two decades BizTalk has been one of the prominent options for integration. BizTalk was developed by using publish-subscribe architecture. In short, BizTalk picks up a message and publishes into the Message Box. Send Ports or Orchestrations subscribes to messages, applies the business logic and sends the message. BizTalk has a wide variety of Line-Of-Business (LOB) adapters which can easily connect different channels.

BizTalk supports Enterprise Application Integration (EAI) and Business-to-Business (B2B) messaging scenarios. Over time a need arose to monitor the integrations and push exceptions to a unified place. BizTalk supports these situations in the form of Business Activity Monitoring (BAM), Enterprise Service Bus Architecture. BizTalk can support event triggering and scheduled triggering as well. BizTalk can also support EDI Functionality with the help of X12, AS2, EDIFACT protocols. Users can automate the Business Process with the help of orchestrations. They can transform the messages with the help of XSLT.

We can easily configure the CICD pipeline with the help of BizTalk Deployment Framework and VSTS which is also now called Azure DEVOPS. We can also have our own PowerShell scripts to customize our deployments. It will also help us to configure some of the monitoring as well.

With the advent of Cloud, we need to support interaction between on-premise and cloud. In the recent versions of BizTalk, we see LogicApps and SB Messaging Adapter which will help us interact between Azure and on-premise. BizTalk can also interact with Rest API’s through the use of web-HTTP adapter. With the recent releases of Feature Packs, we can see Microsoft releasing Outlook 365 Adapters, BizTalk Statistics connecting to PowerBI and all the BizTalk Artifacts being exposed through Rest Service which can help us open the thought of Citizen Integrators. Microsoft has also announced that there will be a next version of BizTalk i.e. BizTalk Vnext. But organizations might need to also look at the transitioning into cloud and leverage some of the cloud benefits.

Limitations

• Even though there is BAM to monitor the transactions and ESB exception portal to hold the exceptions, there is no one unified portal to track all these transactions. These are all separate components and it takes time to install them.
• Understanding Health Tracking of BizTalk is also paramount. At present we have BizTalk Health Monitor which we need to install as a separate product. But it can only solves a limited purpose. We probably need something which can Monitor the Health of BizTalk such as Throttling, Alerts, Statistics etc.
• For some of the Applications we might also look at creating reports and see how many transactions are being processed in BizTalk etc. which helps us to do some real time monitoring.
• BizTalk engine which hugely relies on the XML standard which has been the standard for many years. Now with the support of Mobile application etc., JSON has become an important standard which we need to support. Presently BizTalk supports it by using Json components at the Pipeline stage.

Microsoft supports BizTalk 2016 version up to 2022. Even though there is an advent of cloud, BizTalk will still have its place in this space where clients might want to hold some of the sensitive information on-premise and some of the applications in Cloud.

Azure Integration Services

Azure Integration Services are a set of components which will help you to integrate systems across on-premise and cloud. It helps you to support Service Oriented Architecture, Messaging solutions and Event Triggers etc.

Below are the Services which will help us to do the job.

API Management

API Management helps us to publish all the external and internal API’s and to create a single Gateway. Through its features we can secure our API’s and make it more manageable. It can act as an External Layer or Façade of you API.

Some of the Key Features we can apply with API management with the help of creating polices at the inbound and outbound level are as follows

• Rate Limiting: we can help our API to limit to certain number of transactions per minute.
• Response Caching: Helps us to reduce the Latency and load for data that does not change frequently.
• Authentication: Helps our API’s to access securely.
• Conversions: we can apply the conversions like Json to XML or XML to Json at the inbound or the at the outbound processing.

Still there are lot of features in API management which can help us to manage our API’s securely. API Management also offers us a rich set of analytics like how many times my API Methods are called etc. which will help us to take the decisions effectively.

Logic Apps

Logic Apps is one of the Azure Features which helps us to orchestrate the business workflows and automate the business process. It offers us a rich set of connectors which can help us to connect wide variety of Applications across the cloud and on-premise as well using the On-Premise Gateway etc.

Some of the features are

• Lots of prebuilt Microsoft Managed Connectors which will help us to develop the solutions very quickly.
• Ability to develop Logic Apps from visual studio or VS Code which was announced recently so that we can check-in and later configure the CI/CD Pipeline as well
• Ability to connect to on-premise systems using On Premise Gateway. It can help you to connect to SQL, Oracle, File System etc.
• In addition to prebuilt Connectors, we also the ability to create our own Custom Connectors and publish them so that we can use it in our Logic Apps.
• Can connect to Azure Functions as well to create your own logic and execute them inside the Logic App.
• With the help of an integration account Logic Apps can Support Business -to-Business scenarios Supporting X12, EDI Fact Protocols etc.

Logic Apps can also connect to BizTalk server using on-premise Gateway and BizTalk can connect to Logic Apps as well using the Logic App Adapter. This will help us to create interaction between on-premise and cloud systems which will help us to create Hybrid integration.

Service Bus

Service Bus is one of the solutions for messaging scenarios communicating between on-premise and cloud. It consists of queues, topics and relays. It helps us to create asynchronous messaging, point-to-point messaging, and publish-subscribe to messages using topics.

Some of the features are below

• Ability to Ordered Message Delivery with the help of Queues.
• Subscribe to messages by creating subscription through Topics.
• Create Sequential Patterns with the combination of Service Bus Sessions and Logic Apps. This is to ensure that Messages are grouped together and processed in one application.
• Allows clients to expose Endpoints in cloud using Azure Relay Service.
• Support Dead Lettering to ensure that the messages that are failed can be processed at later time so that no messages are lost.

Event Grid

Azure Event Grid helps us to create reactive solutions. It helps us to react to events in a real time. At present events can be published for Resource Groups, Blob Storage etc. it can be subscribed to by Azure Logic Apps, Functions etc.

Some of the Features are as below.

• Helps you to create Publish-Subscribe Model and Serverless Apps.
• Seamless Integration with Azure Services.

Adding to the below, Azure Functions can play a pivotal role for Integration. In short Azure Functions is a place where we write our own code (C#, Javascript etc.). It can seamless integrate with Logic Apps as well and it can run on different operating systems. Since these components don’t require any infrastructure, they open up the gates for designing Serverless Architectures. We don’t need to depend on infrastructure or maintenance.

We can deploy all these tools through the use of ARM Templates and configure the CICD Pipeline so that we can easily deploy them in Azure.

We have many tools for integration in Azure. Picking the right tool is important depending on scenarios and how you are using it.

Limitations

• Costs might spike if your applications are busy and running heavy load.
• We might have OMS etc. and some other set of tools in Azure which help us to monitor these services. With the Hybrid Integration, we might want all at one place where we can see all our Services and Analytics.

Monitoring

With the advent of transactions present in on-premise and Cloud, monitoring has become much more essential. Below are the few instances which I would feel a monitoring tools are essential

• For people who have worked on BizTalk for a long time, all of us will understand that monitoring the Health of BizTalk is very important. Some of the cases might be to make sure Hosts are not throttling for a consistent period of time, Tracking DBs not getting full. So we might need something to monitor and configure alerts every day.
• In some of the solutions where there is lot of business processes involved, you might need step by step Tracking as well. BAM Portal in a way can give us the solution. But you might need a more sophisticated solution than this to Track the instances.
• At present, BizTalk has BAM portal to Track some of the Data, ESB Portal to Track the Exceptions, BizTalk Health Monitor Or Message Box Viewer to get the health of BizTalk. So There might be a need to have one Unified Portal to monitor all of them.
• With the advent of Hybrid Integration where you will see lot of Transactions moving between On Premise and Cloud, we would need some way to Track these records and understand them.
• In a lot of situations where your Tracked Dbs are getting full or you want to know when Suspended Instances in one Business Application Reaches a Threshold etc. alerts Should be generated which can help you to make corrective action.
• Analytics are also important in present day scenarios where you want to check how many times your Logic App workflows have run to monitor the costs and to understand the traffic in BizTalk Server. This can help us to create our solutions more effectively.
• In some situations, you might also need to control the access so that everybody cannot login to BizTalk Admin Console and perform operations. So you might need some other way to monitor it.

All and all we need a unified portal which is easy to install. We have tools like AIMS, BizTalk 360 etc. which can help us to monitor more effectively in the below ways.

• Real Time Monitoring
• Analytics and Deep insights
• Creating Custom Dashboards

Conclusion

In the past, we have seen that integration has been a black box or integration is only for few people. It’s not anymore. Large number of customers will be using integration and effective monitoring tools with it.

Read more about Hybrid Integration Performance Monitoring