Introduction
Kovai Limited, as a product company, is always trying to address the monitoring challenges in the Microsoft integration space. BizTalk360 product has more than 500 customers across the world, and ServiceBus360 is also gaining momentum in terms of addressing customer challenges in Microsoft Azure integration space. With strong integration background and decades of experience collectively as a team, we were observing one area where enterprises are still struggling is providing end to end functional visibility into Integration solutions implemented using BizTalk or Azure Logic Apps.
In this article, I will explain why an organization should invest in Atomic Scope for Functional End-to-End monitoring needs of their integration solutions.
Functional end to end monitoring
Integration solutions, specially the one created using BizTalk Server and/or Azure Logic Apps, serve very complex and mission critical business processes such as order processing, payments, patient admission etc. Example of one such complex flow in supply chain industry is as shown below.
Functional support teams who are responsible for handling such complex environments need to have full visibility of message flowing within integration platforms. BizTalk Server or Azure Logic Apps does not have built-in mechanism to show these business flows end to end in a way such that the functional support team can understand.
A functional end to end monitoring solution helps the functional teams to understand the business processes being handled in an integration platform, proactively alerts them for any failures, allows them to search a transaction based on various functional parameters.
Characteristics of enterprise grade functional end to end tracking/monitoring solution
In this section, we will see the characteristics of an enterprise grade functional end to end monitoring solution.
Visibility to functional support team and business
In an integration environment, it is important that the functional support team gets end to end visibility into business process in technology/system agnostic way.
Consider an example in which a BizTalk interface consists of a receive location, orchestration and a send port. We need to know the overall status of the message flows and the data on which an activity can be searched.
An enterprise grade functional end to end monitoring system should also allow you to drill down if there are any issues.
Business people value data metrics and it is important an enterprise grade end to end monitoring solution caters the reports as per the needs.
Defining Business process
An enterprise grade end to end monitoring system should provide the ability to define a business process involving various business transactions and stages in a user-friendly manner as to convert system processing elements such receive ports, orchestrations and send ports into stages which a functional support can understand.
Example: Business process Kovai.Sales has a transaction named Ordering and can have following stages.
| Business Process Name | Kovai.Sales |
| Business Transaction | Ordering |
| Stage 1 | ReceivedOrderFromPartner |
| Stage 2 | BizTalkReceive |
| Stage 3 | BizTalkProcess |
| Stage 4 | BizTalkSendToSAP |
Configuration based tracking
Once an integration solution is instrumented with end to end monitoring framework, the tracking data should be configurable — which means users should have ability to define all the data to be collected in configuration without needing to change the solution.
For example, from stages that we defined in previous section (Defining Business Process), if we were to collect data from Stage3 (BizTalkProcess), we should be able to do it in a portal as shown below.
Search based on functional parameters
Once the tracking data is collected and activities are logged, user should be able to search them based on tracked data as parameters. For example, user might want to search activities with an order number. This gives functional teams an edge by enabling them to quickly look for activities and find the status of a message flow.
Correlating transactions
Integration solutions serve business processes and each business process will have multiple business transactions which are related to each other. An end to end functional monitoring solution should be able to relate these transactions using correlation id. This correlation id should be configurable at each transaction level.
Integrated message archiving
For many organizations archiving messages entering and leaving the integration platform is a need for complying with non-repudiation policies or replaying the messages in during failure conditions. An enterprise grade functional monitoring system should provide ability to integrate archived messages with an activity.
Integrated exception handling
Integration solutions generally adapt exception handling mechanisms for logging business and systems exceptions. An enterprise grade functional monitoring system would allow us to integrate these exceptions with end to end activities. With this functional support team will be able to find the failed activities and drill down to exception details.
Notification for failures
Functional support group needs to know when things go wrong. Example, when a stage fails or when there is any system or business exception. An enterprise grade end to end functional monitoring system should provide immediate and summary notifications to functional support groups.
Options for implementing end to end functional monitoring
In the previous section, we identified the characteristics of an end to end functional monitoring solution. In this section, we will see various options available for implementing such a platform in BizTalk/Logic App/Hybrid integration scenarios.
Options for BizTalk
In BizTalk, there are three different approaches we can take to implement end to end functional monitoring solutions.
- Enable Tracking Data: We can enable tracking on processing elements such as receive ports, orchestration and send ports and make use of tools like BizTalk360 to monitor the tracked instances and use features like Graphical Message Flow (https://www.biztalk360.com/graphical-message-flow/)
| Advantages | Disadvantages |
| You will be able to get end to end flow of messages without additional instrumentation in solutions. All you need is a popular product like BizTalk360 | Tracking data only gives in and out events of the receive ports, orchestrations and send ports which functional team need not be aware of. Taking this approach will lead to very familiar issue of functional support becoming a system support team. |
| Not possible to correlate the transactions easily by using a business correlation token. | |
| If the load is high in BizTalk environments, some customers go with Microsoft recommendation of switching off the property/message body tracking and keep only in and out events tracking. This leads to issues where support team will not get all information required their day to day functioning | |
| Tracking data is not designed for functional end to end monitoring |
In summary, tracking data based monitoring is only suitable for system support teams and not for the functional supports teams and it does not meet requirements of an enterprise grade functional monitoring solution.
- Business Activity Monitoring: BAM is the common approach many organizations have taken and is a Microsoft recommended way of achieving end to end monitoring (https://docs.microsoft.com/en-us/biztalk/core/business-activity-monitoring-bam).
| Advantages | Disadvantages |
| BAM allows you to define milestones and activity data in a tracking profile editor | The BAM definitions are defined during the design time and once they are deployed it’s very hard to update in production environments. If a new data needs to be tracked it becomes very complicated. |
| BAM portal allows you to visualize milestones and data being collected | Many times, data to be tracked will be present in looping structures and it is hard to configure such data collection in Tracking Profile Editor (TPE). |
| BAM does not provide a built-in mechanism to integrate archiving and exception handling with the activities | |
| Since BAM data will be collected in message box database and then sent out to BAM tables using tracking host, there will be additional overhead on the BizTalk run time. | |
| Using BAM for end to end monitoring is a specialist skill and it is hard to find developers with expert level knowledge on BAM |
In summary, using BAM is good for implementing functional end to end monitoring solutions; however we will have to live with some restrictions as mentioned above.
- Custom implementation: Many customers have taken this approach of implementing their custom solutions to cater for end to end monitoring requirements. This will include their own custom database and portal with instrumentation to be used in BizTalk solutions.
| Advantages | Disadvantages |
| If properly designed and implemented, this approach would give all benefits of an enterprise grade functional monitoring solution | The integration teams will be focusing more on implementing a framework than delivering core integration solutions. |
| Due to pressed timelines there is a chance that security and auditing aspects of product could be compromised. | |
| With UI/UX capabilities of portal, it is hard to match an enterprise grade product |
Options for Logic Apps
In Logic Apps, there are three different approaches we can take to implement end to end functional monitoring solutions.
- Built in tracking with OMS portal: Logic Apps has built in feature to track the properties at the action level. An example action definition in a logic app is as shown below.
For more details on using this option refer to documentation https://docs.microsoft.com/en-us/azure/logic-apps/logic-apps-monitor-your-logic-apps#azure-diagnostics-and-alerts
| Advantages | Disadvantages |
| You can make use of built in instrumentation in Logic Apps action definition | This only provides raw materials needed for creating and End to End monitoring framework and is not a full-fledged framework. |
| Tracked data can be visualized in OMS portal | OMS is used mainly by system administrators and not by functional support teams or business. |
| Integrating archiving, exception handling with tracked data and visualizing them in correlation is not straight forward and many cases we hit the wall. | |
| OMS is very generic and not meant for bringing all capabilities of functional end to end monitoring product. |
- Custom implementation: We can implement a custom solution by using custom instrumentation, database and portal which provides all the capabilities however integration teams have to spend significant effort in building such platform.
Hybrid Scenarios involving BizTalk and Logic Apps
Currently there is no straight forward approach an organization can take for end to end monitoring of hybrid solutions involving BizTalk and Logic Apps. In recent Feature Pack releases for BizTalk Server 2016, Microsoft released a feature to push tracking data to application settings where as Logic Apps tracking data is available in Diagnostic logs which are completely different worlds all together.
How can Atomic Scope help businesses?
In the previous section, we considered approaches organizations can take for creating functional end to end monitoring solution and realized that there are gaps and challenges in each approach. This is where Atomic Scope comes to the rescue. It has following features making it an enterprise grade functional monitoring solution.
- Monitor Hybrid Integration Solutions
- Easy Business Process Configuration
- Configuration based tracking data collection
- End-to-end Business Workflow Tracking
- Integrated message archiving & exception management
Summary
Atomic scope is a perfect tool for agile integration teams which are delivering value at high rates. It adds value in the following ways —
- Atomic Scope is a complete end to end functional monitoring solution for BizTalk, Azure Logic Apps and Hybrid integrations.
- It brings maximum visibility into integration solutions to functional support teams and helps them to address the issues quickly.
- Atomic Scope reduces the overall cost of delivering integration solutions by taking away around 20 to 30% of the effort needed for implementing end to end monitoring systems.
- Atomic Scope brings rich and intuitive UI/UX features which our functional teams deserve.
If you want to know more about Atomic Scope, please visit our website. You can also request for a demo of the product or take a 30-day free trial.
