Data Warehouse Automation & Real-time Data – Reducing Time to Value in a Distributed Analytical Environment
For a lot of my clients today, the importance of BI and analytics has never been so high. And it’s pretty clear that that opinion is no longer just in middle management. It is also in the boardroom where the belief now appears to be that BI and analytics are mission critical to success. In a lot of C-level discussions I have nowadays, executives want innovation and discovery but they also want to flatten the hierarchies and get mass contribution to business performance across the employee base. As one COO put it to me “I want to find a way to get everyone pulling on the same rope to achieve common business goals”. This is shown in Figure 1 below.
You might argue that self-service BI would achieve that but all the figures in various surveys show that the usage penetration of BI tools in the employee base of most companies has remained stubbornly stuck around 22%-25%. One reason for that is that much of the use of self-service BI has been at tactical levels of the business – the middle sector shown in Figure 1 and not in every day operations itself. Now the attention is on big data and data science using machine learning to develop predictive models to look into the future but these people are even further away from frontline business operations.
So how do you enable mass contribution to business performance? The answer in my opinion is in operations. It is the base of the pyramid in Figure 1. This is where thousands of decisions are made each day. If we could get that right then we really could move needles on those scorecard KPIs executives are looking at in the boardroom.
Operational BI/Analytics is about using business intelligence and analytics to guide people and applications so that they continuously know the best action to take and when to take it in every business process activity. It is about dynamically using BI and analytics to continuously keep a business running optimally while continuously remaining compliant, minimising risk and maximising profitability.
If insights are to guide people to take actions that all contribute towards achieving common business strategy targets and objectives then the precision of BI delivery needs to be improved so that people get role-based, relevant BI and analytics in the context of every task they perform, as the task is performed. This would enable timely and effective operational decisions. It’s about right-time intelligence in the context of every process activity. This includes catering for the performance of tasks by people who are mobile whether they are employees, suppliers, partners or customers something that will cause a much larger number of concurrent users accessing BI.
But it’s not just about BI and analytics for humans. Many self-service systems like web sites, interactive voice response units at contact centres, bank ATMs, travel kiosks, and even self-checkouts at stores can be made smarter with embedded operational BI and analytics. In this case it is applications that need to receive and be guided by insights. It is about empowering people and systems across the enterprise to be able to act on insights at exactly the right time to optimise and improve on business performance.
Therefore if we are going to be effective in the context of specific tasks in specific operational business processes, it needs to be possible to:
- Embed device agnostic ‘active’ user interface components in applications that not only provide visualisation of insight but also provide drill-down or drill-through to pinpoint issues
- Provide on-demand access to BI and analytical web services
- Provide on-demand access to prescriptive analytics for guidance via recommendations
- Provide business users the ability to go further to customize operational BI by creating their own self-service reports, dashboards and data visualisations inside of a business application without needing support from developers or data scientists.
Another key characteristic of ‘smart’ business operations is the ability to detect, analyse and act as soon as specific events or event patterns occur in business operations. This is known as event-driven analytics which involves determining the business impact an event may have. “Event-driven” analytics occurs when an event or event pattern occurs that may have a ripple effect, perhaps big, perhaps small, perhaps noteworthy, perhaps not. The use of BI/analytics in this case is to figure out the business condition signaled by certain events or a pattern of events, the business impact on the business of this detected condition and what action(s) need to be taken to mitigate risk and/or to keep the business optimised and on track to achieving its goals.
All kinds of events can occur in business operations throughout a working day. Examples include a sale of shares on the financial markets, a price change, an order change, an order cancellation, a customers’ birthday, a large withdrawal on a savings account, the closure of an account, a mouse click on a web site, a missed loan payment, a product or pallet movement in a distribution chain (detected via a radio frequency identification [RFID] tag), a change in flow rate in an oil pipeline, a tweet, a competitor announcement etc. Whatever the events, there are literally thousands of these that can occur in business operations on a daily basis – millions in some cases. And while not all events are of business interest, many require some kind of responsive action to seize an opportunity or prevent a problem occurring or escalating. That response may need to be immediate and automatic in some cases or subject to human approval in others.
What ‘event-driven’ provides is the ability to monitor the ‘pulse’ of business as it happens. It means we can set up ‘always on look-out posts’ (bots) to continuously monitor for specific conditions in different parts of the business and act when they occur. It helps us manage and respond in a more timely way. We can use this capability to assist people in operations, middle management and at executive level. The purpose of event-driven automated operational analytics is to detect events that impact (or are predicted to impact) operational costs, revenue, budget, deadlines and customer satisfaction and to take the appropriate action when they occur. This requires automated analysis, automated decisions and event-driven automated alerts, recommendations and actions
Given these requirements a BI vendor who is focused on embedded operational BI/analytics briefed me recently.
Izenda’s self-service BI and analytics platform is designed for OEM use and embedding in other applications. During the briefing, I was reliably informed that they are currently embedded in over 1500 applications including compliance management solutions, ERP systems etc., with many of their customers running against live data and not just data warehouses.
Izenda is a three-tier web application purpose-built for integration to deliver real-time ad hoc reports and self-service analytics. These tiers are as follows:
- An open and extensible ReactJS presentation layer designed for full integration with existing applications and as a self-service BI portal.
This layer allows for integration of pre-built BI pages, like report designers, lists, viewers, dashboards, scheduling and administrative UIs.
It also supports integration of interactive charts, graphs, grids, KPIs, maps and visualisations into pages throughout the operation of an application.
- A .NET NancyFX Web API which exposes both C# methods and classes, as well as a library of REST API endpoints.
This layer also includes Izenda’s In-Memory Fusion adaptor, which is used for cross database reporting and multi-step calculations.
This lightweight application layer can be deployed to Windows Servers, Azure Web Apps and Cloud Services, or AWS’ Elastic Beanstalk.
- A supported relational or non-relational data source like MSSQL, Azure SQL, MySQL, PostgreSQL, Oracle DB, or MongoDB.
The embedded analytics platform enables users to produce pixel perfect operational reports and interactive drillable, operational dashboards in applications to manage different aspects of business operations at different points of a business process. This enables on-demand, role-based BI in the context of the process activity being performed by a user.
They can access a range of DBMS products that include Microsoft SQL Server, Oracle, MySQL PostgrSQL. MongoDB is also supported for access to high velocity data being written to this NoSQL document DBMS. In addition, given that a number of RDBMS vendors now support in-database analytics, it is also potentially possible to leverage on-demand predictions and recommendations in the context of the task being performed by an operational application. One example of this is SQL Server 2016 in-database analytics and the Microsoft R Server.
Getting Started With Operational BI and Analytics
If you want to get started with operational BI and Analytics there are a few things to consider. You must clearly understand your core business processes and the roles of people who participate in those processes e.g. customer facing call centre operator, bank branch counter staff, store manager, salesperson etc. You must also understand:
- The activities (tasks) they perform
- The applications they use to perform the activities in a process
- The relevant BI, analytics and/or actions needed in each process activity
- BI/Analytics examples – report, drillable dashboard, predictions
- Action examples – alerts, on-demand recommendations
You also need to identify the correct integration strategy the fits the user need (e.g. call centre operators have no time to use BI tools) and how that integration of BI/analytics can help users contribute towards achieving strategic objective(s).
During the briefing with Izenda, one area that they frequently revisited was this need to tailor the analytics experience for the user’s role and ability. Stronger user adoption can be achieved by delivering information and avoiding overloading users.
Another key to getting operations staff to make better data driven decisions is to empower them with the ability to create their own reports, dashboards and data visualisations. This is particularly important for operations managers who may be alerted to issues and require further insights specific to the problem or opportunity. Izenda’s platform allows software companies to embed not only viewers but report and dashboard designers. Organisations can tailor the needs of basic and advanced users. This empowers users as they can create new reports and dashboards, allowing them to address new ad hoc questions as they arise.
System administrators can track usage with the administrative UI, which enables them to identify and push out the most utilized report and dashboard types across the organisation to every tenant. Izenda’s copy management functionality allows for this replication of best practices across different tenants or organisations within a multitenant application.
The ability to and tailor experiences for different users in different roles across business operations by embedding in-context analytics in specific process tasks, enables timely operational decision making and helps maximise the effectiveness of peoples’ and applications’ contribution to overall business performance. This is something that can only get better as operational BI and analytics are iteratively introduces across core business processes in the enterprise.