Data Warehouse Automation & Real-time Data – Reducing Time to Value in a Distributed Analytical Environment
Happy New Year to all of you! It’s that time of year again when predictions are made so I thought I would throw my thoughts into the ring for debate. So here goes:
- Consolidation in the BI market will continue now that we have seen the software giants make their moves in 2007. Oracle bought Hyperion, SAP bought Business Objects and IBM announced acquisition of Cognos. In order to compete with this, other vendors will consolidate to try to offer and alternative. So expect to see more mergers in 2008
- The cost of the BI platform will continue to drop amid pressure from the software giants (Microsoft in particular), and open source alternatives (e.g. Pentaho, Jaspersoft, Talend et al). The money to be made is in Performance Management and Data Management.
- Both Performance Management and Data Management technologies will separate from BI platforms (if they haven’t done so already) and become suites of tools in their own right
- The growth in the size of the data management market is set to continue as companies try to standardise of a suite of tools for enterprise data management (an enterprise data management platform) which includes an end-user business vocabulary tool, data modelling tool, data discovery and mapping tool, data quality profiling, data cleansing, data integration (consolidation, federation and synchronisation). This data management platform will be used for data replication, data warehousing, data migration, master data management, data synchronisation and on-demand data management services published in a service registry and available on an enterprise service bus (ESB) in a service oriented architecture (SOA)
- Complex Event Processing (CEP) will become mainstream in 2008 as companies try to analyse data and the business impact of events well before that data arrives in any kind of data warehouse or data mart. This is also known as business activity monitoring (BAM) except we are going to see monitoring of complex events (on the lookout for several events happening before triggering action) and not just single ones
- 2008 will be the year of massive growth in memory exploitation. We will see parallel query execution continue to run across multiple shared nothing nodes in MPP systems with multiple processors, and multiple disks (as is the case today in many parallel relational DBMSs). However the difference here is that we will see this happening against in-memory data on a massively parallel scale in 2008 and beyond. With the volumes of data about to climb higher, and demand for CEP on the increase, we need to access data in memory to respond more rapidly and keep performance optimal. Massively parallel memory is therefore inevitable and will arrive on the scene this year whether that memory be in a single cluster server or deployed over a grid in a virtual memory configuration
- Performance Management is set to grow with BAM, process management, scorecards, dashboards, budgeting and planning and Business Intelligence all being integrated into a component performance management suite (enterprise performance management platform). Performance Management platforms will sit on top of BI platforms but will also integrate with other enterprise infrastructure software such as business process management, portals, enterprise content management systems and live collaboration tools.
- Web 2.0 collaboration will push its way into Performance Management. In particular, socially networked performance management will start to appear so that end users can tag metrics, graphs and reports in order to organise BI and PM content. This user defined categorising of content via tagging is known as Folksonomies and is already heavily used on the public internet on sites like Facebook, MySpace, de.licio.us, Digg, Flickr, Jotspot etc. Now it is coming inside the enterprise and will be applied to BI and PM content as well as other unstructured content. This means that users can see other users’ profiles and the tags that they have used to annotate BI and PM content. From here it means that BI and PM ‘tag clouds’ will form showing popular BI and PM tems that lead to popular BI and PM content and metrics. Also by following BI and PM tags we will see the dynamic formation of BI social networks consisting of people within the enterprise that have similar interests in acting on BI to improve performance. People will also be able to share reports and collaborate with others (in real time – e.g. IM, threaded discussions etc.) in Web 2.0 collaborative workspaces. Wikis (group publishing) will also come together with BI so as to fuel rapidly forming BI and PM workspaces that will be of exceptional value to the business.
- Search and BI are set to explode into popular use in 2008 as search opens the doors to mass access to BI content from a userbase that is not comfortable with BI tools
- BI reports will be capable of being published in document management and records management systems
- Master data management market size will continue to grow as companies try to wrestle with the complexity of their data and get it under control. Information and data architects will continue to be in demand with demand for such professionals potentially outstripping supply
- Companies will have to invest again in data modelling and data modelling skills. There is no doubt that standards here are dropping, many companies still have no data modelling tools at all and also too few people are skilled in good data modelling practices.
- Data management professionals will start to come together into integration competency centres so that people with skills in data cleansing, data integration, data modelling, master data management, enterprise content management, metadata management and ESB XSLT XML data translation are all co-located and can work together to solve the problem of enterprise data management
- Metadata management will become a mission critical issue if it is not already. Business users need access to business metadata to understand what data means and where it came from. Holding this metadata in spreadsheets is no longer acceptable. It must be made available to both end users and shared across multiple technologies. 2008 will see companies looking to act to solve this problem.
Well that’s all I have for now. Let me know your thoughts. I would be most grateful for your comments on any of this. Best wishes for a happy and prosperous New Year!