Data Warehouse Automation & Real-time Data – Reducing Time to Value in a Distributed Analytical Environment
Smart Infrastructure & Smart Applications for the Smart Business – Infrastructure & Application Performance Monitoring
Creating Data Products in a Data Mesh, Data Lake or Lakehouse for Use in Analytics (9-10 June 2022, Stockholm)
This blog entry is co-authored by Mike Ferguson and William McKnight (link) and is being cross-posted on our blogs.
Much discussion abounds about when not to use enterprise information integration (EII). This blog looks at some situations that are not particularly well-suited to the use of EII technology as a solution. Please note that these criteria should be taken only as guidelines.
Generally speaking EII is NOT suited for
· Complex transformations, fuzzy matching and integrating high volumes of data
· A replacement for data warehousing
· Business process management
· Frequent federated query processing with single federated queries integrating data across a large number of data sources. Performance may become an issue as more and more data sources are added to a data integration server. Several vendors do support caching in order to counteract this problem but nevertheless if you plan to integrate data from a wide range of data sources in a single query you would be well advised to benchmark products and compare results before making any purchase
· High volume transaction processing (insert, update and delete) is required to update virtual EII views of data in multiple underlying systems. This is because update processing via EII is still in its infancy and can be subject to product specific restrictions. Also concurrent access to EII servers may cause problems as workload management is missing from many tools. It is recommended that you investigate carefully how well EII vendors support transaction processing and if they support 2-phase commit distributed transaction processing.
· Transaction processing when integrity constraints across data sources are complex and could cause update processing to fail
· A complete solution to enterprise master data management. EII products can potentially provide a virtual view of integrated master data IF EII tools support global unique identifiers (and the mapping of disparate keys to the global one) and also the matching process to integrate data from multiple master data systems of entry does not require complex fuzzy matching. EII may be offered up as a component technology in an MDM solution but it will not provide everything needed for a full complete solution