Part 3: Creating a Reference Data Management Solution

Part 3: Creating a Reference Data Management Solution

In Part 1 of the Reference Data Management (RDM) series, I gave an overview of Reference Data and discussed some of the key features of a RDM System. Part 2 looked into the challenges organizations face while dealing with reference data. This is the third post in this series and here we will explore three key phases involved in creating a RDM solution.

Depending on the tool set you are using and the functionality you want to accomplish, RDM implementation can have varying complexity and time frame. However, from an overall planning and implementation point of view, we can categorize these projects under following 3 phases. Each phase below also lists the tasks involved for successful outcome.

Consolidation:
Authoring and creating a golden copy of reference data starts by consolidating all your reference data sets. Here are the 8 important steps that will put you on the fast track.

Create a list of all reference data sets which are used and required for smooth functioning of your business. This stage is often referred to as discovery. Keep an open mind during this step and create this list based on what is important for the enterprise as a whole rather than what individual applications require.
Categorize your data sets as internal and external data sets. I.e., what are the codes your organization manages internally versus what are industry standard codes.
Create a list of all applications and systems (internal) and providers (external) that create reference data.
Profile data sets to measure the quality. Given the apparent importance of data quality, this step is very crucial (and often discounted).
Analyze data and create rules around your data sets for
1. Required attributes, allowed length, data types, sub types, effective and expiry dates etc.
2. Cleansing and transforming data to maintain quality (No UNKNOWN, DUMMY, -1, -911 codes)
Find efficient ways to extract data so it can be loaded to a reference data hub whilst making sure the business rules are honored.
Design your extract process for easier integration of new reference data sources from both internal and external sources of reference data.
Design your solution to add and update mapping and hierarchical representation of data sets.

Mastering:
Taking reference data from sources and loading it into a reference data hub is half the battle. The bigger challenge is making sure that the golden version of reference data is managed properly on an ongoing basis.

Mastering your reference data includes specialized actions performed by Data Stewards, Business analysts and Subject Matter Experts. These users of RDM system bring in extensive knowledge of business processes of the organization and help in mapping those processes to individual applications. In a nut shell, following are the key tasks involved in mastering your reference data.

Ensure reference data is clean, consistent, accurate and sharable.
Build rules to identify duplicated codes and reference data sets.
Detect the overlapping codes across different source systems and create a single version of reference data sets.
Manage the mapping and relationships between reference data sets. Often referred as transcoding, this step helps you map a single representation of codes (Mostly canonical) into multiple application specific representations.
Maintain hierarchical representation of reference data sets.
Continuously audit reference data sets to capture who (system/user) changed the data, what was changed and when.
Build security around your reference data with attribute level permissions for view/add/update/delete operations.

Distribution:
You create a “golden copy” of any master data so that all downstream systems can leverage it. Reference data is not an exception and needs to be seamlessly integrated with downstream systems. Tasks listed below will help achieve effective distribution of your reference data.

Create an extensible service layer to expose reference data in the form of web services.
Create an infrastructure which can deliver reference data in sub second response time to requesting applications.
Create a flexible mechanism to export and transform reference data that can be consumed by downstream applications.
Support incompatible consuming applications (If any) by providing ad-hoc dump of reference data.
Allow support for exporting delta changes (Incremental or changes since last export).

Hope you liked this post. Please share your opinions via comments.

COMMENTS

3 Thoughts on Part 3: Creating a Reference Data Management Solution

Andrew Cohen

24 Jul 2013

7:40am

Interesting article, its nice to see someone writing about RDM from the experience of customer engagements.

Do customers choose IBM’s RDM solution as a way of furthering their MDM goals (mastering and control of their data) in a top-down way, or are they driven more by application integration concerns (more of a bottom-up type of motivation) ?

regards,
Andrew

Aditya

19 Jan 2017

7:11pm

Prash, best tools you have used in the RDM space? Will be very helpful, as I’m exploring options to deploy a RDM solution in the cloud (AWS) for SMBs. Thanks.

Show Replies

MDM – A GEEK'S POINT OF VIEW