There is quite a bit of confusion among companies about managing data quality, let alone clarity about how a company can best tackle data quality optimisation. Jack de Hamer, co-owner of data management specialist Type2Solutions, defined five simple steps that offer practical help in optimising data quality. Using trivial examples, he also tries to clearly explain the rather abstract concept of data.
Those searching for a definition of data quality management may end up at techopedia.com. This site offers the following (translated) definition of data quality management: 'Data quality management is an administrative process. Within it, guidelines, responsibilities and processes have been established regarding data collection, use and maintenance. This process also includes agreements on data deletion and distribution. A company that wants to set up data quality management properly and make it succeed should put energy into the cooperation between the manager of the system processes (IT) and the manager of the business processes (the business).' Unfortunately, the above definition does not answer how to do that as a business right now.
Based on years of experience, De Hamer defined the following five simple steps that offer practical help in optimising data quality:
Step 1. Define the data set and the elements within it that you want to optimise
Divide and conquer!
This statement applies to warfare, but also to data. The first step of any data optimisation initiative is to divide the data elements into different collections (entities). Examples of entities are: countries, creditors, invoices, contracts, warehouses or articles. By the way, that dividing sounds simpler than it is!
Fortunately, many software vendors have already made this division for you in the past. So you can ride on the efforts of others by simply looking at how the data in your CRM, WMS or ERP system is divided and structured.
It is advisable to start with just one collection that belongs to your 'master data'. After all, you use this master data over and over again and in multiple places within your own systems and organisation. If you digitally deliver data to a web shop or customer, this is often master data as well.
Examples of master data are: items, creditors and debtors. In transactional data, master data is continuously used data. Think, for example, of your energy bill. Each bill (transactional data) contains the products (master data) you purchase as a customer (master data).
Only once you have completed the five steps in this document will you have control over the master data collection. Only once you have gained control over the first set should you select a second one, and only once you have gained control over the second should you select a third. Repeat this process until you have gained control over all the desired master data sets. Only then should you move on to the ‘transaction data’.
This approach ensures a sustained focus. In addition, you will achieve quickly measurable results and remain in control. As an added bonus, optimising your master data also has a positive effect on the quality of the related entities.
Step 2. Determine per data element the rules that apply and automate the checks
Once a collection has been chosen, you need to determine per data element which definitions apply. These may be several. The clearer the definitions are, the easier it is to translate these definitions into rules that can be executed automatically.
Definitions can be both technical and functional in nature. Examples of clear definitions are:
- The element has a mandatory value;
- The element contains unique values;
- The maximum length of the value is 2 characters;
- Value should only contain alphanumeric characters;
- Minimum value length is 6 characters;
- Value may only contain 'Yes' or 'No'.
Step 3. Appoint one person per data collection who is responsible for optimisation
This person, the data owner, is responsible for correcting exceptions (or having them corrected). This data owner can of course be responsible for multiple collections. Depending on the amount and type of corrections, you can choose a manual correction, an automatic correction or a hybrid form. The results of the corrections are evaluated on a regular basis, e.g. daily or weekly, by the business and possibly IT with the data owner.
Step 4. Automatically check data by definition and save the result
It is almost impossible to manually check the quality of large amounts of data on a frequent basis. For automated checking, you need software that can execute your defined rules on the data. The rules within this software search your data at lightning speed and detect and present all exceptions. In addition, the software stores the details of the exceptions and cumulative results in a central database. This allows you to identify, over a period of time of your choosing, trends in the improvement or deterioration of data quality.
Once it is clear to the data owner what exceptions exist, he/she can set out actions and start correcting these exceptions (or having them corrected) and optimising data quality.
Step 5. Correct the exceptions and adjust the scope and definition where and when necessary
Based on the exceptions, the data owner schedules or deactivates corrective actions. Also, if necessary, the data owner engages additional capacity. Additionally, this person can add new definitions or adjust or delete existing definitions. With this process, you continuously improve data quality and the existing data quality optimisation process.
The figure below graphically depicts this optimisation process
Explanation
From the application's data source, the data collection (step 1) is extracted. This data is automatically analysed by the T2S Data Quality Monitor (step 4) based on the rules that follow from the definitions (step 2). The exceptions to the rules are detected and presented to the data owner (step 3). The data owner collects the exceptions and, based on this, decides which of the improvement actions below should be taken.
- Optimising data quality by correcting exceptions;
- Optimisation or fine-tuning definitions.
Using this process continuously improves data quality and the existing data quality optimisation process (step 5).
Finally
In some situations, you will discover that you are behind the theoretical definition of an element but that existing business and system processes do not make it easy to correct the exceptions.
I remember a situation where it was stated as a definition that no business products should be linked to private contracts and vice versa. This in itself was a logical choice but unfortunately there were still some active private contracts with business products. To correct this, there were two options: remove the business product on the contract and link a new product with different terms or; terminate the private contract and list a new business contract. Both choices had significant customer impact. For this reason, it was then decided not to change the contracts but to tighten the definition. The contract start date was included in the definition and all contracts concluded before a certain point in time were not assessed against this rule.
For the homeowners among you: from 2013, such a definition will also be applied to redemption-free mortgages. If you took out a repayment-free mortgage before that date, then its quality was assessed as good by the Dutch government at least you were allowed to deduct the mortgage interest paid. If you took out an interest-free mortgage after that date, then suddenly the interest is no longer deductible and this type of mortgage seems to have become undesirable.
This example clearly shows that data quality and mortgages are highly dependent on market conditions and the corresponding definition. As you may define and change this definition for your data at any time, it is important to have flexibility in the definition and associated rules that test and quantify the data quality of your data.
Jack de Hamer - Co-founder and owner Type2Solutions
