Introduction
In today’s data-driven business landscape, maintaining the highest possible data quality is paramount. Accurate, reliable, and timely information is the foundation for making informed decisions, gaining valuable insights, and driving competitive advantages.
Reltio, a widely recognized leader in the Master Data Management (MDM) space, offers robust data quality capabilities through its Reltio Connected Data Platform. The platform empowers organizations to manage, govern, and leverage their data assets effectively, ensuring improved data quality and consistent data across the enterprise.
However, as businesses evolve and the complexity of their data grows, the need for more flexible and adaptive data quality solutions becomes apparent. In response to this demand, we are excited to introduce an advanced solution for enhancing Reltio Data Quality Validations (DQV). This approach combines real-time processing with the option for adding custom rules, allowing clients to drastically improve their data quality while retaining the agility to adapt to ever-changing business requirements.
Limitations of the existing Reltio Data Quality validations
As powerful as the Reltio Connected Data Platform is, there are certain limitations in its current DQV capabilities that can hinder organizations from achieving their full data quality potential. In this section, we will examine these constraints and underscore the need for more flexible and customizable solutions.
A. Limitations in attribute validations based on other attribute values
One of the key challenges with Reltio DQV is the inability to perform and implement complex validation rules that require the consideration of multiple attributes simultaneously. For instance, businesses may want to enforce rules that dictate the validity of an attribute based on specific conditions related to other attributes, such as the relationship between an individual’s age and their eligibility for certain products or services.
B. Need for more flexibility and control
Reltio’s existing validation capabilities may not provide the level of flexibility and control for organizations to manage their data effectively as they often have unique requirements that extend beyond the out-of-the-box functionality provided by Reltio. Additionally, as businesses evolve, their validation needs may change, necessitating a solution that can adapt to new rules and requirements quickly and seamlessly.
Example Scenario
Consider a scenario where an e-commerce company maintains customer records that include attributes such as “Date of Birth”, “Age”, “Account Creation Date”, and “Loyalty Program Membership Date”. The validation of these attributes is essential to ensure accurate customer information for personalized marketing, customer service, and targeted promotions.
With the existing Reltio DQV, it is challenging to enforce a rule that verifies whether the “Age” attribute is consistent with the “Date of Birth” attribute. In addition, ensuring that the “Loyalty Program Membership Date” is not earlier than the “Account Creation Date” is not easily achievable using the standard validation capabilities.
These limitations can lead to inconsistencies and errors in the data, impacting the overall quality of customer records, and hindering the effectiveness of the company’s marketing and customer engagement strategies.
Advantages of the DQV solution
The cutting-edge solution we propose for enhancing Reltio Data Quality validations offers numerous advantages that can help organizations overcome the challenges mentioned in the previous section. By incorporating real-time processing and custom advanced rules, this solution provides greater flexibility, control, and accuracy in data validation. Let’s explore the key benefits of adopting this approach.
A. Execute API calls
The DQV solution allows organizations to execute API calls, enabling them to hit Reltio endpoints or even third-party services during the validation process. This feature provides additional flexibility and extensibility, allowing businesses to incorporate external data sources and services as part of their data validation efforts.
B. Perform advanced entity profile and crosswalk level checks
The solution enables the execution of diverse checks on attributes with both ov:true and ov:false values, allowing for more comprehensive validation across all attributes. This offers the ability to conduct validations at the crosswalk level, granting enhanced control over data quality.
C. Perform attribute checks based on dependency attributes
This approach addresses the issue of validating many attributes simultaneously by enabling attribute checks based on dependency attributes. This allows organizations to create validation rules that consider multiple attributes and their relationships. This feature is crucial for businesses that require complex validation logic based on various conditions and interdependencies between attributes.
D. Custom validation logic
The ability to add custom validation logic is a significant advantage of our solution. Organizations can create validation rules tailored to their specific business needs and requirements, ensuring a more accurate and relevant data quality assessment. Moreover, running validation rules only for the attributes that have changed for a specific event is supported, which further enhances efficiency and reduces processing time.
E. Filtering on event types and object types
The DQV solution allows users to filter validations based on event and object types, providing greater control over the validation process. This feature enables organizations to focus on specific areas of their data landscape and address the most critical data quality issues.
F. Real-time processing
Real-time processing is a key feature of the DQV solution, allowing organizations to validate their data as it changes, rather than relying on batch processing or periodic validations. This capability ensures faster detection and resolution of data quality issues, leading to more accurate and up-to-date data for decision-making and analysis.
Implementing the DQV Solution: A Step-by-Step Guide
Now that we have discussed the advantages of the advanced solution for enhancing Reltio Data Quality validations, let’s dive into the step-by-step guide on implementing this solution. While this guide focuses on using AWS, keep in mind that the same principles can be applied to the other cloud service providers supported by Reltio – Google Cloud Platform and Microsoft Azure.
Example Architecture Diagram
Step 1: Configure Reltio event streaming SQS queue
The first step in implementing the solution is to configure an Amazon Simple Queue Service (SQS) queue for Reltio event streaming. This queue will be responsible for processing and validating the data in real-time. Follow Reltio’s documentation on adding an external queue configuration: Add an external queue configuration | Reltio
Apply the following settings to the external streaming queue:
- Choose “Delta” as a payload type: This setting ensures that only the changed attributes (deltas) are sent to the queue for processing, reducing the amount of data that needs to be processed and improving performance.
- Set the desired event and object type filters: Configure the event and object type filters according to your organization’s requirements. This allows you to selectively focus on the relevant events and apply only the necessary validation rules, thereby minimizing unnecessary work.
Step 2: Create Lambda function
After setting up the SQS queue, create an AWS Lambda function to handle the data processing and validation logic. As a serverless computing service, Lambda runs your code in response to events, such as changes in messages within the SQS queue. Make sure that your Lambda function includes the custom validation logic and rules tailored to your organization’s data quality requirements.
AWS Lambda Policy Template with Minimal Permissions
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"sqs:DeleteMessage",
"sqs:ReceiveMessage",
"sqs:GetQueueUrl",
"sqs:GetQueueAttributes"
],
"Resource": "YOUR_SQS_ARN"
}
]
}Refer to the section “Advantages of the DQV solution” for insights on how to fully maximize the potential of the DQV solution.
Step 3: Add SQS trigger to the Lambda
The final step is to add an SQS trigger to the Lambda function, linking it to the Reltio event streaming SQS queue created in Step 1. This configuration ensures that the Lambda function is triggered whenever a new message is sent to the SQS queue. As a result, your custom validation logic will be applied in real-time, allowing you to quickly identify and address data quality issues.
Conclusion
In conclusion, the modernized solution for enhancing Reltio Data Quality validations offers a powerful and flexible approach to overcoming existing validation limitations. By combining real-time processing and custom advanced rules, organizations can improve their data validation efforts and ensure the accuracy and reliability of their data assets.
As data continues to play a crucial role in business strategy, investing in data quality solutions that can adapt to ever-changing requirements is essential. Explore the potential of custom advanced rules and real-time processing in your organization’s data quality efforts by implementing the proposed solution and unlocking the full potential of your data assets.
