Data Quality Engineer
Indexed description
Overview
The EDM Data Quality Engineer is responsible for making sure the data produced by the team is reliable and accurate. This includes reviewing data engineering pipelines and analytic solutions prior to production release to our end consumers.
Responsibilities
- Identifying data quality issues, sub-optimal data processes, and recommending corrective actions.
- Gather data quality requirements from stakeholders.
- Evaluate new data sources, data pipelines, ETL processes, and data transformations for quality and attribution to support product requirements.
- Validation of ingestion process (schema validation, completeness, duplicate handling).
- End-to-end integration testing between multiple independent systems and interfaces.
- Works with data engineers, analysts, and stakeholders to develop manual and automated test cases, and present formal test results for evaluation.
- Perform regression testing when pipelines, schema, or logic change.
- Create and maintain documentation of test plans, test cases, and testing results.
- Oversee Quality Engineering team, including recruiting/training/onboarding, as well as managing work assignments and goal achievement.
- Monitor and validate data SLAs and freshness requirements.
- Ensure compliance with data governance and security standards, as well as support audit and compliance activities.
- Support data integrity monitoring, including assessing data cost and evaluating data transformation job performance.
- The work activity includes reviewing complex data sets, leveraging technologies used to process these disparate data sets and understanding the correlations as well as patterns that exist between these different complex data sets.
- Provides concise status updates and challenges to senior management.
- Validate BI dashboards and reports for:
- KPI accuracy and aggregation logic
- Filter, slicer, and drill-down behavior
- Data refresh schedules and incremental loads
- Consistency between backend data and front-end visualizations
- Perform cross-report validation across multiple reporting tools (Tableau, PowerBI, SSRS/SSAS)
- Validate calculated fields, measures (e.g., revenue, growth %, net difference), and derived metrics.
- Perform validations to ensure alignment between business requirements and dashboard outputs.
- Perform -
- Functional testing of data workflows
- Regression testing
- System Integration Testing (SIT)
- User Acceptance Testing (UAT) support
- Performance and volume testing of data pipelines
- Data migration validation
- Impact analysis for upstream/downstream changes
- Conduct root cause analysis on defects and collaborate with engineering teams for resolution.
- Validate data lineage and trace data flow across multiple systems.
- Perform negative testing and edge case scenario validation.
- Present formal test results.
- Support testing activities with structured evidence documentation.
- Design, develop, and maintain automated data validation scripts using SQL, Python, or relevant automation frameworks.
- Automate regression testing for ETL workflows and incremental loads.
- Perform schema validation, null checks, duplicate detection, referential integrity checks, and transformation logic validation through automation.
- SQL (advanced queries, joins, window functions)
- Python (Pandas, PyTest or similar frameworks)
- ETL Tools (Informatica, ADF)
- Cloud platforms (AWS, Azure, Data Bricks)
- BI Tools (Power BI, Tableau)
- CI/CD tools (Azure DevOps)
- Agile tools (JIRA)
- BA/BS in Computer Science, engineering or related field, or equivalent experience.
- Hands-on experience in AWS and Databricks.
- 3+ years of experience in testing data platforms.
- 3+ years of development experience with Oracle and/or SQL Server.
- 3+ years of Experience with Python/Pyspark frameworks/libraries.
- Experienced with data normalization and denormalization techniques.
- Experienced in testing largescale event based streaming architectures.
- Experienced in Delta Lake concepts (ACID, Time Travel, Schema Enforcement)
- Experienced in testing data transformation.
- Experienced in Agile methodology.
- Experienced working in Databricks
- Preferred working knowledge of cloud architectures on AWS .
- Strong communication skills, with the ability to present complex technical solutions in a manner that can be interpreted by a general audience. This includes proficiency in Powerpoint, Visio, Drawio, etc.
- Experienced in working with cross functional teams, building alignment and collaboration.
- Preferred certification Databricks Certified Data Engineer Associate/Professional.
- Experience in Spotify delivery methodology is preferred.
- BA/BS in Computer Science or related field, or equivalent experience.
- 3+ years of experience in area of data management and/or data curation
- 3+ years development experience with DataBricks in AWS cloud space.
- Annual Bonus Plan
- Private Medical Care
- Group Life Insurance
- MyBenefit Cafeteria platform
- Lunch Vouchers
- Holiday Allowance
- Paid Lunch Break (30 minutes)
- Tax-deductible costs
- Hybrid work (flexibility to work 2 days/week from home)
Create a free Caio profile to unlock more results and save your role and location preferences.
Unlock free search