Senior Data Engineer

Job Description

Overview:

About Presight:

Presight is an ADX-listed public company with Abu Dhabi based G42 as its majority shareholder and is the region’s leading big data analytics company powered by GenAI. It combines big data, analytics, and AI expertise to serve every sector, of every scale, to create business and positive societal impact. Presight excels at all-source data interpretation to support insight-driven decision-making that shapes policy and creates safer, healthier, happier, and more sustainable societies. Today, through its range of GenAI-driven products and solutions, Presight is bringing Applied AI to the private and public sector, enabling them to realize their AI strategy and ambitions faster.

The Opportunity

As a Senior Data Engineer, you will be responsible for building and maintaining the infrastructure that supports data collection, processing, and storage, working closely with data scientists, analysts, and other stakeholders to ensure that data systems are reliable, scalable, and secure. Your work will be crucial in enabling data-driven decision-making across the organization. This is a key technical role focused on developing and optimizing the company's data infrastructure which involves designing and implementing data pipelines, ensuring data quality, and collaborating with cross-functional teams to support various data initiatives.

Responsibilities:

As a Senior Data Engineer, you will be responsible for developing and maintaining data systems to support the company’s strategic goals. Your role will encompass a range of activities focused on data pipeline development, data quality, and cross-functional collaboration.

• Data Pipeline Architecture and Development

Design, construct, install, test, and maintain highly scalable data pipelines with a focus on machine learning models and analytics.

• Data Integration

Work closely with data scientists, ML engineers, and stakeholders to ensure that data is accessible, consistent, and reliable for ongoing projects.

• API and Data Services

Develop and maintain APIs for data access and manipulation, and integrate with external data services as needed.

• Data Storage

Manage and optimize data storage solutions for both structured and unstructured data, where structured data includes relational databases and unstructured data includes Text Image Audio and Video, Search Engines like Elasticsearch and NoSQL databases, to support the requirements of machine learning models.

Understand data engines and structure to effectively design solutions for transactional, analytics, and search purposes.

• Data Quality and Governance

Implement processes to monitor data quality and ensure production data is always accurate and available for key stakeholders.

• Collaboration and Support

Collaborate with ML engineers to assist in data-related technical issues and provide architectural guidance and solutions.

• Security and Compliance

Ensure compliance with data security and privacy policies.

• Documentation

Maintain clear and up-to-date documentation including data dictionaries, metadata, and architectural diagrams.

Qualifications:

Skills and attributes for success

• Bachelor’s degree in Computer Science, Engineering, Mathematics, or a related field; or equivalent work experience.
• 7+ years of experience in a Data Engineering role.
• Programming languages like Python and SQL and managing huge scale data potentially Terabyte to Petabyte.
• Hands-on experience with big data technologies like Spark (Using PySpark / Scala) and Flink.
• Familiarity with machine learning frameworks such as TensorFlow, PyTorch, or similar.
• Strong understanding of data warehousing or Lake-house concepts, ETL processes, and data modeling.
• Experience with API development and integration with data services.
• Experience with cloud platforms like Azure.
• Knowledge in DevOps, CI/CD methods, and containerization technologies like Docker or Kubernetes.
• Experience with real-time / streaming data processing.

Technical stack

• Programming Languages: Python, SQL
• Query Engine: Trino
• Big Data Technologies: Spark, Flink
• Unstructured Data: Text, Image, Audio & Video
• Databases: Clickhouse, MySQL, PostgreSQL, MongoDB, Cassandra, HBase, Redis
• Cloud Platforms: Azure
• API Development: RESTful APIs, GraphQL, OpenAPI
• Data Services: Kafka, RabbitMQ
• Containers: Docker, Kubernetes