Data Engineer

Insight Global is looking to hire a Sr. Data Engineer for a retail client based in Vancouver. This role is a hybrid position and requires 3 days onsite per week in Downtown Vancouver. You will be joining a team that works heavily with customer data—owning the customer data pipelines that gather data coming from multiple sources and consolidating that data for different use cases. Some use cases include segmentation, data and analytics, advanced analytics for ML models to generate product recommendations, etc.

You will have a strong knowledge of the Azure infrastructure and can build Azure pipelines. You will make general recommendations to the client on how to improve the current infrastructure. This role is not tied to a certain body of work; rather, you will support multiple workstreams and work with the Product teams to prioritize. Other duties and responsibilities include:

Data pipeline creation and maintenance

Stack: GCP, Azure Cloud, Azure Databricks, Snowflake

Includes engineering documentation, knowledge transfer to other engineers, future enhancements, and maintenance

Create secure data views and publish them to the Enterprise Data Exchange via Snowflake for other teams to consume

Data pipeline modernization and migration via Databricks Delta Live Tables (DLT) and Unity Catalog

Leverage existing CI/CD processes for pipeline deployment

Adhere to PII encryption and masking standards

Orchestration tools: ADF, Airflow, Fivetran

Languages: SQL, Python

Data Modeling: Star and Snowflake schemas

Streaming: Kafka, Event Hubs, Spark, Snowflake Streaming

DevOps Support:

Support improvements for current CI/CD processes

Production monitoring and failure support

Must-haves:

5+ years of experience as a Data Engineer

Experience with Azure Data Factory, Azure Databricks, Snowflake, and Storage Accounts

Experience with Python, specifically PySpark for developing and optimizing pipeline builds

Experience with DBT/DLT and working with Medallion Architecture

Experience with CI/CD principles and best practices, including Azure DevOps and Repos

Experience with Azure Cloud deployments and configurations

Experience with bug tracking and task management software such as JIRA

Experience managing outages, customer escalations, crisis management, and similar circumstances