Senior Data Engineer

Job Summary

Our client, a major entertainment and media company, is looking for a Senior Data Engineer to own their Core Data platform. This is a 12 month contract, based in Glendale, CA. This person will build and operate batch and streaming Spark pipelines on Databricks, manage platform governance including Unity Catalog and ACLs, and provide solution architecture across AWS, Kubernetes, and Airflow. Looking for someone with real production experience on Databricks and Spark, not just exposure to big data tools generally, plus the ability to explain Spark architecture to both technical and non-technical stakeholders.

This position is W-2 only. We do not work with third-party firms or C2C arrangements for this role.

Core Responsibilities

• Design, write, test, and deploy data pipelines using PySpark, Scala, SQL, and Python
• Meet with stakeholders to gather requirements and translate them into scalable data platform solutions
• Diagnose errors and audit platform activity using Databricks tooling (API, SDK, CLI)
• Explain Spark architecture and pipeline behavior to stakeholders to diagnose root causes and recommend solutions
• Manage platform governance, including Unity Catalog, ACLs, lineage, and data discovery/privacy tooling
• Build and maintain Kubernetes containers and containerized utilities for platform services
• Troubleshoot connectivity and integration errors across platform components
• Provision access, monitor platform health and cost, and evaluate stakeholder requests
• Maintain documentation of platform changes, standards, and pipeline configurations

Required Skills/Experience (Must-Haves)

• Bachelor's degree in Computer Science, Information Systems, or related field, or equivalent work experience
• 5+ years of data engineering experience developing and operating large-scale data pipelines
• Deep hands-on experience with Databricks and Apache Spark (batch and streaming)
• Pipeline development in PySpark and/or Scala
• Strong understanding of Spark architecture, executors, stages, partitioning, shuffle, and performance tuning
• Advanced SQL with performance tuning capabilities
• Hands-on production experience with Airflow (MWAA)
• Experience managing Databricks platform governance: ACLs, Unity Catalog, lineage
• Proficiency in Python plus at least one additional language (Scala, Kotlin, or SQL-driven pipeline tooling)
• AWS-primary experience: compute, storage, networking, IAM
• Docker and Kubernetes for containerized data platform services

Preferred Skills/Experience (Nice-to-Haves)

• Master's degree
• Snowflake experience
• Additional cloud-based data platform experience beyond AWS
• CI/CD and DevOps practices (Git-based workflows)

Key Competencies and Behaviors

• Comfortable translating stakeholder requirements into scalable data platform solutions
• Strong analytical and problem-solving skills, self-starting on ambiguous issues
• Works well across multidisciplinary teams including engineers, architects, and product managers
• Comfortable owning platform reliability, not just building pipelines and moving on
• Can manage multiple concurrent priorities across development, governance, and platform administration

Work Environment

• Location: Glendale, CA
• 2-3 days onsite

Compensation & Benefits

• Pay Range: The approximate pay range for this position is between $60.00 and $86.00 per hour. Please note that the pay range provided is a good faith estimate. Final compensation may vary based on factors including but not limited to background, knowledge, skills, and location. We comply with local wage minimums.
• Medical, Dental, & Vision Insurance Plans
• Employee-Owned Profit Sharing (ESOP)
• 401K offered

About KellyMitchell

At KellyMitchell, our culture is world class. We’re movers and shakers! We don’t mind a bit of friendly competition, and we reward hard work with unlimited potential for growth. This is an exciting opportunity to join a company known for innovative solutions and unsurpassed customer service. We're passionate about helping companies solve their biggest IT staffing & project solutions challenges. As an employee-owned, women-led organization serving Fortune 500 companies nationwide, we deliver expert service at a moment's notice.

Marketing Disclosure

By applying for this job, you agree to receive calls, AI-generated calls, text messages, or emails from KellyMitchell and its affiliates, and contracted partners. Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You can reply STOP to cancel and HELP for help. You can access our privacy policy at https://www.careers.kellymitchell.com/privacy-policy

[143075]