Data Engineer / BI Analyst

Job Title: Data Engineer / BI Analyst

Location: NJ (Hybrid – Morristown, NJ)

Type: Full-time

Salary: up to $105k annually plus benefits, 401k, PTO, etc (Based on Experience)

This position is open only to individuals authorized to work in the United States without current or future sponsorship

** Candidates should have some experience writing PySpark jobs on Azure Databricks to ingest and transform data across a medallion lakehouse **

We are seeking a skilled and motivated Data Engineer to join our team, with deep hands-on experience building and optimizing data pipelines and lakehouse solutions in Databricks. In this role, you will collaborate with cross-functional teams to design, develop, and operate scalable, reliable data products that drive business value.

Key Responsibilities:

• Design, build, and maintain batch and streaming data pipelines using Databricks (Spark, Delta Lake, Jobs/Workflows).
• Partner with data scientists, analysts, and application teams to deliver trusted, well-modeled data sets and features in the Databricks Lakehouse.
• Optimize Spark jobs (partitioning, caching, join strategies) and Databricks cluster configurations for performance, scalability, and cost.
• Implement data quality checks, observability, governance, and security controls (e.g., Unity Catalog, access policies) within Databricks.
• Troubleshoot and resolve pipeline failures, data issues, and production incidents; perform root-cause analysis and implement preventative improvements.

Qualifications:

• Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience).
• Proven experience delivering production data engineering solutions in Databricks, including Apache Spark and Delta Lake on the Microsoft Azure cloud platform.
• Strong proficiency in SQL and Python (or Scala), including building ETL/ELT pipelines and writing efficient Spark code.
• Experience with data modeling and lakehouse/warehouse patterns (e.g., medallion architecture) and distributed data processing concepts.
• Excellent problem-solving skills and attention to detail.
• Strong communication and teamwork abilities.

Preferred Skills:

• Hands-on experience with Databricks ecosystem capabilities such as Unity Catalog, Delta Live Tables, MLflow, and/or Databricks SQL.
• Experience implementing CI/CD and automated testing for Databricks notebooks/repos and deployment of Jobs/Workflows.
• Knowledge of data security, privacy, and compliance controls in cloud lakehouse environments (including role-based access and auditing).