Senior Data Engineer _ Python with Spark

Must Have Technical/Functional Skills

• Strong hands-on experience in Python programming for data engineering and data processing.
• Extensive experience with Apache Spark (PySpark) for large-scale data processing and distributed computing.
• Strong knowledge of SQL and experience working with relational and non-relational databases.
• Experience in building and maintaining ETL/ELT pipelines for data ingestion and transformation.
• Good understanding of data warehousing concepts, data modeling, and data architecture.
• Experience working with big data technologies such as Hadoop ecosystem, Hive, or similar platforms.
• Familiarity with cloud platforms (AWS, Azure, or GCP) and related data services.
• Hands-on experience with data pipeline orchestration tools such as Airflow or similar.
• Knowledge of data formats such as Parquet, Avro, JSON, and CSV.
• Experience with performance tuning and optimization of Spark jobs and data pipelines.
• Strong problem-solving skills and ability to work with cross-functional teams.

• Design, develop, and maintain scalable data pipelines using Python and Spark.
• Build and optimize ETL/ELT workflows for processing large volumes of structured and unstructured data.
• Work closely with data analysts, data scientists, and business stakeholders to understand data requirements.
• Develop efficient and reusable data processing frameworks and components.
• Perform data validation, cleansing, and transformation to ensure quality and consistency.
• Optimize and tune Spark jobs and SQL queries for performance and scalability.
• Collaborate with DevOps and platform teams to deploy and manage data solutions in cloud environments.
• Ensure data security, governance, and compliance standards are met.
• Troubleshoot production issues, perform root cause analysis, and implement fixes.
• Participate in design discussions, contribute to data architecture decisions, and promote best practices.
• Work in an Agile environment, supporting sprint activities and continuous improvement initiatives.

Discretionary Annual Incentive.

Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.

Family Support: Maternal & Parental Leaves.

Insurance Options: Auto & Home Insurance, Identity Theft Protection.

Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.

Time Off: Vacation, Time Off, Sick Leave & Holidays.

Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Salary Range: $100,000 - $115,000 a year

Qualifications: BACHELOR OF COMPUTER SCIENCE