A dynamic technology company is looking for an experienced Senior Data Engineer (Spark & Python Specialist) with deep expertise in building, optimizing, and maintaining high-performance data processing engines and modern data lakehouse architectures. The ideal candidate will contribute to technical excellence, refactor legacy ETL logic, and support scalable, cloud-agnostic data solutions.

Responsibilities:

• Apply Spark best practices for memory management, shuffle tuning, and partitioning to optimize data pipelines.

• Develop and maintain modular Python/PySpark data pipelines with Delta Lake/Parquet in a cloud-agnostic environment.

• Refactor complex SQL-based ETL into maintainable Python libraries.

• Build and optimize Medallion Architecture layers (Bronze/Silver/Gold) in the data lakehouse.

• Support code-first orchestration using tools like Airflow or Dagster, reducing dependency on GUI-based orchestration.

• Participate in code reviews, mentor junior engineers, and contribute to automated testing frameworks (Pytest).

• Collaborate with data scientists, analysts, and business stakeholders to translate requirements into actionable data solutions.

• Lead initiatives to modernize data workloads, introduce new technologies, and drive best practices.

• Ensure compliance with data security, governance, and quality standards.

Requirements:

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field.

• 6+ years of experience with Spark/PySpark, with the ability to optimize complex DAGs and diagnose performance issues.

• Advanced Python skills, including building reusable libraries and implementing automated testing.

• Strong SQL (T-SQL) skills for interpreting and migrating legacy logic.

• Hands-on experience with data lakehouse technologies, including Delta Lake and Parquet.

• Experience with Azure Synapse Analytics, Dedicated SQL Pools, and Data Factory for complex pipelines.

• Familiarity with containerization (Docker) and open-source standards for cloud-agnostic workloads.

• Proven collaborative experience within technical teams and mentoring junior engineers.

Benefits

• Salary: negotiable.

Contact RPO Recruitment for your next career opportunity.

• Our client is offering a highly competitive salary for this role based on experience.
• Apply for this role today, contact Jarad Pentz at RPO Recruitment or on LinkedIn.
• You can also visit the RPO Recruitment website: rporecruitment.us or email us your
  CV: [email protected].

We will contact you telephonically in 3 days should you be suitable for this vacancy. If you are not suitable, we will put your CV on file and contact you regarding any future vacancies that arise.