Data Engineer (Databricks, Python, Pyspark and SQL)

SSI
3.7 out of 5 stars
Mississauga, ON
$60–$70 an hour - Contract

Job details

Pay

  • $60–$70 an hour

Job type

  • Contract

Location

Mississauga, ON

Full job description

Role: Data Engineer (Databricks, Python, Pyspark and SQL)

Key Responsibilities:

  • Design, develop, and maintain scalable and efficient data pipelines using Databricks, Python, Pyspark, and SQL.
  • Write optimized and complex SQL queries to extract, transform, and load data.
  • Develop and implement data models, schemas, and architecture that support banking domain requirements.
  • Collaborate with data analysts, data scientists, and business stakeholders to gather data requirements.
  • Automate data workflows and ensure data quality, accuracy, and integrity.
  • Manage and coordinate release processes for data pipelines and analytics solutions.
  • Monitor, troubleshoot, and optimize the performance of data systems.
  • Ensure compliance with data governance, security, and privacy standards within the banking domain.
  • Maintain documentation of data architecture, pipelines, and processes.
  • Stay updated with the latest industry trends and incorporate best practices.

Required Skills and Experience:

  • Proven experience as a Data Engineer or in a similar role with a focus on Databricks, Python, Pyspark, and SQL.
  • Strong understanding of data warehousing concepts and cloud data platforms, especially Snowflake.
  • Hands-on experience with release management, deployment, and version control practices.
  • Solid understanding of banking and financial services industry data and compliance requirements.
  • Proficiency in Python scripting and Pyspark for data processing and automation.
  • Experience with ETL/ELT processes and tools.
  • Knowledge of data governance, security, and privacy standards.
  • Excellent problem-solving and analytical skills.
  • Strong communication and collaboration abilities.

Preferred Qualifications:

  • Good Knowledge in Azure and Databricks in highly preferred.
  • Knowledge of Apache Kafka or other streaming technologies.
  • Familiarity with DevOps practices and CI/CD pipelines.
  • Prior experience working in the banking or financial services industry.

Pay: $60.00-$70.00 per hour

Experience:

  • Data Engineer: 8 years (preferred)
  • Python scripting and Pyspark : 5 years (preferred)
  • Banking or financial services industry: 4 years (preferred)

Work Location: In person