Job Summary:

We are seeking an experienced Data Engineer Consultant to join our team as a senior individual contributor responsible for designing, implementing, and optimising robust data infrastructure and pipelines. This role offers the opportunity to work hands-on with cutting-edge AWS and Databricks technologies whilst providing technical guidance to team members and collaborating with stakeholders across the organisation on complex data engineering challenges.

Key Responsibilities:

• Data Architecture & Engineering - Design and implement enterprise-scale data architectures , including data lakes, warehouses, and real-time streaming platforms . Develop and maintain ETL/ELT pipelines that efficiently process large volumes of structured and unstructured data from diverse sources. Ensure data quality, governance, and security standards are embedded throughout all data engineering processes.
• Technical Implementation - Hands-on development of Databricks notebooks using PySpark, Python, and SQL for ETL automation . Create and optimise PySpark scripts for efficient data extraction, transformation, and loading from large datasets. Implement custom data manipulation, validation, and error handling solutions to enhance ETL robustness.
• Technical Guidance - Provide technical mentorship to junior data engineers and analysts. Lead code reviews, establish best practices, and drive adoption of modern data engineering tools and methodologies. Collaborate with cross-functional teams including data scientists, analysts, and software engineers to deliver integrated solutions.
• Performance & Optimisation - Monitor and optimise data pipeline performance, implementing solutions for scalability and cost-effectiveness . Conduct testing, debugging, and troubleshooting of data transformation processes. Verify data integrity throughout pipeline stages and resolve complex technical issues.

Essential Skills:

Technical Skills:

• AWS cloud platform expertise (Redshift, S3, Glue, EMR, Kinesis, Lambda)
• Databricks platform proficiency with PySpark, Python, and SQL implementation
• Data warehouse design and implementation (Amazon Redshift, Snowflake on AWS)
• ETL/ELT migration project experience and pipeline development
• Real-time streaming technologies (Kinesis Data Streams, Kafka)
• Data exchange platforms and API integration (AWS Data Exchange, Partner APIs)

Communication Skills:

• Clear written and verbal communication of technical concepts to non-technical stakeholders
• Presentation and reporting abilities for executive audiences
• Active listening to understand business requirements
• Stakeholder management across technical and business teams
• Meeting facilitation for technical design sessions

Leadership Skills:

• Technical mentorship and guidance for junior engineers
• Conflict resolution in cross-functional project environments
• Decision-making under pressure for critical data system issues
• Knowledge transfer and coaching abilities
• Change management for data platform migrations

Organisational Skills:

• Time management across multiple concurrent data projects
• Resource allocation for optimal pipeline performance
• Multi-tasking coordination of data engineering deliverables
• Attention to detail in data quality and governance processes
• Documentation of technical specifications and system architecture

Problem-Solving Skills:

• Critical thinking for complex AWS data architecture challenges
• Creative solution development for scalability and performance issues
• Issue escalation and resolution for production data systems
• Adaptability to evolving business and technical requirements
• Strategic planning for AWS data infrastructure roadmaps

Interpersonal Skills:

• Relationship building with business stakeholders and technical teams
• Negotiation of technical requirements and project timelines
• Emotional intelligence in managing diverse project teams
• Cultural awareness in global data initiatives
• Customer service orientation for internal data consumers

Qualifications:

Education Requirements:

• Bachelor's Degree in Computer Science, Engineering, Data Science, or related field
• Master's Degree in Data Engineering, Computer Science, or MBA (preferred)
• Relevant diploma or certificate in AWS cloud technologies or data engineering

Professional Certifications:

• AWS Certified Data Engineer - Associate (preferred)
• AWS Certified Solutions Architect or AWS Certified Big Data - Specialty (preferred)
• Databricks Certified Data Engineer Associate or Professional
• Amazon Redshift or Snowflake certifications
• Agile/Scrum Master certification (preferred)

Experience Requirements:

• 4-7 years of data engineering experience with AWS-native implementations
• Proven experience working as part of a data engineering team on ETL migration projects
• Hands-on experience implementing Databricks notebooks using PySpark, Python, and SQL for ETL automation
• Demonstrated expertise developing PySpark scripts for efficient data extraction, transformation, and loading from large datasets
• Experience utilising Python for custom data manipulation, validation, and error handling in ETL processes
• Proficiency employing SQL for complex joins, aggregations, and database operations within Databricks environments
• Track record of testing, debugging, and optimising data transformation processes for accuracy and performance
• Experience verifying data integrity throughout pipeline stages and resolving troubleshooting issues
• Proven ability collaborating with cross-functional teams to align ETL migration tasks with project goals and deadlines
• Experience in government sector data projects and compliance requirements (preferred)
• Experience in HR domain including workforce analytics, payroll systems, and employee data management (preferred)
• Experience providing technical mentorship to junior team members (preferred)

Working Location : Central

**We regret to inform that only shortlisted candidates will be notified. Personal data collected will be used for recruitment purposes**