Roles and Responsibilities

• Work closely with Product Owner for the development, implementation and maintenance of data engineering products for Business Intelligence (BI) and Data warehouse projects.
• Scope of work includes:

a) Design, develop and deploy data engineering solutions (including data tables, views, marts and etc) in data pipeline, data warehouse, operational data store, data lake and virtualisation.

b) Perform data extraction, cleaning, transformation, and flow. Web scraping may be also a part of the work scope in data extraction.

c) Design, build, launch and maintain efficient and reliable large-scale batch and real-time data pipelines with data processing frameworks.

d) Integrate and collate data silos in a manner which is both scalable and compliant.

e) Ensure adherence to quality processes throughout the data product lifecycle, including development, deployment, validation, change management and documentation, with particular emphasis on production environment standards.

f) Ensure proper data governance practices and maintain compliance with security policies and regulatory requirements.

g) Collaborate with Project Manager, Data Architect, Business Analysts, Frontend Developers, Designers and Data Analyst to build scalable data driven products.

h) Be responsible for developing backend APIs & working on databases to support the applications.

i) Work in an Agile Environment that practices Continuous Integration and Delivery.

j) Work closely with fellow developers through pair programming and code review process.

k) Collaborate with business stakeholders to gather and analyse requirements for data engineering solutions, ensuring alignment between technical implementation and business objectives.

l) Support business users in designing and developing front-end reports, dashboards and interactive visualisations to enhance organisational data analysis capabilities.

m) Provide ongoing operational maintenance and technical support for deployed data engineering products and systems.

n) Manage incident response and service requests, ensuring timely resolution within established Service Level Agreements.

o) Participate in regular project audits and provide training support to team members and stakeholders as required.

Knowledge and Skills

a) Proficient in general data cleaning and transformation (e.g. SQL, VQL, pandas, R, etc) to ensure data accuracy and consistency.

b) Proficient in building ETL pipeline (eg. SQL Server Integration Services(SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECS Container task, Eventbridge, AWS Glue, Spring).

c) Proficient in database design and various databases (e.g. SQL, PostgreSQL, AWS S3, Athena, mongodb, postgres/gis, mysql, sqlite, voltdb, cassandra, etc).

d) Experience in cloud technologies such as GPC, GCC (i.e. AWS, Azure, Google Cloud).

e) Experience and passion for data engineering in a big data environment using Cloud platforms such as GPC, GCC (i.e. AWS).

f) Experience with building production-grade data pipelines, ETL/ELT data integration.

g) Knowledge about system design, data structure and algorithms.

h) Familiar with data modelling, data access, and data storage infrastructure like Data Mart, Data Lake, Data Virtualisation and Data Warehouse for efficient storage and retrieval.

i) Familiar with rest api and web requests/protocols in general.

j) Familiar with big data frameworks and tools (eg. Hadoop, Spark, Kafka,RabbitMQ).

k) Familiar with W3C Document Object Model and customized web scraping(e.g. BeautifulSoup, CasperJS, PhantomJS, Selenium, Nodejs, etc).

l) Familiar with data governance policies, access control and security best practices.

m) Comfortable in at least one scripting language (eg. SQL,Python).

n) Comfortable in both windows and linux development environments.

o) Interest in being the bridge between engineering and analytics.

Good communication skills to work closely with stakeholders, technical lead and fellow team members