Data Engineer
HORIZON COMPUTER MANAGEMENT PTE. LTD.
3 days ago
Posted date3 days ago
N/A
Minimum levelN/A
Key Responsibilities
Data Engineering & Architecture
Big Data & Distributed Systems
Programming & Automation
Cloud Platforms
Data Governance & Security
Preferred Skills
Leadership & Collaboration
Qualifications
- Design, build, and maintain robust, scalable, and secure data pipelines for batch and real-time data processing.
- Develop and optimize ETL/ELT workflows to extract, transform, and load data from multiple sources.
- Architect and implement data warehouses, data lakes, and lakehouse solutions on cloud or on-prem platforms.
- Ensure data quality, lineage, governance, and versioning using metadata management tools.
- Collaborate with Data Scientists, Analysts, and Software Engineers to deliver reliable and accessible data solutions.
- Optimize SQL queries, data models, and storage layers for performance and cost efficiency.
- Develop and maintain automation scripts for data ingestion, transformation, and orchestration.
- Integrate and process large-scale data from APIs, flat files, streaming services, and legacy systems.
- Implement data security, access control, and compliance standards (GDPR, ISO 27001).
- Monitor and troubleshoot data pipeline failures, latency, and performance bottlenecks.
Data Engineering & Architecture
- Strong expertise in data modeling (dimensional/star/snowflake schemas) and data normalization techniques.
- Proficient in ETL/ELT tools such as Apache NiFi, Talend, Informatica, SSIS, or Airbyte.
- Advanced knowledge of SQL and distributed computing concepts.
- Experience with data lake and warehouse technologies such as Snowflake, Redshift, BigQuery, Azure Synapse, or Databricks.
- Deep understanding of data partitioning, indexing, and query optimization.
Big Data & Distributed Systems
- Hands-on experience with Hadoop ecosystem (HDFS, Hive, HBase, Oozie, Sqoop).
- Proficiency in Apache Spark / PySpark for distributed data processing.
- Exposure to streaming frameworks like Kafka, Flink, or Kinesis.
- Familiarity with NoSQL databases such as MongoDB, Cassandra, or Elasticsearch.
- Knowledge of data versioning and catalog systems (e.g., Delta Lake, Apache Hudi, Iceberg, or AWS Glue Data Catalog).
Programming & Automation
- Strong programming skills in Python, Scala, or Java for data manipulation and ETL automation.
- Experience with API integration, REST/GraphQL, and data serialization formats (JSON, Parquet, Avro, ORC).
- Proficient in shell scripting, automation, and orchestration tools (Apache Airflow, Prefect, or Luigi).
Cloud Platforms
- Expertise in at least one cloud ecosystem:
AWS: S3, Redshift, Glue, EMR, Lambda, Athena, Kinesis
Azure: Data Factory, Synapse, Blob Storage, Databricks
GCP: BigQuery, Dataflow, Pub/Sub, Cloud Composer - Strong understanding of IAM, VPC, encryption, and data access policies within cloud environments.
Data Governance & Security
- Implement and enforce data quality frameworks (DQ checks, profiling, validation rules).
- Knowledge of metadata management, lineage tracking, and master data management (MDM).
- Familiarity with role-based access control (RBAC) and data encryption mechanisms.
Preferred Skills
- Experience with machine learning data pipelines (ML Ops) or feature store management.
- Knowledge of containerization and orchestration tools (Docker, Kubernetes).
- Familiarity with CI/CD pipelines for data deployment.
- Exposure to business intelligence (BI) tools like Power BI, Tableau, or Looker for data delivery.
- Understanding of data mesh or domain-driven data architecture principles.
Leadership & Collaboration
- Work closely with cross-functional teams to define data requirements and best practices.
- Mentor junior engineers and enforce coding and documentation standards.
- Provide technical input in data strategy, architecture reviews, and technology evaluations.
- Collaborate with security and compliance teams to ensure data integrity and protection.
Qualifications
- Bachelor's or Master's Degree in Computer Science, Data Engineering, Information Systems, or related field.
- 5-10 years of professional experience as a Data Engineer or similar role.
- Professional Certifications preferred:
AWS Certified Data Analytics / Big Data Specialty
Microsoft Certified: Azure Data Engineer Associate
Google Professional Data Engineer
Databricks Certified Data Engineer
JOB SUMMARY
Data Engineer
HORIZON COMPUTER MANAGEMENT PTE. LTD.
Singapore
3 days ago
N/A
Contract / Freelance / Self-employed
Data Engineer