As a member of the SGLake project team within the Development Section, Office for Information and Communications System (SGIS), the Subscriber shall contribute to the project goal of establishing a major new platform for the Department of Safeguards: the Data Lakehouse. In particular, the Subscriber shall provide expert guidance in new technology and platforms, lead task implementation, support project delivery and implementation, and promote a Data Lakehouse best practice approach.
Responsibilities:
Design and implement data products using the newly established Data Lakehouse architecture.
Advise and validate architectural components in close collaboration with the IT Solution Architect.
Design, build and maintain data pipelines and architectures, integrating and managing large volumes of data from various data sources into the established Data Lakehouse.
Ensure data quality, data integrity and data consistency in the project.
Collaborate with business users, data scientists, analysts, development teams and other stakeholders to design and provide relevant data services and products.
Optimize data flows, database queries and performance tuning.
Requirements:
University degree in Computer Science, Data Science or similar.
5 years of relevant professional experience working with complex data integration platforms supporting data analysis and data services.
In-depth experience in data engineering and related tools like Spark, Flink, Trino, Airflow, Iceberg, Jupyter, dbt core, meltano, Kafka, Debezium, S3, elastic.
