Data Engineer

JOB DESCRIPTION

To plan, design and implement scalable and robust data models and physical data models and ensure effective movement, collection, integration, storage and provisioning of data to meet business objectives through sound understanding of business unit requirements and frequent collaboration with relevant stakeholders (such as Enterprise Architecture, Database and Solution Architects, Business Intelligence Developers, Data Scientists and Product Owners, etc.)

• Provide input into the budget and manage and report on budget usage that reflects delivery of planned work within agreed parameters
• Control expenditure and identify process improvements to contain and reduce costs
• Build working relationships across teams and functional lines to enhance work delivery, collaboration and innovation
• Deliver customer experience excellence in own service delivery aligned to Organisational values and service standards
• Conduct hands-on work related to profiling, documenting and validating the clients
• Execute remediation actions as agreed with client Install or update required system components
• Implement the necessary components and frameworks required for automated deployments and task scheduling
• Build unit and systems tests to ensure successful delivery of components into production
• Manage the user acceptance testing UAT and associated signoff through change control
• Ensure compliance with relevant statutory, legislative, policy and governance requirements in area of accountability
• Ensure implementation of relevant policies, governance and practice standards across the business
• Maintain knowledge on relevant legislative amendments, industry best practices and internal compliance procedures and requirements
• Ensure compliance is adopted in terms of systems and procedures as laid out by organisation and review in response to audit findings and changes
• Develops an understanding of risks and risk management approaches
• Implement and provide input into the development of governance and compliance procedures and processes within area of specialisation and identify risks
• Educates others and makes suggestions for improvements
• Networks and participates in specialist risk forums where required
• Test relevant data coming from database to ensure it is not corrupted to ensure use of best practices and technologies
• Perform data quality assessments and introduce monitors and alerts to ensure data quality is maintained throughout all managed systems
• Confer with end users, clients, or senior management to understand and translate business requirements for complex development or into technical requirements enhancements
• Assist with gap analysis and business cases including cost and effort analysis
• Work with team and stakeholders to continually assess and redefine data technology stack to support changing data patterns and business use cases and to bridge the gaps between Data Engineering, Business and Product by constantly collaborating with all parties to understand data needs
• Take ownership of data delivery end-to-end for specific business use cases, as well as set and manage SLAs (Service Level Agreements)
• Tackle complex, intricate production-scale problems with tenacity, following them wherever they may go throughout the technology stack
• Work closely with Data Head/Manager and the Data Architect to meet business area relevant objectives and ensure delivery of effective solutions
• Build and/ or propose infrastructure to automate extremely high volumes of data delivery and creatively solve data volume and scaling challenges within area of accountability
• Contribute to the design and architecture of innovative solutions to difficult problems
• Create change scripts and set up scripts to be ready and implemented in production
• Contribute and add value to architecture forums and solutions design
• Utilise various components and tools which make up the data platform for enhanced service delivery
• Create requirements and design of the technical architecture, Design and development, testing, and deployment of the proposed solution
• Present technology solutions to senior leadership and influence architectures that will lead the transformation of our data analytics platform
• Conduct research on emerging technologies and provide expertise on technical environment or tools
• Recommend and implement technologies that improve cost effectiveness and systems flexibility for own area
• Provide Analytic infrastructure or big data technologies related support to Data operations and Analytic teams
• Grow online technical knowledge platform, identify best practices and develop guidelines for optimum usage of tools
• Provide required support for project roll out and support team members
• Design, implement, quality assure and deploy ETL (Extract, Transform and Load) transformations to enhance the Data Eco System and workflows that load, analyse and archive customer transactional data
• Train relevant employees on accessing Data Eco System and how to make the best use of its contents
• Design, build, and evolve custom ETL processes for feeding consumer systems
• Acquire and collect data via ETL processes from source systems into the Reservoir and Enterprise Data warehouse, adhering to the appropriate data platform standards
• Integrate data from multiple sources through the enterprise data platform architecture to meet the business objective
• Develop and maintain the physical data marts and databases
• Automate tasks related to data pipelines for the deployment of operational analytics
• Prepare and provision data for advanced analytical modelling by data scientists (as and when applicable)
• Perform data quality assessments and introduce monitors and alerts to ensure data quality is maintained throughout all managed systems
• Participate in planned activities that are appropriate for own development
• Develop, encourage and nurture collaborative relationships across area of specialisation