Senior Data Engineer (NHS)
Our Future Health, United Kingdom
Job Description
We are looking for a Senior Data Engineer to join our growing data engineering team at Our Future Health. The Senior Data Engineer will bring an in-depth knowledge of NHS data and data solutions to help solve some of the key challenges around a programme of work at industrial scale with global significance.
This is a new role to join our growing data engineering team at Our Future Health. It’s a role with an inspiring set of data challenges, for what will eventually be a comprehensive view of the health of 5 million people in the UK, including directly gathered information, genetics, NHS records and other linked data.
Our plan is to bring together 5 million volunteers from right across the UK who will be asked to contribute information to help build one of the most detailed pictures we have ever had of people’s health. Researchers will be able to use this information to make new discoveries about human health and diseases. So future generations can live in good health for longer.
What you’ll be doing:
Your key responsibilities will include but not be limited to:
- Lead the build of data pipelines from data providers to our primary data store and trusted research environment.
- Produce logic for data transformation steps as code, which meets the requirements for our end users and builds well curated, accessible and quality controlled data for analysis.
- Keep abreast of best practice in data engineering across industry, research and Government and facilitating the adoption of standards.
- Providing technical input into the upstream parts of the data pipeline, including the specification and transfer of data from data providers.
- You will confidentially communicate to and between technical and non-technical stakeholders as well as facilitate discussions within a multidisciplinary team including scientists, software engineers, product managers and other data engineers.
Requirements
To succeed in this role, you will also have some of the following skills:
- Experience working in an agile development team.
- Experience leading data engineering projects and developing people and teams from a technical perspective.
- Can listen to the needs of technical and business stakeholders and interpret them, and effectively manage stakeholder expectations. Can support or host difficult discussions within the team or with diverse senior stakeholders.
- You have the detailed knowledge and understanding of NHS data such as hospital administrative data, disease registries or primary care data, and how they can be used to support research.
- You have experience with the design and deployment of complex data schemas.
- You have experience in validating and cleaning complex datasets.
- You can design, build and test pipelines based on feeds from multiple systems using a range of different technologies. You know how to create repeatable and reusable products.
- Understanding and working knowledge of information governance and data security approaches appropriate for sensitive health data.
Technical requirements:
- Experience with version control and Git/GitHub.
- Good understanding of cloud environments (ideally Azure), and experience building and maintaining pipelines with tools such as Dagster, dbt, Airflow, Prefect.
- Understanding of containerisation (e.g. Docker) and deployment (e.g. Kubernetes).
- Understanding of common data transformation and storage formats, e.g. Apache Parquet.
- Experience with Spark, Databricks, data lakes.
- You have the knowledge of health record coding systems, data standards (e.g., ICD, READ and SNOMED codes), data models (e.g. OMOP), data exchange formats (e.g. FHIR).
- Highly proficient in Python.
- You follow best practices like code review, clean code and unit tests.
Benefits
- Up to 75,000 per annum basic salary.
- Generous company pension package with employer contributions of up to 12%.
- 30 days annual leave (plus bank holidays.)
- Individual development budget
- Flexible and remote working arrangements and a lovely new office in Holborn, Central London.
Join us - let’s prevent disease together.