Lead/Senior Software Engineer (Data Pipelines)
Our Future Health, United Kingdom
Job Description
We are hiring a Lead/Senior Software Engineer to design, build and operate the platforms that power Our Future Health. This is an exciting opportunity to join the UK’s largest ever research programme! In this role, you will join a small team that will enable us to delivery our mission of enabling cutting edge health research across a cohort of 5 million volunteers!
At Our Future Health, our mission is to transform the prevention,?detection?and treatment of conditions such as dementia, cancer, diabetes, heart disease and stroke. We’re looking for people to join us on our journey. If you’re looking for a new challenge where you can contribute to helping future generations live in good health for longer, then we’re keen to speak with you.
Requirements
What you’ll be doing
- As a Software Engineer for our?data platforms, you’ll be focussing on the tooling and pipelines that allow ingestion and processing of questionnaire, NHS, and genetic datasets, at the petabyte scale. This includes working on the secure primary data store and the de-identification and sharing of data into accredited Trusted Research Environments (TREs).
- In this role, you’ll work collaboratively with other engineers from data, platform and Technical Operations
- This role will challenge you by being a blend of architecture, software and platform engineering.
- We are looking for someone who is confident leading software development, troubleshooting, design and architecture skills as we create new systems.
- You will collaborate with colleagues to develop plans related to the architecture, technical design, data flows, scaling and functionality of the services.
- You’ll support the building and design of significant technology components within the projects.
- You will participate in testing, troubleshooting and live operations issues that are escalated from other teams.
What you won’t be doing
- Working in a siloed environment with no freedom to make decisions.
- Working in a place where you can’t see the impact your expertise makes.
Requirements
To succeed in this role you will be able to demonstrate a large portion of the following skills and experience:
- You’ll have fluency and strong data engineering experience in at least one of the following: Java, Python, or Scala, in a startup- or scaleup-like environment.
- Experience in the design and deployment of complex data schemas and flows and associated tools.
- Good knowledge and experience working with relational databases such as Postgres.
- Good experience with distributed storage, processing engines such as Apache Hadoop/Spark.
- You'll be confident working on CI/CD and test-driven design (TDD) best practises.
- Hold experience with one of the following Python tools such as pytest and mypy.
- Experienced working with Microsoft Azure, but accept familiarity with any of AWS or CGP
- Comfortable working within an Agile environment
- Experience designing for scale and troubleshooting scaling and performance problems; good understanding of technology stacks from networks up to applications.
- Excellent communication skills
Additionally, these skills would be nice to have:
- Experience with open-source workflow management/batch schedulers like Airflow, Prefect, or Dagster
- Comfortable with API design and architectural styles such as REST, OAuth, and architectural patterns such as microservices, serverless, federated systems
- Experience with DevOps automation / infrastructure-as-code toolsets such as Docker, Terraform, Ansible, and Kubernetes
Benefits
- Generous company pension package with employer contributions of up to 12%.
- 30 days annual leave (plus bank holidays.)
- Continuous career development with regular appraisals and learning and development opportunities.
- A lovely new office in Holborn, Central London we offer flexible and remote working arrangements