Python Back End Developer

At the core of the role is the development, maintenance and support of our ETL and data engineering frameworks that are custom built using Microsoft Synapse and Microsoft Data Lake for data storage and an Apache Airflow ETL Framework.

Requirements

• Understand the requirements
• Understand the design specifications
• Work with stakeholders to clarify acceptance criteria, edge cases, and non‑functional expectations (performance, availability, security).

Development

• Develops Airflow Operators within the company's Framework.
• Designs and develops backend services, APIs, and integration components in Python.
• Develops ETL pipelines and transformation logic.
• Organises code review and pull request of own and AI generated code.
• Raises Pull Requests and ensure timely code merge
• Documents implementation in meaningful inline comments and updates design documentation where required.
• Uses GenAI tools (Github Copilot / MS Copilot) to improve coding productivity
• Produces clean, maintainable code and follows coding standards.

Testing

• Creates test cases and ensures code quality is meeting expectations
• Reviews test results and provides updates and reports on any issues.

Technologies

• Ensures all Airflow/Python ETL processes and data is correct and up to date.
• Designs and develops backend services, APIs, and integration components in Python.
• Datawarehouse Concepts and SQL queries
• Maintains security and access to the warehouse.
• Ensures data quality is in accordance with the SLAs.
• Expert skills in development of python scripts and use of standard libraries.
• Ensures all code is of high quality and adheres to standards and best practices.
• Expands the ETL frameworks in line with the architecture requirements.
• Uses GenAI tools such as Github Copilot and MS Copilot for coding and development
• Reviews code changes of peers and ensures all code is of high quality.

Operational

• Performs rostered operational monitoring and system checks (rotational system across the team, generally one week every 3-4 weeks)
  • Ensures system health checks are performed and any issues identified.
  • Performs error triage during operational monitoring and manages any operational issue to conclusion
• Ensures Regression Test Cases are developed and available in Operation Monitoring Dashboard.
• Ensures data quality is maintained at all times.
• Ensures excellent Security processes and standards are in place and well maintained.
• Ensures pipelines and ETL processes are up and running within the agreed SLAs.
• Resolves System Errors in a timely manner and ensures any issues are resolved with the SLAs.

Must Have

• Must be confident in Python backend programming and data libraries eg. Pandas
• 3-5 years commercial Python (recent)
  • Backend and Database interaction
• Efficient in Standard Query Language (SQL)
• Good written and verbal communication

Nice To Have

• Working experience in Data processing and/or Data Engineering,
• Exposure to Big Data technology, eg Spark, Spark Streaming
• Apache Airflow
• Various GenAI Models such as GPT, Claude
• Familiar with Github Copilot
• Equivalent programming language (Java, C#, Scala)
• Big Data streaming, Spark Streaming, Kafka Streaming, Flink
• Experience working with Data Lakes ideally Azure Data Lake
• Experience with Parque File formats
• AZURE Synapse experience a plus