Iain Hunter - Lead Data Engineer | AWS & Data Architecture

About

Lead Data Engineer with 20+ years experience building scalable data infrastructure on AWS. Expert in Python, Terraform, and serverless architectures. Specialized in designing end-to-end data platforms, automating ETL workflows, and optimizing pipelines for performance and cost-efficiency. Skilled at translating business requirements into practical solutions while leading agile teams and mentoring engineers.

Tech Stack

Python AWS Terraform SQL Pandas Snowflake Prefect.io AWS Lambda AWS Step Functions Amazon S3 AWS DMS Streamlit Data Engineering Data Architecture

Certifications

AWS Certified AI Practitioner

SnowPro Core

AWS Certified Developer

Certified ScrumMaster

Oracle 9i Certified Associate

Projects

Hunter Labs AWS Serverless Data Lakehouse

Hunter Labs AWS Serverless Data Lakehouse is the platform behind my data products, designed to stay event-driven and serverless so it costs almost nothing when idle. Terraform provisions the AWS foundation, Python Lambdas and Step Functions move new files through ingestion and sanitisation, and S3 Tables with Apache Iceberg provide the lakehouse layer for Athena, DuckDB, Streamlit apps, and Prefect.io curation flows.

Python Terraform AWS Lambda Step Functions S3 Tables Apache Iceberg DuckDB Athena Prefect.io Streamlit

User Sessions Map

User Sessions Map is an interactive Leaflet visualisation that replays anonymised user activity across a global map. It loads static JSON session and office data, animates activity pulses over time, tracks active countries and leading markets, and stays deployable as a lightweight static web project.

Static HTML JavaScript Leaflet JSON S3 CloudFront

View User Sessions Map

Hidden Gems

Hidden Gems is a small movie discovery site that surfaces one curated film each day from Netflix and Prime Video. It combines AWS-hosted data pipelines, Prefect orchestration, Terraform-managed infrastructure, and a lightweight static front end to publish daily picks, refresh JSON content, invalidate CloudFront, and push matching posts to X, with a clean landing page, direct watch links, and a fully automated publishing flow behind the scenes.

Python Prefect.io DuckDB Apache Iceberg AWS S3 CloudFront Terraform Static HTML

View Hidden Gems

YFinance & Streamlit - Magnificent 7 Stocks Tracker

YFinance & Streamlit - Magnificent 7 Stocks Tracker follows Apple, Microsoft, Nvidia, Amazon, Alphabet, Meta, and Tesla through an analytics pipeline. A daily scheduled job pulls adjusted market data from YFinance, lands the raw extracts in S3, and runs them through the Hunter Labs AWS Data Platform. Lambda and Step Functions handle ingestion and validation before curated price history is written into an Apache Iceberg data lake. Streamlit then reads the lakehouse-ready data, visualising interactive price trends, ticker comparisons, returns, and recent movement views on top of the same serverless platform.

Python YFinance Streamlit AWS ECS AWS Lambda Step Functions S3 Apache Iceberg

Let's Connect

Ready to discuss your next data engineering project?

Let's Connect