Data Engineer
Pipelines, storage, and modelling data at scale.
- SQL fundamentalsWhat a SELECT actually does in execution order rather than in the order you type it, and why GROUP BY, HAVING, and WHERE are not interchangeable filters on the same data.
- Window functions, CTEs, and recursive queriesRANK vs DENSE_RANK vs ROW_NUMBER measured on real ties, LAG for period-over-period deltas without a self-join, and the recursive CTE mechanics that make a bill-of-materials query possible.
- Query optimisationSARGable predicates measured against the non-sargable rewrite that "does the same thing," why HAVING after GROUP BY costs more than WHERE before it, and reading a plan for where the actual cost is hiding.
- Dimensional modellingStar schema versus a normalized OLTP schema, and why "when did this customer live at this address" is a modelling decision — slowly changing dimensions — not a query you can bolt on afterward.
- Data profilingWhat to measure before trusting a new dataset — cardinality, null rate, and the mean-vs-median gap that tells you whether "average" is even a meaningful number for this column.
- Data cleansingWhy deduplication needs a definition of "same" before it needs code, imputation changes a column's statistical properties whether you notice or not, and cleansing logic that silently drops rows is the failure mode to design against first.
- Data quality frameworksGreat Expectations, dbt tests, and data contracts solve the same underlying problem at different points in the pipeline — what breaks when quality checks run after the damage is already downstream.
- Pipelines and orchestrationWhy a pipeline step has to be idempotent before it can be safely retried, what a DAG actually buys you over a cron job, and the backfill that silently double-counts because a task wasn't designed to run twice.
- Data ingestion and change data captureBatch versus streaming as a latency-and-cost trade, why polling for changes misses deletes and intermediate states, and what a CDC connector reading the transaction log actually captures that a timestamp column can't.
- dbtWhat "transform in SQL, version like code" actually buys you, why the materialization you pick trades build time against query time, and the incremental model bug that silently reprocesses the same rows forever.
- SnowflakeStorage and compute billed separately and what that actually buys you, warehouse sizing as a credits-per-hour multiplier, and the idle-warehouse cost that silently doubles a bill nobody looks at closely.
- Python for data — pandas, vectorisation, and polarsA measured 50x gap between a Python loop and a vectorised pandas operation, the SettingWithCopyWarning that means your edit silently didn't happen, and why polars closes both problems by construction rather than by convention.
- Document and PDF extractionWhy a PDF has no concept of a table, how layout-aware extraction reconstructs structure from position rather than reading markup that doesn't exist, and the silent column-misalignment failure that looks fine until someone checks the numbers.
- Data visualisation principlesWhy a truncated y-axis isn't a style choice, when a chart is genuinely the wrong tool for the question, and the aggregation-before-charting decision that determines whether the picture matches the data underneath it.
- Power BI — storage modes and DAX contextImport versus DirectQuery as a freshness-for-speed trade, and context transition — the DAX mechanism that turns a row context into a filter context — which is where most "why is this measure wrong" bugs live.
- PostgreSQL in productionWhat EXPLAIN actually says once you know where the loops divide, why VACUUM is not optional, and how a table can be 500,000 rows and still choose to scan every one of them — correctly.
- Indexes, joins, and reading a query planWhy a LEFT JOIN can return more rows than either table has, how the four join types relate as sets, and where the selectivity crossover actually sits — measured, not assumed.
- MongoDBEmbed or reference, why a document's growth pattern decides the schema more than the query pattern does, and the write-amplification bug that only appears once an array stops being small.
- CosmosDB and partition keysWhy the best-distributing partition key is often the wrong one, how RU/s gets divided among partitions you don't control, and the migration you can't avoid once a key choice turns out wrong.
- NoSQL data modellingModel by access pattern, not by entity — why a NoSQL schema starts from the queries you'll run rather than the things you're storing, and what breaks when a new query arrives that the model didn't anticipate.
- pgvectorVector search inside Postgres — what you gain by not standing up a second datastore, what HNSW costs in index build time and memory, and when the "one less service" argument stops being the right trade.
- File formats and object storageRow-oriented versus column-oriented storage, why Parquet reads 10x less data for an analytical query than CSV, and what Delta and Iceberg add on top that a folder of Parquet files can't do alone.
- GCP — projects, IAM inheritance, and Cloud RunWhat's genuinely different about GCP once you already know AWS and Azure — the project hierarchy IAM inherits down through, and Cloud Run's per-request billing as a distinct point between Lambda and a always-on container.
- How LLMs workThe mental model an engineer actually needs — next-token prediction, attention, context as the only state, and which observed behaviours follow from the architecture rather than from a bug.
- PromptingWhat actually moves the needle — structure, examples, and output contracts — and how to tell a prompt problem from a retrieval problem before spending a week on the wrong one.
- EvaluationHow to know whether a change made things better — building a golden set, choosing metrics that survive contact with production, and using an LLM judge without fooling yourself.
- EmbeddingsHow text becomes a vector, why similarity is an angle rather than a distance, and the failure modes that make a retrieval system quietly return the wrong documents.
- Vector searchApproximate nearest neighbour search — what HNSW actually trades away, why recall is a dial rather than a property, and the memory cliff that turns a fast index into a slow one.
- RAGRetrieval-augmented generation as a pipeline of separately-measurable stages — and why debugging it end-to-end is the most expensive mistake in the field.
- Chunking and retrievalWhy chunk size is not a knob you can reason about monotonically, what overlap actually buys, and how an answer that is provably in your corpus becomes unretrievable.
- Reranking and hybrid searchWhy embeddings are bad at identifiers, how to fuse two rankings whose scores are not comparable, and where a cross-encoder earns its latency.
- Machine learning fundamentalsThe vocabulary an AI engineer needs — what training actually optimises, why accuracy is usually the wrong metric, and how to read a model someone else built.
- ML data preparationLeakage, splits, encoding and imbalance — the stage where most model quality is won or lost, and where the bugs are silent by construction.
- AI data readinessAssessing whether a corpus can support an AI feature at all — the audit to run before the project, and the findings that should stop it.
- Knowledge graphsWhen entities and relationships beat embeddings — multi-hop questions, aggregation, and the extraction cost that decides whether a graph is worth building.