Engineering at scale.

Convolutional neural network for finger-drawn sketches. Four experiments to close the training-inference gap, most notably switching from pre-rendered 28×28 bitmaps to native stroke-vector rendering at 128×128. Built an X-Ray visualization to see what each layer learns. Live input via finger, mouse, or webcam through a Streamlit app.

Five algorithms scoring the same molecule for drug-induced autoimmunity risk. The spread between their probabilities is the headline; the chemistry interpretation runs once and stays the same regardless of which model the user picks.

Two PCA pipelines on audio: lossy compression on raw waveform blocks, and denoising on STFT spectrograms. Without spectral subtraction enabled, stationary noise hides at the top of the variance ranking instead of the bottom, and the denoiser does almost nothing.

Spark jobs that processed billions of records were dominated by shuffle cost on EMR. Pushed filters and projections before joins so data shrank before any wide transformation. Used broadcast joins for small dimension tables to skip shuffle entirely. Pre-partitioned hot datasets so repeated joins reused the layout instead of reshuffling.

Net result: faster jobs, smaller clusters, lower bills.