Tooling: Git, GPUs, experiments and the Hugging Face Hub

Git: a time machine for your code

Git records snapshots (commits) of a directory and lets you branch, merge and travel between them; GitHub hosts those repositories and adds pull requests, issues and a public portfolio. The subset you need daily:

# shell commands, shown here for reference
# git init                      start tracking a folder
# git status / git diff         what changed since the last commit
# git add -A && git commit -m "Add baseline logistic regression"
# git switch -c feature/knn     create and switch to a branch
# git push -u origin feature/knn  publish the branch; open a pull request on GitHub
# git pull                      bring in others' commits
# git log --oneline --graph     read the history

Commit small and often with messages that say why. Three things must never be committed: secrets (API keys — use a .env file listed in .gitignore and read it with os.environ), large data or model weights (use the Hugging Face Hub, DVC or cloud storage; Git chokes above ~100 MB), and notebook output cells full of images (strip outputs with nbstripout or clear them before committing, otherwise every diff is unreadable). A repository with a README, a pinned environment and a src/ folder is already more professional than most.

Where to get a GPU

Deep learning needs a GPU for anything beyond toy models because matrix multiplication parallelizes across thousands of cores (4.12 Compute, hardware and scale basics). The question is which one, and the honest answer for a learner is: free ones first.

VRAM is the quantity to reason about: weights × bytes per parameter + optimizer states + activations (4.12 Compute, hardware and scale basics does the arithmetic). When renting, set a budget alert, use spot/preemptible instances with checkpointing, and shut down in the same session you started. For API-based work (8.4 Prompting as programming, 9.6 Generation side: context assembly, citations, structured outputs and tool calling) the same discipline applies: set a monthly hard cap in the provider dashboard, cache responses during development, and prefer smaller models while iterating.

Reproducibility: seeds and their limits

A result you cannot reproduce is an anecdote. Four layers of reproducibility, from cheapest to most complete:

1. Pin the environment — uv.lock or requirements.txt with exact versions (2.1 Python for ML, fast).
2. Seed every source of randomness — Python's random, NumPy, and PyTorch (CPU and CUDA). Data shuffling, weight initialization and dropout all draw from these.
3. Log the configuration — every hyperparameter, the data version and the Git commit hash, saved next to the results.
4. Determinism flags — torch.use_deterministic_algorithms(True) and cudnn.benchmark=False make GPU kernels pick deterministic (sometimes slower) implementations.

import random, numpy as np, torch
def seed_everything(seed=42):
    random.seed(seed); np.random.seed(seed); torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

Even with all four, bit-for-bit identical results across different GPUs or library versions are not guaranteed — floating-point addition is not associative, so parallel reductions can differ in the last bits and the differences compound over training. The practical standard is therefore statistical reproducibility: run with 3–5 seeds and report the mean and spread (1.8 Statistics essentials). A difference between two methods smaller than the seed-to-seed spread is noise, not a finding — an idea you will meet again in evaluating LLMs (8.7 Evaluating LLMs).

Experiment tracking

After ten runs you will not remember which learning rate produced which curve. Trackers solve this: Weights & Biases (hosted, free for individuals) and MLflow (open source, runs locally) both record configs, metrics over time, system stats and artifacts, and give you a dashboard to compare runs. The pattern is three calls: init(config=asdict(cfg)) at the start, log({"loss": loss.item(), "lr": lr}, step=step) inside the loop, and finish() at the end. Log everything cheap (loss, metrics, learning rate, gradient norm, throughput) and the hardware you used; log rarely the expensive things (sample predictions, images). Name runs descriptively and tag them. Compare runs by the metric that matters on held-out data, never by training loss alone (3.5 Evaluating models: metrics, validation and the bias–variance trade-off).

The Hugging Face Hub

The Hub is the GitHub of machine learning: 1M+ models, hundreds of thousands of datasets, and Spaces for hosting demos, all versioned with Git under the hood. You will use it in three ways. Consume: from datasets import load_dataset; ds = load_dataset("imdb") streams a benchmark dataset with splits and features already defined, and AutoModel.from_pretrained("...") downloads pretrained weights (7.5 Encoder-only: BERT and friends (and sentence embeddings)). Publish: huggingface-cli login once, then model.push_to_hub("you/my-model") uploads weights plus a model card — a README describing the data, training procedure, intended use, limitations and evaluation. Writing a model card forces the questions that 8.12 Safety, risks and responsible AI formalizes. Demo: a Space with a app.py built on Gradio turns a model into a web app in twenty lines, which is how the projects in 5.10 Project: train and deploy an image classifier and 8.13 Project: fine-tune a small LLM with LoRA and ship a chat app get shipped.

The datasets library deserves a note of its own: it is Arrow-backed, so datasets larger than RAM load lazily and map transforms are cached; ds.to_pandas() and Dataset.from_pandas connect it to 2.3 pandas, data wrangling and plotting.

A project skeleton

# learnai-lab/
#   pyproject.toml, uv.lock, .gitignore, .env (never committed), README.md
#   data/        raw/ and processed/ (git-ignored; documented in README)
#   notebooks/   exploration only; numbered 01_eda.ipynb, 02_baseline.ipynb
#   src/lab/     data.py, model.py, train.py, evaluate.py  (importable code)
#   configs/     yaml or dataclass configs per experiment
#   runs/        checkpoints and logs (git-ignored; tracker has the metadata)
#   tests/       small tests for data shapes and a one-batch training smoke test

Move code from notebooks into src/ as soon as you run it twice. A README that states what the project does, how to set it up (uv sync) and how to reproduce the main result (uv run python -m lab.train --config configs/base.yaml) is the single most valuable artifact for your portfolio (11.3 Portfolio project ladder).

Where this shows up later

• Every project lesson assumes this skeleton and a tracker — 3.14 Project: end-to-end tabular ML with scikit-learn, 5.10 Project: train and deploy an image classifier, 8.13 Project: fine-tune a small LLM with LoRA and ship a chat app, 9.11 Project: a cited RAG assistant over your own documents, with evals, 10.12 Project: build a research agent with tools, an MCP server and an eval harness.
• VRAM and FLOP arithmetic get their full treatment in 4.12 Compute, hardware and scale basics and 7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization; cost control for APIs returns in 9.9 Production: latency, cost, caching, guardrails, security and privacy.
• Seeds, variance and error bars underpin honest evaluation — 3.5 Evaluating models: metrics, validation and the bias–variance trade-off, 8.7 Evaluating LLMs, 10.9 Evaluating agents.
• Model cards and dataset documentation are part of responsible AI — 3.13 Feature engineering, data quality and data-centric ML, 8.12 Safety, risks and responsible AI.