/ the-workshop
Vizuara's Kernel Engineering Workshop
Eight live foundational lectures and six deep-dive workshops on modern topics — from the three performance regimes to DeepSeek's DSpark and AI-generated kernels. Enrolled students get the complete 72-chapter book, the GPU-Puzzles track and quizzes, the guided projects, worklog assignments, and the "You vs the machine" capstone.
8 foundational live lectures 2/week · 3 hours each · 4 weeks
L1
How fast can this go?
The three regimes, the roofline, and a top-down tour of the silicon. Live: predict-then-measure PyTorch ops.
L2
The CUDA programming model
Grids, warps, SIMT, and the nvcc→PTX→SASS story. Live: your first kernels + GPU Puzzles.
L3
The memory hierarchy in anger
Coalescing, bank conflicts, occupancy. Live: the matrix-transpose ladder under Nsight Compute.
L4
GEMM worklog I
Kernels 1–4: naive (1.3%) to 1D block-tiling (36.5%). Hypothesis → profile → number, every step.
L5
GEMM worklog II
Kernels 5–10: 2D tiling, float4, autotuning, warptiling (93.7%). Live: the SASS '8 loads → 2 loads' moment.
L6
Tensor cores, the second worklog
mma.sync, fragments, swizzling, the precision menu. Live: a WMMA GEMM beating our best SIMT kernel.
L7
Profiling & debugging like a pro
Nsight Compute deep-read + the vLLM workflow: sanitizer, core dumps, cuda-gdb. Live: 3 sabotaged kernels.
L8
Attention: the kernel that ate the world
Online softmax, FlashAttention v1 built live, why decode is GEMV. Capstone kickoff.
6 deep-dive workshops modern kernel inference topics
W1
FlashAttention from scratch
Full forward pass, online-softmax rescaling, causal masking; FA2/FA3 ideas.
W2
Beating cuBLAS on an H100
TMA + WGMMA + warp specialization, assembled into a library-beating GEMM.
W3
Triton → CUTLASS → TileLang
The abstraction ladder: Triton in 40 lines, then CUTLASS the hard way.
W4
Inference-serving kernels
Prefill vs decode, PagedAttention, fusion, and quantized (FP8/W4A16) kernels.
W5
Blackwell & NVFP4
tcgen05, Tensor Memory, CTA pairs, and the 2000µs→22µs FP4 GEMV journey.
W6
DeepSeek, DSpark & AI-written kernels
FlashMLA/DeepGEMM, speculative decoding, KernelBench and the human+AI+profiler loop.
Careers · interview prep
Interview-ready the questions you'll answer cold
Kernel-engineering interviews at NVIDIA, the neoclouds and the frontier labs test exactly what this workshop drills — hardware–software co-design, not framework trivia. By the end of the cohort you can answer these from muscle memory, and back every answer with a worklog of kernels you built and profiled yourself:
L1–L2
Execution model
“What is warp divergence, and what does it cost inside an if/else?” · “Define occupancy — why isn't 100% always fastest?” · “Walk me through nvcc → PTX → SASS.”
L3
Memory hierarchy
“What is memory coalescing, and how much bandwidth do you lose without it?” · “What causes shared-memory bank conflicts, and how do you kill them?” · “Registers vs shared vs global — when each?”
L4–L6
GEMM & tensor cores
“Write a tiled matmul and take it from naive toward cuBLAS.” · “How do 2D block / warp / thread tiling and float4 loads help?” · “How do tensor cores change the kernel?”
L7
Profiling & debugging
“How do you profile a kernel — which Nsight Compute sections first?” · “A kernel is slow: walk me through finding the bottleneck.” · “How would you debug a race, a NaN, or a hang?”
L8 · W1 · W4
Attention & inference
“Why is naive attention memory-bound, and how does FlashAttention fix it?” · “Explain online softmax.” · “Why is decode GEMV-bound, and what does PagedAttention buy you?”
W2 · W5
Modern hardware
“What do TMA and WGMMA give you on Hopper?” · “Write a kernel that scales across GPU architectures.” · “What changes on Blackwell with NVFP4 microscaling?”
Before you join
Prerequisites
We build CUDA from the silicon up, so no prior GPU or kernel experience is assumed. You'll get the most out of the cohort if you arrive comfortable with:
- Basic parallelism — threads, and the idea of many things running at once
- A working, high-level understanding of LLM inference — roughly what happens when a model generates a token
- A little Python, and the willingness to pick up some C/C++ as we go
Every cohort ends with a graded exam, so you leave with a clear, honest measure of what you've mastered. Multi-GPU, cluster and fleet-scale topics are shared as supplementary lecture notes — the live sessions stay focused on getting the most out of a single GPU.
For companies & partners
Hire our graduates. Partner on the frontier.
Kernel engineers are one of the hardest hires in AI. By the end of a cohort, our graduates have built a GEMM from naive to 94% of cuBLAS, FlashAttention from scratch, Hopper & Blackwell kernels (TMA, WGMMA, NVFP4), and DeepSeek-grade inference kernels. Three ways to work with us:
hiring
A hiring pipeline
Consider the strongest of each cohort for your kernel-engineering roles: a warm, pre-vetted pool of engineers with exactly the skills on your job descriptions.
capstone
Sponsor a capstone
Give us a real kernel problem your team cares about. We run it as a sponsored capstone, you see the solutions and the talent, and your company is credited on the project.
compute
GPU credits & partnership
Sponsor H100/B200 hours for students' capstone work and become a founding partner, with your logo on the workshop, the site, and the certificate.
Founding partners— announced soon —
