writing gpu kernels = writing programs on the gpu to make it run faster
oftentimes this looks like making matrix multiplications happen faster (compute-bound work), or, for memory-bound work (a) reducing the number of trips to slow memory and (b) overlapping/hiding memory latency behind compute so fetches and math happen concurrently and minimizing idle GPU time
the job is broken into two phases. first is understanding where in the system to optimize, what parts will return most bang for buck, profiling, and the second is figuring out how to optimize it and how to make sure the optimizations hold with scale & edge cases
optimizations that work on a single node might not work on multi-node systems
gpu kernel work is continuous since new kernels have to be re-written for new hardware that gets released (yearly). the math itself though doesn’t really change. math is more foundational and new methods take lots of time to develop (say, 50 years)
field has high barrier to entry
lots of prerequisites: linear algebra, calculus, gpus, cpus, machine architecture, systems programming
historically, no roadmap, but now there’s more resources to get people into the field
required reading: computer architecture: a quantitative approach