Getting Started with Deep Learning

• Hello world, PyTorch style: The PyTorch team’s introduction to deep learning introduces you to all the basic concepts and guides you through training your own neural network
• TinyGrad MNIST - A guide to solving MNIST (the hello world of deep learning) using TinyGrad
• Getting started in JAX - The JAX team’s guide to getting started with JAX.
• Getting started with PyTorch Geometric: For those of you wanting to develop GNNs, start here! They also have some lecture videos you may find useful.

Books

• Alice’s Adventure in Differentiable Wonderland: An introductory text book to differentiable programming (using frameworks like PyTorch or JAX to create neural networks).
• The Art of Science and Engineering: Turing award winner Richard Hamming’s Art of Science and Engineering is a cult classic book that covers a wide array of topics, from mathematics to ‘style’. Don’t be put off by some of the very technical chapters, they can be dense but rewarding. During his time at the US Naval college he also delivered a lecture series based on his book chapters.

Websites

• Deep learning engineering tricks: A blog post a wrote a while ago discussing different tricks you can use within your deep learning model to improve its performance.
• Scaling Your Model in JAX An online book created by Google DeepMind that provides a rich resource in how to scale your deep learning models.
• The Illustrated Transformer: A very famous post explaining how the Transformer works under the hood. Recommended reading for anyone starting out in deep learning and wanting to use Transformers.
• Transformers are Graph Neural Networks: A very good post explaining an alternative way of viewing how the Transformer works.

Lectures

• Karpathy’s Zero to Hero Series: Andrej Karpathy completed his PhD in image captioning at Stanford before going on to be a founding member of OpenAI. He then moved on to run Tesla’s self driving team, before returning to OpenAI. In this lecture series he introduces the foundations of deep learning, starting from backpropogation, and goes all the way to the Transformer.
• 3Blue1Brown’s Introduction to Deep Learning: An excellent animated lecture series introducing the key theory behind deep learning.
• Richard Hamming’s Learning to Learn Series: Based on his book The Art of Science and Engineering, he recaps his career discussing what worked (and what didn’t). In later lectures he covers more abstract concepts such as “taste”. His lecture titled You and Your Research is considered a ‘must watch’ in many circles of computer science.

Papers

• Attention is All You Need: This famous 2017 paper from Google introduces the Transformer architecture, the breakthrough that has lead to the modern AI revolution. If you are working in deep learning, it is essential to understand this paper.
• Neural Message Passing for Quantum Chemistry: This paper introduces the concept of “message passing” for graph neural networks. It presents a framework for “messages”, “update functions”, and “readout functions” that has been widely adopted in the GNN literature.
• Vision Transformer (ViT): This paper introduces the ‘Vision Transformer’ or ‘ViT’ and shows how a Transformer can be used to replace traditional computer vision architectures such as CNNs. PyTorch provides pretrained ViT models.
• Vision GNN (ViG): This paper introduces Vision GNN (ViG) which shows how a GNN can be used instead of a Transformer to provide a vision backbone. It works by constructing a kNN graph over patch features. Section 3.1 contains an interesting discussion on how they minimised the oversmoothing problem associated with GNNs

Developer Environment

You’re going to spend a lot of time working on your project, so it’s a good idea to invest some time in making your development environment comfy. Start by doing little things like changing the font (maybe enable font ligatures?), and changing the default colour scheme - there is more to life that ‘light’ and ‘dark’ themes! Then take a look at some other tools - what do people in industry use? Ask yourself, what causes friction when you’re working on your project? Is there a way to fix it? What stops you from openening your laptop and picking up where you left off? If an idea strikes you between lectures, how quickly could you implement it?

You should learn to use your editors to the point where they become an extension of you. How much can you do without the mouse? Software engineering is your craft, treat it as such.

• WSL2 and Windows Terminal - Finally, a good command line on Windows! (Other than powershell of course)
• i3/tmux/nvim setup: How quickly can you move between projects?
• tmux: Tmux, the terminal multiplexer
• Fuzzy Finders: The best way to move around your codebase! Look at tools like fzf or plugins for JetBrains and VSCode