Optimization problems subject to PDE constraints form a mathematical tool that can be applied to a wide range of scientific processes, including fluid flow control, medical imaging, biological and chemical processes, and many others. These problems involve minimizing some function arising from a physical objective, while obeying a system of PDEs which describe the process. It is necessary to obtain accurate solutions to such problems within a reasonable CPU time, in particular for time-dependent problems, for which the “all-at-once” solution can lead to extremely large linear systems. In this talk we consider Krylov subspace methods to solve such systems, accelerated by fast and robust preconditioning strategies. A key consideration is which time-stepping scheme to apply — much work to date has focused on the backward Euler scheme, as this method is stable and the resulting systems are amenable to existing preconditioners, however this leads to linear systems of even larger dimension than those obtained when using other (higher-order) methods. We will summarise some recent advances in addressing this challenge, including a new preconditioner for the more difficult linear systems obtained from a Crank-Nicolson discretization, and a Newton-Krylov method for nonlinear PDE-constrained optimization. At the end of the talk we plan to discuss some recent developments in the preconditioning of multiple saddle-point systems, specifically positive definite preconditioners which may be applied within MINRES, which may find considerable utility for solving optimization problems as well as other applications. This talk is based on work with Stefan Güttel (University of Manchester), Santolo Leveque (University of Edinburgh), and Andreas Potschka (TU Clausthal).