Formal Formulation of Optimization constraint types of conditions FONC and SONC Derivatives Directional Derivatives numerical methods Finite-Difference Method Forward Difference Central Difference Backward Difference Complex-Difference Method exact methods: autodiff Forward Accumulation cooool: Dual Number Method Bracketing (one dimensional optimization schemes) Fibonacci Search Quadratic Search Shubert-Piyavskill Method Descent Direction Iteration Line Search Approximate Line Search Sufficient Decrease Condition Curvature Condition Trust Region Methods First-Order Methods good ol gradient descent Conjugate Gradient Hyper-gradient Descent Second-Order Methods Newton’s Method or approximate it using Secant Method Direct Methods Cyclic Coordinate Search Accelerated Coordinate Search Powell’s Method Hooke-Jeeves Search Generalized Pattern Search opportunistic search dynamic ordering Nelder-Mead Simplex Method