Besides the least square model based on 2- norm, there are other models based on other norms, such as the best uniform approximation model based on infinite norm. The best uniform approximation problem of a variable can be solved by Remez algorithm.
Some models can be solved by general optimization algorithm (random gradient descent method is one of them), and sometimes they can be solved by algebraic method, for example, some problems can be solved by nonlinear eigenvalue problems.
Generally speaking, both modeling and solving need to be considered, but they cannot be completely confused.