Q1: Are convex subsets assumed to be in vector spaces or topological vector spaces?
Convex sets are easily defined in topological vector spaces over a subfield of $\mathbb{C}$, following the usual definition: $A$ is a convex set if $\forall x,y\in A$ we have $tx+(1-t)y\in A$ for $t\in[0,1]$. Basically means we can draw a line between any two points and the line segment is contained in $A$.
For non-euclidean spaces convexity generalises to geodesically convex set see "Convex functions and optimization methods on Riemannian manifolds" by Constantin. There are a few other generalisations. For the most abstract there are Convexity spaces which is an axiomatic description but I can't remember much about this.
Q2: Are convex functions defined to be from convex subsets in vector spaces or topological vector spaces to general field or some special field such as R or C?
Firstly I have seen a convex function defined without the need for its domain to be a convex set at all! It required the function to only be convex on any convex subset of its domain. (I don't think is standard though.)
Secondly, let $f:I\rightarrow K$ be a convex function where $I$ is a convex subset.
If $K=\mathbb{R}$ then $f$ is convex iff $f(tx+(1-t)y)\leq tf(x)+(1-t)f(y)$, $t\in[0,1]$. Is suspect this could be generalised to any ordered field.
If $K=\mathbb{C}$ then we can't use this definition as $\mathbb{C}$ is not a totally ordered field. Instead we have $\mathbb{C}\sim\mathbb{R}^2$ and use the hessian: $f$ is convex iff $\nabla^2f$ is positive semi-definite. However this requires $f\in\mathcal{C}^2$.
Q3: What are the special cases that have been extensively studied and have significance in application?
The majority of the applications in convex optimisation are with a real vector space and a convex real-valued function. There are so many applications for this the mind boggles. A short memory dump is:
- Least squares problems
- Linear programming: network flow problems, mircoeconomics.
- Quadratic programming: This forms the basis of so many optimization algorithms
- Entropy maximisation