After asking a few questions earlier, I think I've been able to describe my main issue. Could someone show me why this identity is true:
$$\frac{\partial \textbf{x}^{T} B \textbf{x}}{\partial \textbf{x}} = (B + B^{T})\textbf{x}$$
I've gone through things like Matrixcookbook which seem to just show this identity without any derivation of it. I have tried to unsuccessfully derive the formula below:
Via product rule:
$\frac{\partial \textbf{x}^{T} B \textbf{x}}{\partial \textbf{x}} = \frac{\partial \textbf{x}^{T}}{\partial \textbf{x}}B\textbf{x} + \textbf{x}^{T}\frac{\partial B \textbf{x}}{\partial \textbf{x}}$
I realize product rule for scalars is not the same for matrices, but it still holds as long as order is preserved.
At this point, I'm stuck because I'm not sure if $\frac{\partial \textbf{x}^{T}}{\partial \textbf{x}}$ is identity $I$.
I assume this definition of the Jacobian:
$\begin{bmatrix} \dfrac{\partial y_1}{\partial x_1} &; \cdots &; \dfrac{\partial y_1}{\partial x_n} \\ \vdots &; \ddots &; \vdots \\ \dfrac{\partial y_m}{\partial x_1} &; \cdots &; \dfrac{\partial y_m}{\partial x_n} \end{bmatrix}$
And if I write $\frac{\partial B \textbf{x}}{\partial \textbf{x}}$ component by component with the definition above, it becomes simply $B$.
Thus,
$\frac{\partial \textbf{x}^{T} B \textbf{x}}{\partial \textbf{x}} = B\textbf{x} + \textbf{x}^{T}B$
which is not correct, the left side is a nx1 vector and the right side is a nx1 vector.
I'm not sure where my faulty assumption is. I've gone through various sources which don't explain the situation I showed above. Am I missing some very simple point that those books assume?