There is a standard technique we learned for dealing with integrals of the form $(*) \int_{a}^{\infty}f(x)\cos (\alpha x)dx$ or $\int_{a}^{\infty}f(x)\sin (\alpha x)dx$ when $f(x)$ is continuous and positive in $[a, \infty)$.
If $\int_{a}^{\infty}f(x)dx$ converges, then (*) converges absolutely by the comparison test because $|f(x)\sin (\alpha x)|\leq f(x)$ (or $|f(x)\cos (\alpha x)|\leq f(x)$).
If $\int_{a}^{\infty}f(x)dx$ diverges while $f(x)$ is decreasing and $\lim_{x\to\infty}f(x)=0$ then the integrals (*) converge conditionally by Dirichlets test.
I'll illustrate the technique on your integral:
First, lets substitute $t=x^2$, then $(**) \int_{0}^{\infty} \cos(x^2) \mathrm dx=\int_{0}^{\infty} \frac{\cos(t)}{2 \sqrt t} \mathrm dt$.
Let $f(x)=\frac{1}{2\sqrt x}$ then $f(x)$ is decreasing and $\lim_{x\to\infty}f(x)=0$. Also $cos(x)$ has a bounded anti-derivative. Therefore (**) converges by Dirichlets test.
Now we observe that $|\frac{\cos(x)}{2 \sqrt x}|\geq \frac{\cos^2(x)}{2 \sqrt x}=\frac{1}{4\sqrt x}+\frac{\cos(2x)}{4\sqrt x}$. By the same arguments above $\int_{0}^{\infty}\frac{\cos(2x)}{4\sqrt x}dx$ converges. But then if we assume that $\int_{0}^{\infty}\frac{\cos^2(x)}{2 \sqrt x}$ converges, we get that $\int_{0}^{\infty}\frac{1}{4\sqrt x}$ converges and that's obviously not true.