There are a couple of things at play here, which coincidentally(?) mirror the difference between mathematicians who say "vector" to mean anything can be added commutatively and multiplied by a scalar, and physicists who say "vector" to mean a 3- (or 4-)dimensional quantity that transforms properly under change of coordinates.
While arbitrary rotations in 3 dimensions do not commute, infinitesimal rotations do. (In fact any "infinitesimal" transformations commute, as you can see by multiplying $I + \epsilon A$ and $I + \epsilon B$ and ignoring second-order terms in $\epsilon$.) Since angular velocity can be thought of as "infinitesimal rotation per infinitesimal time", it ends up being a vector (in the mathematical sense) even though rotation itself isn't. Similarly, angular momentum is the derivative of kinetic energy with respect to angular velocity, so it is a gradient, which is a (dual) vector (to angular velocity).
There's an interesting mapping between these infinitesimal rotation matrices and vectors (in the physical sense) which only works in 3 dimensions. If you think of an infinitesimal rotation as $I + \epsilon A$, one can show that $A$ must be antisymmetric. This means its diagonal entries are 0, leaving only 3 degrees of freedom in the off-diagonal entries. Such a matrix can be associated with a vector using the cross product:
$$\begin{bmatrix}0 & -\omega_z & \omega_y \\\ \omega_z & 0 & -\omega_x \\\ -\omega_y & \omega_x & 0\end{bmatrix} \mathbf{r} = \mathbf{\omega} \times \mathbf{r}$$
So angular velocity isn't "naturally" a vector in physical space, but rather lives in a different 3-dimensional vector space. That's why once you interpret it as a physical vector, it turns out to transform slightly differently, as J.M. has pointed out in the comments.