Since the last value for $j$ is $k^2-1$, none of the terms of the sum are $k$; they are all between $1$ and $k-1$.
How many $1$'s will be in the sum? Well, we'll get $1$ when $j$ is any number between $1^2$ and $2^2-1$; then we'll get $2$ for each number between $2^2$ and $3^2-1$. Then we'll get $3$ for each number between $3^2$ and $4^2-1$. Etc.
So, if $n\leq k-1$, how many times does it show up in the sum? It shows up exactly the number of times that there are integers between $n^2$ and $(n+1)^2-1$, inclusively. This is
$$(n+1)^2 - n^2 = n^2 + 2n + 1 - n^2 = 2n+1.$$
So your sum has $2(1)+1 = 3$ ones; $2(2)+1 = 5$ twos; $2(3)+1=7$ threes; etc. Up to $k-1$, which appears exactly $2(k-1)+1 = 2k-1$ times.
So we get that
$$S = \sum_{r=1}^{k-1} r(2r+1) = 2\left(\sum_{r=1}^{k-1}r^2\right) + \sum_{r=1}^{k-1}r.$$