What you are forgetting is that you have more than one inequality.
The inequality $a^2\geq 8$, by itself, only gives you $|a|\geq \sqrt{8}=2\sqrt{2}$, or equivalently as you (almost) write:
$$a\geq 2\sqrt{2}\qquad\text{or}\qquad a\leq -2\sqrt{2}.$$
Added: Note that you took $a^2\geq 8$, a weak inequality, and "translated" it into "$a\gt 2\sqrt{2}$ or $a\lt -2\sqrt{2}$." Technically, this is incorrect because you have now excluded the possibility of equality. Since you were starting with $\geq$, you should have continued with $\geq$ and $\leq$.
However, you have other inequalities that need to be satisfied as well.The second inequality tells you that $0\lt a \lt 6$ must be satisfied. How does this mesh with the inequalities you already have?
If $0\lt a \lt 6$ and $a\leq -2\sqrt{2}\lt 0$, then that tells you that $a$ must be both greater than and less than zero. This is a very big problem for $a$, which decides to solve this problem by ceasing to exist. There is no solution to the inequality $0\lt a\lt 6$ that also satisfy $a\lt -2\sqrt{2}$.
So putting these two together, you know that you have $a\geq 2\sqrt{2}$, and $0\lt a \lt 6$. Since satisfying $a\geq 2\sqrt{2}$ automatically guarantees you that $0\lt a$, you can combine them all by saying that the first two inequalities you have are equivalent to
$$ 2\sqrt{2} \leq a \lt 6.$$
Finally, we need to add in the condition that $a\lt \frac{11}{3}$. Since $\frac{11}{3}\lt 6$, if $a$ is less than $\frac{11}{3}$ it will also automatically be less than $6$. It still needs to be greater than or equal to $2\sqrt{2}$, though. So combining them all you get that you must have
$2\sqrt{2} \leq a \lt 6$ in order to satisfy all three (four, if you count "$0\lt a\lt 6$" as two inequalities) inequalities you have.
Okay, the original problem was: "if the two roots of $x^2-ax+2=0$ lie in the interval $(0,3)$, then find the values of $a$."
The roots are $\frac{a\pm\sqrt{a^2-8}}{2}$. So you assumed that you only needed to ask for $a^2-8\geq 0$ in order for the roots to be real. However, the use of the word "both" suggests that in fact they want there to be two distinct roots, which would require $a^2-8\gt 0$. If you take $a=2\sqrt{2}$, the lower limit I suggested earlier, then you have a single (repeated) root at $\sqrt{2}$ (which satisfies the condition). If they want two roots, then you need the discriminant positive, not merely nonnegative.