The numbers you describe are in fact all irrational.
Given a fixed rational number $p/q$, any other rational number $a/b$ that approximates $p/q$ (but is not equal to $p/q$) can only do so to order $1/b$. This can be seen as follows:
$$ p/q - a/b = (pb - qa)/qb,$$
which is at least $(qb)^{-1}$ if it is not zero. So rationals cannot be approximated too well.
Now consider the series $$\sum x_n/n!$$ and the partial sums $S_j$. The partial sums $S_k$ have denominator $k!$. Their error from the infinite sum is at most $\sum_{n>k} 1/n!$ which (from standard upper bounds via geometric series) little oh of $1/k!$ (as in the standard proof for $e$).
So basically, the problem is that the series converges "too rapidly" for the sum to be rational. A rational number cannot be approximated too well (well in the denominators) by other rationals.
Note that there are similar Diophantine approximation results for algebraic numbers, cf. Roth's theorem. Unfortunately that's not applicable here, though, since the approximations $S_k$ are little oh of the denominator, not little oh of a power of the denominator.