To solve $n(n+1)(n-4)\le 600$ algebraically, expand the left side and subtract 600 from both sides, so that you're comparing a polynomial to 0, find the zeros of the polynomial—which divide the number line into regions—and test a value in each region to see which regions satisfy the inequality.
Simpler, especially since you're only looking for positive integer solutions, would be to graph $y=x(x+1)(x-4)-600$ and look for where it is below the x-axis.
edit: Under very tight time constraints and without a calculator/computer, I'd have approached the original problem differently. The original statement ${n+1\choose n–2}–{n+1\choose n–1}\le 100$ (as well as the restatement ${n+1\choose 3}–{n+1\choose 2}\le 100$) can be interpreted as a statement about the (n+1)st row of Pascal's triangle: that the difference between the numbers 2 and 3 positions in from the edge is less than or equal to 100. It should be clear that the difference between adjacent entries in a row of Pascal's triangle is increasing, so it is sufficient to write successive rows until the statement fails:

The bold/darkest numbers are the relevant ones:
n=1: $0-1\le 100$;
n=2: $1-3\le 100$;
n=3: $4-6\le 100$;
n=4: $10-10\le 100$;
n=5: $20-15\le 100$;
n=6: $35-21\le 100$;
n=7: $56-28\le 100$;
n=8: $84-36\le 100$;
n=9: $120-45\le 100$;
n=10: $165-55>100$.
Also, note that the lightest numbers, below and to the left of the relevant numbers, are not used at all, so you don't actually need to compute them.