Another approach, which can't work in general (see Noah's answer), but will surely work in this case, is to find a normal form for each element of the group, and then see whether there are finitely or infinitely many.
In practice, that means imagining a word in x and y, and then applying the relations as much as possible to simplify it, and then trying to figure out (and prove!) what the possible different "irreducible words" (i.e. words that can no longer be simplified) are.
In your case, the first thing one would note is that x can only appear to the 1st power
(since any higher power can be simplified using x^2 = 1), while y can only appear to the powers +1 or -1 (for the same reason). Also, we can't have too many expressions of the
form xy or yx in a row, because of the third relation.
One can keep going like this. I didn't, but what I imagine is that one can have
expressions of the form x y x y^{-1} x y x y^{-1} ... that are arbitrarily long,
and inequivalent, explaining the infinite order of the group.
It shouldn't be so hard
to settle the question from this point of view by sitting down with pencil and paper and just playing around with different words to get a feel for what kinds of reductions can
take place. (In geometric arguments like Grigory's, one uses a geometric context, such as
an action on a hyperbolic tiling, as a more conceptual way of understanding the relations and distinguishing inequivalent words. But in this case I'm sure it won't be hard to see
everything directly from the presentation.)
Added after rereading the question: what I am suggesting is precisely that even when the answer might be infinite, you can still hope to find a coset enumeration, at least in a case
like this with relatively simple relations.