You need to specify the problem better. Do you count rotations and reflections as different? For example, imagine a regular pentagon drawn on a sheet of paper with a corner up. Starting from the top corner and going clockwise, maybe we have 0,1,2,3,4 (using numbers instead of colors). Is this different from 1,2,3,4,0? How about from 0,4,3,2,1?
The easiest situation is if you say these are all different. Then the naive approach would be to say you have $q$ choices for the top corner, $q-1$ choices for each of the next three corners (as you can't have neighboring corners that match) and $q-2$ choices for the last, giving $q*(q-1)^3*(q-2)$, but this ignores the fact that the first and fourth corners could be the same color, giving $q-1$ choices for the last.
One way to deal with this situation is to separate into classes based on which corners match the top one. If we designate the top corner 0 and count clockwise, you can match the top color only in no corner, corner 2, or corner 3. So matching none, we have $q*(q-1)*(q-2)^3 $ as at corner 1 you can use any color but the color at corner 0, but for each other you have two eliminated. Matching corner 2, we have $q*(q-1)*1*(q-1)*(q-2)$ and matching corner 3 we have $q*(q-1)*(q-1)*1*(q-2)$. Add them all up and you have your answer.
In general, the answer has to be a fifth degree polynomial. For large $q$ the constraint of not matching colors won't reduce the count very much. So if you count the solutions for six different q (I suggest 0 through 5), you can fit a fifth degree polynomial through them.
If you want to count rotations and reflections as the same there are two possibilites. One way is to define a standard position and only count those cases in standard position. In this case this is easy. You would say that the smallest number has to be at the top, and that corner 2 is less than corner 3. A pitfall would be to say the smallest is at the top and corner 1 is less than corner 4. It could be that corner 1 and 4 are the same, and it could be that one of corner 2 and 3 is the same as corner 0. But counting how many configurations satisfy this constraint may not be easy. It is again a fifth degree polynomial. The other case is to list the configurations of matching corners and see how many times they each get counted. So if no corners match there are $q*(q-1)*(q-2)*(q-3)*(q-4)$ possibilities, but you have counted each one 10 times, so divide these by 10. Then you can have one pair or two pair of corners matching. These are probably easier to count by the standard position approach. If one pair matches, say they have to be positions 0 and 2. So we have $q*(q-1)*1*(q-1)*(q-2)$ choices for this. And so on.
Sorry for taking out the asterisks from my expressions, but it rendered in italics and run over itself when they were there. I think they made it easier to read.