For the case you're interested in it is true, because the set of finite order $n$-by-$n$ matrices over $\mathbb{N}$ is precisely the set of permutation matrices. In fact, every invertible element of $M_n(\mathbb{N})$ is a permutation matrix. (Here $\mathbb{N}$ contains $0$, and invertible means invertible in the monoid $M_n(\mathbb{N})$ with multiplication.)
Suppose that $B=(b_{ij})\in M_n(\mathbb{N})$ is not a permutation matrix. Then either one of the entries of $B$ is greater than $1$, one of the rows or columns of $B$ has more than one nonzero entry, or one of the rows or columns of $B$ is zero. Let $A=(a_{ij})\in M_n(\mathbb{N})$ be invertible.
Suppose first that $B$ has an entry greater than $1$, i.e., $b_{i_0j_0}\gt1$ for some $i_0$ and $j_0$. Since $A$ is invertible, its $i_0^\text{th}$ column is not zero, so there is an $i_1$ such that $a_{i_1i_0}\gt0$. Then if $AB=(c_{ij})$, we have $c_{i_1j_0}\geq a_{i_1i_0}b_{i_0j_0}\gt1$, so that $AB\neq I$. Hence, $B$ is not invertible in this case.
Now suppose that $B$ has a row with more than one nonzero entry, i.e., $b_{i_0j_0}\gt0$ and $b_{i_0j_1}\gt0$ for some $i_0$ and $j_0\neq j_1$. Again, there is an $i_1$ such that $a_{i_1i_0}\gt0$, and we have $c_{i_1j_0}\geq a_{i_1i_0}b_{i_0j_0}\gt0$ and $c_{i_1j_1}\geq a_{i_1i_0}b_{i_0j_1}\gt0$, so again $AB\neq I$. A similar argument applies if $B$ has a column with more than one nonzero entry, either by multiplying in the other order, or by considering the transpose. Thus in either case $B$ is not invertible.
This leaves only the case that $B$ has a zero row or column, in which case it is even a zero divisor. Thus in any case, $B$ is not invertible.