As described in Brualdi and Cvetkovic's Combinatorial Approach to Matrix Theory, rather than associating a directed graph in the usual way to a matrix, it's more natural to associate the König digraph for the purposes of thinking about matrix multiplication. For an $m \times n$ non-negative integer matrix $A$, the König digraph is a bipartite digraph with vertex classes $X$ (with $n$ elements) and $Y$ (with $m$ elements) such that the number of edges from $x \in X$ to $y \in Y$ is $A_{yx}$. Composition of matrices then corresponds to "plugging" one digraph into another, and whatever purpose you want to multiply adjacency matrices for, I think it will be more natural (or at least equivalent) phrased in this language instead.
What I've described is a category whose objects are the non-negative integers $[n], n \ge 0$ and where a morphism from $[n]$ to $[m]$ is a collection of arrows from an $n$-element set to an $m$-element. This category is a certain refinement of the category of finite sets and relations and is a very natural setting for certain combinatorial arguments (as well as for, as the book says, a combinatorial approach to matrix theory). I don't know if it has a name, though.