you could make sth inspired by RSA, where you do not need two keys. like this:
calculate everything modulo $p$, where $p$ is prim. that will then be the GF[$p$] (GaloisField).
now instead of finding a generator $g$ for that GF[$p$] and calculating anything in powers of $g$, you could use any large numbers for multiplication and addition. this allone will then be very unsafe, but it will be very chaotic in combination with some bit-operations.
the encoding and decoding functions can be combined out of simple unsafe functions. the combination makes them more unpredictable, especially with the bit-operations.
(calculate once the reciprocal ($m^{-1}\equiv m^{p-2}$) of every $m$, that you use to multiply; you'll need it for decryption.)
let's define functions enc1 and dec1 based on multiplication:
$enc1_m(x)=x·m$ (mod p)
$dec1_m(y)=y·m^{-1}$ (mod p)
..................................................
and another pair based on addition:
$enc2_a(x)=x+a$ (mod p)
$dec2_a(y)=y-a$ (mod p)
..................................................
and another pair based on bit-shuffling, but that need to be invertible and never generate a value $\geq p$:
let $(\odot,\oplus,\otimes,\neg)$ be the bitwise (and,or,xor,not) operations
let $permutateAndXorLowestNBits_{s}(x,n)$ change the lowest n bits of x so that it is invertible. in other words, this condition holds:
$$\forall x,s,n: permutateAndXorLowestNBits_{s}^{-1}(permutateAndXorLowestNBits_{s}(x,n),n)=x$$
let N(x) be the number of lowest bits of x that can be "safely" changed so that after changing those bits, N will result in the same number and the value never exceeds p. in other words, these two conditions hold: $$x \oplus (2^{N(x)}-1) < p$$ $$\forall y. x \odot \neg(2^{N(x)}-1) \leq y \leq x \oplus (2^{N(x)}-1) \Rightarrow N(y)=N(x)$$
$enc3_s(x) = permutateAndXorLowestNBits_{s}(x,N(x))$
$dec3_s(y) = permutateAndXorLowestNBits_{s}^{-1}(y,N(y))$
..................................................
now, the resulting key will be (m,a,s) and the functions:
$enc_{m,a,s}(x)=enc1_m(enc2_a(enc3_s(x)))$
$dec_{m,a,s}(y)=dec3_s(dec2_a(dec1_m(y)))$
but any other more complec combination would be valid, too. remember, that any combination of several enc1s and enc2s can be expressed as one single combination of enc1 and enc2; so, alternate it with enc3.
i do not know, how hard it will be to break this if only p is known. without the bit-operations, a linear equation would break this, but with those bit-operations it will be very chaotic. does anyone know that?
otherwise, you could still make sth like a Feistel cipher, but i do not know wether it would be more secure or how to measure that.