I am neither a mathematician nor particularly good at maths but here is my 'pennyworth' on the problem.
(1) Weighing the same denomination coins (assuming they are the same 'model') is extremely accurate with a reasonable digital scale. Once one is weighing upwards of twenty or thirty coins, the likely deviation from the true number of coins drops dramatically (thanks to the wonders of The Central Limit Theorem). I have used this approach to take inventories of paper work sets and I was surprised by just how accurate the item counts were using a cheap digital scale weighting up to 10 kilogrammes and with an accuracy of +/- one tenth of a gram. I had a weight to number conversion calculation in a spreadsheet but obviously a purpose-made scale 'would incorporate the conversion on a chip. There should be scope for 're-calibration' given that coins change.
(2) With a mixed bag of coins, it strikes me that no matter how cunningly one might design the coin weights, the more coins one had, the greater the number of possible weight combinations. At some point, these would 'eat into' the maximum accuracy range of the balance. If this is so (and this is only my intuition, this should be proven or disproven), then this would suggest a maximum number of coins that could be weighed at any one time. It could be objected that prime numbers solve this problem but if we think of the scale's accuracy as defining a minimum measurable number space, I suspect that populations of high primes or combinations with even relatively few primes could fall in the same number space and not be discriminated. Again, it would take a mathematician to show whether this intuition is grounded or foundless.
[P.S. Reading the foregoing posts more carefully, I see that Qiochu has already found the upper bound for a given set of weights]. This suggests another approach, namely, batching mixed coins into maximum number of coins or total weight and then weighing (and thus accurate counting) using Qiochu's system.
(3) One also should think of practicalities:
(a) At the moment, coin weights are not designed for counting by weight.
(b) Most people I have seen counting coins do so from a till in which the coins have already been sorted into denominations. This suggests that in the real world, the approach in the first paragraph is the best one.
(4) A fourth reflection is that with the advent of cash cards, 'money' cards, smart phone apps and other 'virtual purse' solutions, maybe the problem of counting coins is one that will not be with us for much longer. Perhaps coins could also be made another way, incorporating a passive (secure and non-rewritable!) RFID tag to 'tell' counting machines what denomination they are.