I have lots of ideas for this. Probably too many!
already suggested, you could use aluminum foil instead of the brass. I just measured the roll I have here in my kitchen and it is conveniently right about 1 mil (.001”) thick – so great for fine-tuning if you need that, or just use 4 layers to get the thickness you mentioned.
But it sounds like cutting the sheet cleanly enough is the main issue you are having with the brass? In that case, aluminum may be even more troublesome when using that "razor compass" you mentioned. But I wonder if you might have better luck just using sharp scissors (probably just for the outer diameter) and/or a craft knife? If you can find cylindrical-shaped objects that are close to the right size for your cutouts, that would help to hold the sheet(s) in place and serve as a guide for cutting with the knife. Maybe that would be enough to keep it from buckling or distorting too much? Another aid for cutting might be to back up the material with a stiffer layer of something (perhaps card stock?) or possibly sandwich it between two such layers. You might have to hold the stack together with double stick tape to make it work, but if you are able to cut through all the layers then I would think that the backing would help the metal keep its shape better. Having said all that, I tend to agree with Spanik
that it is probably OK of the piece distorts a little or has rough edges when you cut it anyway. Although I can't speak for the brass, I find that with aluminum I can smooth it out by just burnishing it against a smooth hard surface, using the back of a fingernail or really almost any object with a smooth edge. It is really soft.
In the end if the thin metal sheet won't cooperate, there are a lot of other materials that could probably work just as well (or better). In fact, "they" make plastic shim stock kits that are designed to pretty much serve the same purpose as the metal shim stock. The main difference is that the plastic tends to be a little easier to cut into custom shapes using common tools like scissors or utility knives. In lieu of one of these commercial plastic shim kits, a little scrounging can usually turn up some sort of thin plastic sheet material that has the right thickness for you - maybe a sheet of mylar wrapping, or some packaging material, perhaps the ESD bag that an old electronics part came in, or maybe even a piece of film? ...just throwing out some possibilities.
Finally, have you considered plain ole’ paper? I just measured the thickness of an ordinary sheet of white paper sitting here on my desk, and it happens to be right about 4 mils thick. Paper is about as straightforward as anything to cut, and also offers the ability to first print the pattern you are trying to cut out directly onto the paper. As far as I know, paper is pretty dimensionally stable as well, and once sandwiched in place isn’t going to tear or do anything unexpected. Of course having just said that, I realized that it only applies for a one-time assembly. Is the interface at this adapter going to be repeatedly connected and disconnected? If so, then maybe paper isn't the best material. Still, I suppose adhesive could be applied to keep it in place and then it might be OK. ...kind of like your current solution with the tape, now that I think about it!