I have just run your code with a small calling function that passes the same input as I used before and the result was
EC50 dilute 200 m M - 0.01 m M
for the first sample text.
Also, the fact that the "dot" doesn't match line terminators is not an issue for you given your sample text as there are no RTF tags that cross a line boundary. The only part that might be affected is the first alternative, but that would show up by the pattern not matching (and therefore not replacing) any text of the form:
{\tag.....<line break>
...}
You would be left with (at least) the "{" in the output text.
Just a note about your pattern: it will not handle correctly the situation (as in your first example) where there are nested "{" "}"s. Given the text:
{\rtf1\{\fonttbl{\f0}{\f1}}
(edited down from your first example), your pattern will first match the "{\rtf1\{\fonttbl{\f0}" part and then the "{\f1}" part, leaving the "}" behind. This is because regexs can't (in general) count and therefore cannot be used to find matching items - this is often called the "balanced (or matching) parentheses problem". While some regex variants have extensions that allow this to be overcome, VBScript is not one of them.
Susan
Edit: Just a thought - try setting the ".global" to false and placing the match in a loop. While this will not fully emulate the action of the "Global" operation (it will re-scan the text form the start which the Global operation will not), it might give you an idea as to why the pattern is incorrectly matching the final "M" and therefore removing it. If nothing else, you should see the text being incrementally modified and see the intermediate stages.