Michael Ash's Regex Blog

Update to CSS Minification

2008-04-27T22:06:00Z

This is a C# 2.0 enhancement of a C# port of YUI Compressor 's CSS minification code I got a little carried away with ideas for this, they were all regex based which really is what motivated me to work on it. However after I thought I was done I learned not everything worked. It did what I wanted it to do but what I wanted wasn't the correct thing. I really should have just stopped with my original ideas. The last idea for my original changes was to take 2 or more individual subset properties...(read more)

Follow up to Additional CSS minifying regex patterns

2008-04-18T20:04:00Z

OK, there regexes were discussed in the previous post this is mostly just their application. This is a C# 2.0 enhancement of a C# port of YUI Compressor 's CSS minification code Since I was doing this is C# I took full advantage of it's regex engine, namely using lookbehinds and delegates for some replaces. Almost all the regexes after the "New Test" comment are the new or modified regexes from the ported version. There is also one new and two modified expressions before that comment....(read more)

Additional CSS minifying regex patterns

2008-03-27T20:55:00Z

NOTE: All the regex referenced on this page written by me are using IgnoreCase = true I was looking at the regexes used in the YUI Compressor to minify CSS and came up with a couple of more that I think could help the process. The code and port I was looking at was already trimming unneeded zeros used for the top, right, bottom, left values with a simple string replace. But there were three separate replaces being done. It was pretty simple to come up with a regex to handle all the cases (Pseudo...(read more)

A touch of Character Class

2008-01-31T16:49:00Z

The square brackets character class is one of the more misunderstood of the basic regex features. This feature is supported in virtually all regex implementations. In fact off the top of my head I don't know an implementation that doesn't support it. Maybe it's not well documented in most tutorials or maybe the samples are not clear enough or maybe users are just skimming over the details for this one but I see this feature misused quite often. This type of character class simply matches...(read more)

Remember where you come from

2007-10-01T06:42:00Z

One thing I've noticed among rookie regex users is that they focus way too much on what they are trying to match and not on where they are trying to match it from. There is a tenancy to drastically under estimate the importance of their source data, which they are trying to apply their regex against. I see this all the time on forums where posters asking for help almost never post any sample data, and most of the few that do post a sample they made up on the spot. Over on they RegexAdvice construction...(read more)

Are you ready for regex?

2007-06-01T17:02:00Z

Who should be using regular expressions?

This has been on my mind for a while, there are some people shouldn't be using regular expressions. People who don't know what regular expressions are for. Now you can use regexes in a variety of ways, and you can debate either side on whether certain applications are good uses or bad uses, but that's not what I'm talking about. I'm not even talking about people who don't know how to write regexes, well or at all. I'm talking about people trying to use regular expressions but have know idea what regular expressions do. And I don't mean that you can't decipher a regex pattern by looking at it. I mean you don't understand the general concept that regular expressions match patterns in data.

I've seen several post over on the regexadvice.com forums in the past few months which go something like “I have this problem someone told me I should use a regular expression” but unless that same someone bother to explain what a regular expressions does or why it is suited for your task, don't be in such a hurry to plug in a regex.

I'm not saying you need to master regex before using them just get your head around the basics (matching). At the very least you should associate regex to wildcard searches, just more powerful. If you are at least that far, proceed with trying to implement your regex. But if you are thinking something completely different slow down there. Unless you have a basic understanding of what regular expressions do, even simple tasks become way more difficult than they should be. Even if you get someone to help you write a regex and still don't know what it's doing you are likely to continue to incorrectly try to use them.

A common result from this lack understanding I've been seeing is trying to perform a task solely via writing a regex pattern, that regexes themselves have no capacity to perform. Not that a regex couldn't be a part of the solution but in the end it's not even what will do what the person wants done, usually it a coding task, but the regex can at least do some of the prep work of the task. Even if some implementations of the regex engine allow code to be called, it is still the code doing the heavy lifting. The regex is only finding the data. One could save themselves hours of wasted time if the just understand the basics of what they are getting themselves into.

Another thing I've seen, related to the not having a basic understanding, is a task that is not only perfectly suited for using a regex but a very common application of a regex task and the person asking the question asking “Is this some that a regex can do?” Of the issues I've mention this one is the most and the least understandable. The most because regular expressions are not simple to everyone, to some it comes easy to others it never comes completely. And some of the documentation isn't the greatest, though there is plenty of good documentation out there these days. So I can understand why some can't get their head about exactly how to write a regex. But it the least understandable when they are asking how to write a regex so common, and usually simple, that dozens of tutorials and/or articles on regexes use the very regex they are asking for as an example.

I think the main problems of people who fall into category is

Lack of research. There are more than enough tutorials out on the web and more than a few books that have simple samples to give you an idea of what a regex does. I find it very hard to believe when someone says they couldn't find a regex for basic (US) zip codes. Did you even look? Or are you only using a regex for this task because someone told you that you should and ran out to find someone to write it for you? or are you just cheating on your homework?
Misunderstanding what regular expressions understand. There seems to be more than a few people who think a regex understands the context of the data it matches. That it not only know how to match the data but it know what it means, either in context to their application or to the world at large. Those are the people that think a regex for matching zip codes know what zip codes are used for and where they would be used. Sorry but that's not the case. Regexes understand nothing of what your data means to you. As far as the regex engine is concerned it's just a string of characters. It's up to the regex author know the context of the data they want to match and to shape the regex accordingly to return only the relevant data. This may make it seem like the regex understands the data it's matching but that not what happening. What happen is the person who wrote the regex understood the data and the problem set so well they were able to construct a regex the only match the relevant values.
That regex is a full blown programming language. It's not. I've seen questions about wanting a regex to compare numbers, tell time or do some other function completely outside their realm but something most programming languages have a feature to deal with or let you write code that can. I've never seen any regex documentation promoting such features so I can't imagine why someone thinks a regex can perform these task. Other than my previous point where they saw the results of a well written regex and speculated on what and how much work the regex did. Like I mentioned some implementation allow you to perform function calls but that is more an add-on of the programming environment you are using than a generic regex feature. Realize that regular expressions are one of many features of your programming language, not the other way around.

If you've read this far and you didn't know what regular expressions did or didn't do before hopefully by now you have some idea. And if you already knew what they did just stop to consider the next time you advise someone to use a regex, you make sure you get across the high level point that you are “matching something (a character pattern) in a string” before you get into the more complex aspects of what a regex can do.

You've got your sub-matches in my matches

2007-06-01T16:47:00Z

Hello boys and girls. Wow it's been a while since I've done this. I want to touch on a very useful but often overlooked feature of regex, grouping. While I haven't been blogging I have been active on a message board here or there. A question I see quite often is “I want to find a match in a string but I don't want part of the match” or “I need the value of this portion of the string” Now often I see solutions to these type of questions that involve look-arounds. Any why they certainly work they aren't the only way to achieve the desired results. New regex users seem to believe they can only access the full match. Most regex engine support groups, where in a match you can access a certain portion of the full match. Groups are identified by parenthesis. Every pair of plain parenthesis is a group. For example the regex pattern

/(Hello)\x20(world)/

There are two groups in the regex. The regex itself matches the string “Hello world”, group 1 contains the string “Hello”, group 2 contains the string “world”. Note neither group contains the space between the two words but it is part of the full match. Most implementations of regular expressions allow you a ways to access these groups. They are contained in a collection inside the match object. Now you'll need to consult your regex documentation to know the exact layout but in most of these implementation there is a zero based collection where Item 0 is the full match and Item 1..n is whichever group element your regex contains, if any.

Now if you notice I said every pair of plain parenthesis is a group. The reason I stress plain is because there are other group constructs, the aforementioned look-arounds being some. Now support for the other constructs vary in implementations so again consult your regex documentation to see which ones you have. The other grouping constructs consist of a open parenthesis immediately followed by a question mark, which then is followed by the characters that define that particular grouping construct. Again consult your documentation to see which characters define what. I'm not going to go into all of them here but they basically fall into two categories. Capturing and non-Capturing. The plain parenthesis I've mentioned above are a capturing group. However capturing requires extra resources in some case you need the extra speed, but you still to group a certain part of the pattern together for either necessity or readability or both. This is where you'll what to using a non-capturing group (?:pattern), a open parenthesis followed immediately by a question mark followed immediately by a colon. The difference here is that the data matched in the group is not add to the collection of submatch in the Match object.

Taking our previous example and making the first group non-capturing

/(?:Hello)\x20(world)/

Where before we had two groups here we only have one. Group 1 contains the string”world”. Now this example is not a very practical use of a non-capturing group. Typically you'd use them in more complex regexes that have a grouping but you really don't care about the sub-matches.

One more quick thing about capturing groups basically each left parenthesis is the index of the sub-match in the groups collection. So if you have nested parenthesis count every (plain) left parenthesis to know which index to use to reference it. Some of the advance grouping constructs and regex options can affect the ordering but if you are using them hopefully you've read their effects so I won't go over that here.

/((Hello)|(Goodbye Cruel))\x20(world)/

The above regex has 4 capturing groups (not counting group 0). Can you find them? Now it should match either the string “Hello world” or “Goodbye Cruel world” Now I want to point out that not all the groups will participate in the match, but the are still part of the Groups collection. There will always be 4 groups, just one will always be empty. Which one depends on which string was matched.

If “Hello world” was matched the groups are

Hello
Hello
(empty)
world

If “Goodbye Cruel world” was matched the groups are

Goodbye Cruel
(empty)
Goodbye Cruel
world

in both case group 0 would be the full match

If you note in both case two groups contain the same value. Even if you need to know whether “Hello” or “Goodbye Cruel” was match, you certainly don't need to know it twice. Plus the inner parenthesis have different index you'd have to check if you want to use those. This is where you'd use the non-capturing group to simplify your groups collection.

/((?:Hello)|(?:Goodbye Cruel))\x20(world)/

Now we are back down to two groups. Group 1 contains either “Hello” or “Goodbye Cruel” depending on which string was matched. Group 2 always contains “world”

However keep in mind in some cases you'll want to use the inner index do determine which group was matched. So using non-capturing groups isn't necessarily a better thing it just depends on if you need to access those groups or not. But if you are not doing anything with them don't capture them.

These are just two of the basic grouping constructs and they are general supported across implementations of regex, but not always. But if they are you can use the to easily dissect larger matches.

Named Groups to the Rescue

2005-09-28T20:06:00Z

I was asked to modify some text that had been built incorrectly. Basically insert some text at a certain point. First I use a regex to find the text, then insert the new value within that match. Now since the inserted value goes inside the matched text I simply wanted use backreferences and the replace method. Simple right? Well not so much.

Now the text is in a field of various rows of a database table and the text to be inserted comes from another of the fields in the same row and is an alphanumeric value. So the inserted text value is dynamic, so I can’t simply hard code the replacement text. So the replacement text is built dynamically for each row. The text to be modified is a certain attribute somewhere in the text. For this example lets say it’s “id=xyz”, which is constant for all records. Now the new text will be inserted right after the equals sign.

So for

source =“ {some stuff} id=xyz {more stuff}”
newText = “ab1”

you get

“{some stuff} id=ab1xyz {more stuff}”

Simple enough. You use this regex \bid=xyz\b to match the text. Then split it in to groups so you can use backreferences in the replace. So your final regex looks like this:

\b(bid=)(xyz)\b

Now group 1 contain the text up to your insertion point (id=)

And group2 contains the text after your insertion point (xyz)

So your replacement string for your regex is “$1(new data goes here)$2)” , where (new data goes here) = some alphanumeric value pulled from a second field in a row.

Doing this is in .Net my code looked something like this pseudo-code

Regex regexFind = new Regex(“\b(bid=)(xyz)\b”);

Get Records

For each row

fieldA = rowFieldA (source text)

fieldB = rowFieldB (insert value)

fieldA = regexFind.Replace(FieldA,String.Format(“$1{0}$2”,fieldB))

The Format method of the string create a replacement string for each row.

Look good? Works find for our example but there is a problem.

For our example value of “ab1” the string format produces “$1ab1$2” which is exactly what we want, but as this field is alphanumeric so it could begin with a number which causes a problem. Say for the next record the value of the text to be inserted is “12a” the format method produces a replacement string of “$112a$2”, which is not good. Syntactically it’s fine but it’s not what we want, because instead of trying to inserts some text between group 1 and 2, which is what we want to do, it is trying to insert text between group 112 and group 2. As there is no group 112 it assumes $112 is literal text so your final result is “id=$112axyz”

Ok this is where named group become handy (necessary?). If you used name groups in your regex and replacement string you can avoid this problem

Change the regex to \b(?<att>bid=)(?<val>xyz)\b

And your replacement string to “${att}(new data goes here)${val})”

Now if you are using the string format method there is one more hoop you have to jump through because the regex engine and format method both use the curly braces there is a conflict and the format method will complain so you have to write it like this

String.Format("${0}att{1}{2}${0}val{1}","{","}",newValue)

To get the desired replacement string.

When I started writing this I thought this was the only way to get this to work which means you could only solve this with a regex engine, like .Net, that supported named groups, but I’ve thought of a second way.

But whenever a group in your replacement string can be followed by a digit you may want to consider using named groups to avoid unexpected surprizes

Making your regex code ready.

2005-05-18T16:44:00Z

There are times when regular expression you’ve written or someone written for you needs a little tweaking before you add it to your code and the tweaking is required because the syntax of the language conflicts with your regex. For example when part of your regex pattern contains a double quote and the language you are using uses double quotes as string delimiters. If you just cut and paste the pattern in your code the pattern’s quotation will terminate your string prematurely. Now the code way to fix it is to escape the quotation in pattern. This solution requires altering the regex and how the character is escapes depends on the language being used. The regex itself allows you to escape character with the \ character. The language being used may or may not recognize that as escape character for its syntax. And it may be confusing later when you look at the regex and can’t remember why you escaped a character that the pattern itself doesn’t need it, But there is another way. Hex values

Most regex implementations support a hex syntax \x##, where # is a hex digit.

So if you use \x22 instead of double quote and \x27 for single quotes the regexes become more cookie cutter ready.

Another useful hex value is \x20 which is a space. This is especially useful in .Net where there is an option on a regex to ignorewhitespace in the pattern. Turning this option on allows end of line comments in the regex but with the exception of inside a character class, ignores typed in spaces within the patterns, which would be problematic if a space was part of the pattern to match. So you could break a working regex if you later decide to add this option. This happened on the Regexlib when the option was first turned on. A lot of patterns that were written before the switch was flipped suddenly stopped working.

Speaking of .Net when it comes to name groups you can’t use the hex notation to define the group name using the single quote syntax . However you can avoid any issue with single quotes by using the alternate syntax.

Word Break.

2005-02-09T18:17:00Z

The list-detail design is very commonly used with web pages, where you have a list of links that lead to more detailed information of each entry. Sometimes the text of the list is simply a snippet of a much longer string of text in the detail. A common way to handle this is to use a string function to return the first n characters of the string and display that in the list. The problem with this is that tends to make the break right in the middle of a word. Which isn’t major problem but can be aesthetically displeasing or may accidentally form another word you didn’t mean to put on your site. When facing this issue I came up with a simple regex to allow me to break on whole words. ^(?:[ -~]{n,m}(?:$|(?:[\w!?.])\s)) Where n = the minimum number of characters to match And m = the maximum number of character to allow in the match. Now in instance I’m considering a word to be one or more ACSII non white-space characters. The way it works is after matching n ASCII characters it tries to match either the end of the string or a letter or sentence ending punctuation followed by a white space. So it will accept as many characters, including white spaces as it can up to m and still satisfy the rest of the match. Otherwise it backtracks until the regex is satisfied. So if you wanted a minimum of 2 characters and a maximum of 75 the regex would be ^(?:[ -~]{2,75}(?:$|(?:[\w!?.])\s)) and if you applied it the Gettysburg Address “Four score and seven years ago, our fathers brought forth upon this continent a new nation: conceived in liberty, and dedicated to the proposition that all men are created equal. …” (only the 1st paragraph shown for the example but you could apply the full text) Taking the first match you get “Four score and seven years ago, our fathers brought forth upon this ” There are a few problems with the regex that can be improved. First off it only accepts basic ASCII displayable characters, decimal 32 to 126 with mean the text must be in that range. I did it this way because it give you the US alphabet, digits and commonly used symbols and punctuation which was all I needed at the time. Other characters would need to be added. Also if the first word character count exceeds your maximum length no match will be found You can make this regex a little dynamic by putting inside a function that takes the your string, the max and min values as input.

Making Dynamic XHTML pages valid with a Regex

2005-01-26T21:47:00Z

I have a website that I use primarily to expand and sharpen my web development skills. The latest effort in that regard had been writing valid markup. Although there is some usable function to this site I use mainly for practice. Consequently I don’t work on it full time. Instead work on it in spurts, with different editors, apply new scripts, new ideas and such at different times so the output is far from consistent. So I went to a lot of effort to first make my HTML valid, the later convert that HTML to XHTML and make that valid. However being that the content of most of the pages is dynamically generated, when validating the XHTML it didn’t occur to me right away that something in the content could make my page not validate in certain situations. Case in point ampersands. According to the W3C (http://www.w3.org/TR/xhtml1/#C_12) you should not have solitary ampersand characters in your markup. Ampersands declare the beginning of a entity references. If you wanted to display the ampersand character you should use the & entity reference instead of (&). Keep in mind most browsers let you get away with incorrect usage so I didn’t have do any to get my page to render but see as the whole point of writing valid mark up is not to depend on the browser fixing my sloppy code I didn’t want to just let this go. The first time I encountered this problem on a page the fix was pretty simple. I was working with ASP so I simply wrapped ASP’s Server.HTMLEncode method around the variables containing the database derived content. Simple. Problem solved. Ummm not quite. The problem occurred on another page inside a dropdown list. My first thought to use the HTMLEncode method again was a horrible failure. Unlike my previous fix where I was pulling one row’s column, let’s say First Name, out of a database and sticking in a single variable for latter use, all the markup for the dropdown was being created in a separate function using ADO’s Getstring method. So the variable containing the dropdown contained the full XHTML markup (the Select and option tags). HTMLEncode turns the less than and greater than signs of the tags into entities which in turn basic displays your intended markup source code on the page.

Again the intent in to turn the solitary ampersands (&) into the & entity. So I decide to use a regex to fix this problem. I modified this regex http://www.regexlib.com/REDetails.aspx?regexp_id=626 which matches entities to this

&(?!(?i:\#((x([\dA-F]){1,5})|(104857[0-5]|10485[0-6]\d|1048[0-4]\d\d|104[0-7]\d{3}|10[0-3]\d{4}|0?\d{1,6}))|([A-Za-z\d.]{2,31}));)

Which matches ampersands which are not part of an entity. So using the replace pattern of “&” I can replace just the solitary ampersand.

Now some of you might think this is overkill since I could use VBScript’s Replace function to get the result I want. And in this very specific case that is true. There were only a handful of ampersands to replace. However this issue came up other places where a simple replace would not be easily implemented. The truth of the matter I didn’t come up with this regex for this problem. This was originally designed for dealing with XML files that contained solitary ampersands AND entity references. The VB or VBScript Replace won’t work it that case. For example that the following line

using HTMLEncode you get

Using the VB replace function you get

Neither being what you want.

So the regex replace allows your content to contain entities and won’t mess them up and you get the desired output

Higher plane solution

2004-11-08T17:45:00Z

I’ve come up with a workaround to Unicode plane bug I talk about earlier

Originally I was trying to identify Unicode values outside of the ASCII range, code point higher than 127, for a replace. However the problem was characters in any other plane besides plane zero returned two matches per character. Meaning I could get two incorrect replacements for those characters. Turns out the replacement would just not taken place which was still a problem.

The following two regexes allowed me to get around this behavior.

Plane 0 excluding standard ASCII pattern (?![\uD800-\uDBFF])(?![\uDC00-\uDFFF])[\u0080-\uFFFF]

Non-Plane 0 pattern [\uD800-\uDBFF][\uDC00-\uDFFF]

The ranges D800-DBFF and DC00-DFFF are the High and Low surrogate values respectfully. Surrogate code points are the values in UTF-16 encoding of the two 16-bit code units that make up a Supplementary Character

You may notice in the plane 0 pattern the surrogate pairs are in separate lookaheads. You might think that they could be combined into one look ahead, but this won’t work. I’m not exactly sure why this is, but doing so will get you a match on the low surrogate of a non-plane 0 character.

I've posted these pattern to the regexlib. Unfortunately most of the characters used in the examples can't be displayed on the site. The sample characters I used are

Plane 0 : ☻

Plane 1: 𐌈, 𐍈 and 𐐏

Flavor of the month

2004-10-05T19:58:00Z

It seem most regex engines based their implementation on what had been done in the PERL version, with some implementing a little more, others a little less. This is what generally keeps regexes from being portable across different implementations. This often leads to confusion and syntax errors when you take a canned regex that was designed or tested in a different environment than it will be used. I’ve talked before about knowing the syntax

of the implementation the regex will be used to save yourself the trouble of trying to fix regexes that aren’t incorrect but just written to be supported by a different engine.

There is a new testing tool on the Regexlib. It’s pretty neat and a great improvement over the previous version. However it pretty much targeted for Microsoft regex engines. Realizing not everyone is working in that environment and more importantly allowing myself (and others) a chance to play around with regular expressions using syntax not currently supported correctly or at all by those engines, I’ve found several online regex testers for a variety of languages and platforms.

Environment/Language Site

Microsoft(.Net, VBScript ,Jscript) http://www.regexlib.com/ReTester.aspx (currently requires IE for VBScript and Jscript testing)
Javascript http://www.unusualresearch.com/regex/toolrei.htm or http://www.regular-expressions.info/javascriptexample.html
Java
1. Jakarta ORO(Perl5, AWL, GLOB) http://jakarta.apache.org/oro/demo.html
2. Jakarta Regexp http://jakarta.apache.org/regexp/applet.html
3. java.utitl.regex http://www.fileformat.info/tool/regex.htm
PHP (preg, ereg) http://www.quanetic.com/regex.php

Please refer to each site’s documentation (if there is any) on what features and advanced syntax is supported. But here is a quick highlight of something one site supports that the other may or may not.

.Net = Named Groups and Look-Behinds

ORO = POSIX syntax

java.utitl.regex = Character Class Subtraction, Possessive quantifiers

JScript Bug

2004-10-05T15:26:00Z

I saw a comment on the Regexlib that expression

        ^(?=.*\d)(?=.*[a-z])(?=.*[A-Z]).{8,15}$

didn’t work for JavaScript.

I responded mainly because the alternative offered didn’t wasn’t really an equivalent expression.   But when I looked at the original I really didn’t see why it shouldn’t work in JavaScript and there is any. This does work fine in JavaScript.  However it DOES NOT work in Jscript (or VBScript).

This expression is suppose to match a string of 8 to 15 characters containing at least one digit and one lowercase and one uppercase English letter.

The suggested use is for password requirements.

There appears to a bug with the client-side Microsoft regex engines.

Here’s what’s expected to happen. Each look-ahead test is executed one at a time.

1) Is there a digit in the string?

2) Is there a lowercase letter in the string?

3) Is there a uppercase letter in the string?

If all three are true the final test for the string being 8 to 15 characters is executed. If this is also true we have a match. This is how the engine works in .Net (and other non-Microsoft regex engines)

Here is what’s happening in with the client-side (VBScript/Jscript) Microsoft regex engines.

1) Is there a digit, followed by 7 to 14 characters

2) Is there a lowercase English letter, followed by 7 to 14 characters

3) Is there a uppercase English letter, followed by 7 to 14 characters

After one look-ahead finds the character it’s looking for it tries to satisfy the .{8,15} portion of the regex. It does this for each look-ahead making the regex rather sporadic in its matching. Using the Regexlib's testing tool while 123456aB7 should match , like it does in .Net, it doesn’t because the letters aren’t follow 7 more characters after the lowercase and 7 after the uppercase. However aB12345678 matches because there are 7 characters (but not the same 7) after each letter case and the digit. So in effect this forces the input to be at least 10 characters because the client-side evaluation isn’t possible in fewer characters. And all the look-ahead conditions must be satisfied by the 8^th character to the left of the end of the string.

Switched at birth

2004-09-13T15:28:00Z

Seem there is another bug in the .Net regex engine concerning Unicode

The characters matched by the Unicode categories for quotes are backwards.

Pi – InitialQuotePunctuation Indicates that the character is an opening or initial quotation mark. Signified by the Unicode designation "Pi" ( punctuation, initial quote ).

Pf – FinalQuotePunctuation Indicates that the character is a closing or final quotation mark. Signified by the Unicode designation "Pf" ( punctuation, final quote )

However \p{Pi} will match a closing quote and \p{Pf} will match an opening quote. This is backwards.

I should note the quote characters in question are the Unicode left and right ‘single’ and “double” quotation marks not the ASCII double quotes (") or apostrophe (')