Got more questions? Find advice on: ASP | SQL | XML | Windows
in Search
Welcome to RegexAdvice Sign in | Join | Help
HomeBlogsForumsPhotosDownloads
RegexAdvice » Regular Expressions » Construction Advice » Parsing logs for errors NOT matching a list.

Parsing logs for errors NOT matching a list.

Last post 01-11-2009, 7:40 PM by RichardRay. 5 replies.
Sort Posts: Previous Next

  •  01-11-2009, 1:04 PM 50040

    Parsing logs for errors NOT matching a list.

    Reply Quote

     I am using a PERL-based tool for regex on the Windows platform.

    I have a vendor-supplied application that produces voluminous log output. Error events are included with all kinds of other events; I've been told to ignore about a dozen different error messages, and more from time to time. I need to extract ERROR and FATAL entries from the log, but not extract lines that are for certain error descriptions. I have successfully extracted just the ERROR and FATAL rows with this expression:

      (\S*[\d] ERROR|\S*[\d] FATAL\S*)

    A sample of the data returned is below. I need to exclude from that the rows that contain error descriptions I've been told I can ignore. In the sample below that would include the rows containing 'GetLineItemRule' and 'Unable to execute forward'. I have been experimenting with capture groups and negative lookahead but have had no success; there's something I just don't grok about that concept. I can post a fragment of the original log, but it would have to be very long to be representative, and I don't seem to have permissions to attach a file to this post.

     

    Here's a short sample of the matching lines from the log file that the above regex returns correctly:

    ===================================================================================================

    2009-01-11 04:43:51,490 1 ERROR com.rtp.acropolis.rtpone.RtpOneOrder - GetLineItemRule : Unable to get rules. Rule will not be loaded. Rule type (1) [<reqres><req><LineItem-RetrieveRule><OrderId>22897426</OrderId><LineNumber>4</LineNumber><RuleKeyword>FIXEDDATEAGE</RuleKeyword></LineItem-RetrieveRule></req><res></res></reqres>]

    2009-01-11 04:43:51,490 1 ERROR com.rtp.acropolis.rtpone.RtpOneOrder - GetLineItemRule : Unable to get rules. Rule will not be loaded. Rule type (1) [<reqres><req><LineItem-RetrieveRule><OrderId>22897426</OrderId><LineNumber>4</LineNumber><RuleKeyword>PRODUCTDATEAGE</RuleKeyword></LineItem-RetrieveRule></req><res></res></reqres>]

    2009-01-11 04:43:51,490 1 ERROR com.rtp.acropolis.rtpone.RtpOneOrder - GetLineItemRule : Unable to get rules. Rule will not be loaded. Rule type (1) [<reqres><req><LineItem-RetrieveRule><OrderId>22897426</OrderId><LineNumber>4</LineNumber><RuleKeyword>REQUIRESSHIPPING</RuleKeyword></LineItem-RetrieveRule></req><res></res></reqres>]

    2009-01-11 04:43:51,490 1 ERROR com.rtp.acropolis.rtpone.RtpOneOrder - GetLineItemRule : Unable to get rules. Rule will not be loaded. Rule type (2) [<reqres><req><LineItem-RetrieveRule><OrderId>22897426</OrderId><LineNumber>4</LineNumber><RuleKeyword>INVENTORYPOOL</RuleKeyword></LineItem-RetrieveRule></req><res></res></reqres>]

    2009-01-11 04:43:51,490 1 ERROR com.rtp.acropolis.rtpone.RtpOneOrder - GetLineItemRule : Unable to get rules. Rule will not be loaded. Rule type (2) [<reqres><req><LineItem-RetrieveRule><OrderId>22897426</OrderId><LineNumber>4</LineNumber><RuleKeyword>CHILDREGPROFILEREQUIRED</RuleKeyword></LineItem-RetrieveRule></req><res></res></reqres>]

    2009-01-11 04:43:51,490 1 ERROR com.rtp.acropolis.rtpone.RtpOneOrder - GetLineItemRule : Unable to get rules. Rule will not be loaded. Rule type (2) [<reqres><req><LineItem-RetrieveRule><OrderId>22897426</OrderId><LineNumber>4</LineNumber><RuleKeyword>ROSTER</RuleKeyword></LineItem-RetrieveRule></req><res></res></reqres>]

    2009-01-11 04:43:51,490 1 ERROR com.rtp.acropolis.rtpone.RtpOneOrder - GetLineItemRule : Unable to get rules. Rule will not be loaded. Rule type (2) [<reqres><req><LineItem-RetrieveRule><OrderId>22897426</OrderId><LineNumber>4</LineNumber><RuleKeyword>ATTACHESKISCHOOLINFORMATION</RuleKeyword></LineItem-RetrieveRule></req><res></res></reqres>]

    2009-01-11 04:43:51,490 1 ERROR com.rtp.acropolis.rtpone.RtpOneOrder - GetLineItemRule : Unable to get rules. Rule will not be loaded. Rule type (2) [<reqres><req><LineItem-RetrieveRule><OrderId>22897426</OrderId><LineNumber>4</LineNumber><RuleKeyword>PRINTATHOME</RuleKeyword></LineItem-RetrieveRule></req><res></res></reqres>]

    2009-01-11 04:43:51,490 1 ERROR com.rtp.acropolis.rtpone.RtpOneOrder - GetLineItemRule : Unable to get rules. Rule will not be loaded. Rule type (2) [<reqres><req><LineItem-RetrieveRule><OrderId>22897426</OrderId><LineNumber>4</LineNumber><RuleKeyword>RENTALPROFILEREQUIRED</RuleKeyword></LineItem-RetrieveRule></req><res></res></reqres>]

    2009-01-11 04:44:02,662 1 ERROR com.rtp.acropolis.rtpone.RtpOneInventory - AddInventoryPoolDate : <reqres><req><InventoryPool-CreateInventoryDate><OrderId>22897426</OrderId><LineNumber>10</LineNumber><ProductCode>38450</ProductCode><InventoryPoolInventoryDate>2009-02-19</InventoryPoolInventoryDate></InventoryPool-CreateInventoryDate></req><res></res></reqres>

    2009-01-11 08:16:59,647 1 ERROR com.rtp.web.ecom.eprofile.actions.Login - Unable to locate IP.

    2009-01-11 08:20:05,709 1 ERROR com.rtp.web.ecom.estore.actions.AssignCustomer - Unable to assign guest to line item: order type = order, lineItemNumber = 1

    2009-01-11 08:24:53,756 1 ERROR com.rtp.acropolis.rtpone.RtpOneCustomer - Retrieve Customer unable to retrieve customer: [<reqres><req><Customer-Retrieve><OrderId>22897495</OrderId><CustomerId>1241144</CustomerId></Customer-Retrieve></req><res></res></reqres>]

    2009-01-11 08:29:18,943 1 ERROR com.rtp.web.ecom.estore.actions.SellProduct - GetProducts: No start defined.

    2009-01-11 01:59:02,865 6 ERROR com.rtp.web.ecom.estore.actions.GetProducts - Can not determine a display category code.

    2009-01-11 02:15:32,818 6 ERROR com.rtp.web.ecom.eprofile.bl.CrmBL - GetAllProfiles: Couldn't load family members for [1238591]

    2009-01-11 07:13:59,459 6 ERROR com.rtp.web.ecom.estore.actions.GetProducts - Can not determine a display category code.

    2009-01-11 07:14:18,506 6 ERROR com.rtp.web.ecom.estore.actions.GetProducts - Can not determine a display category code.

    2009-01-11 08:29:18,959 1 FATAL com.rtp.acropolis.controller.RedirectChainCommand - Unable to execute forward [/onestore/estore/views/displayProducts.aspx].

     

     

     =================================================================================================== 

    What I would like to get is only these lines:

     =================================================================================================== 

    2009-01-11 04:44:02,662 1 ERROR com.rtp.acropolis.rtpone.RtpOneInventory - AddInventoryPoolDate : <reqres><req><InventoryPool-CreateInventoryDate><OrderId>22897426</OrderId><LineNumber>10</LineNumber><ProductCode>38450</ProductCode><InventoryPoolInventoryDate>2009-02-19</InventoryPoolInventoryDate></InventoryPool-CreateInventoryDate></req><res></res></reqres>

    2009-01-11 08:16:59,647 1 ERROR com.rtp.web.ecom.eprofile.actions.Login - Unable to locate IP.

    2009-01-11 08:20:05,709 1 ERROR com.rtp.web.ecom.estore.actions.AssignCustomer - Unable to assign guest to line item: order type = order, lineItemNumber = 1

    2009-01-11 08:24:53,756 1 ERROR com.rtp.acropolis.rtpone.RtpOneCustomer - Retrieve Customer unable to retrieve customer: [<reqres><req><Customer-Retrieve><OrderId>22897495</OrderId><CustomerId>1241144</CustomerId></Customer-Retrieve></req><res></res></reqres>]

    2009-01-11 08:29:18,943 1 ERROR com.rtp.web.ecom.estore.actions.SellProduct - GetProducts: No start defined.

    2009-01-11 01:59:02,865 6 ERROR com.rtp.web.ecom.estore.actions.GetProducts - Can not determine a display category code.

    2009-01-11 02:15:32,818 6 ERROR com.rtp.web.ecom.eprofile.bl.CrmBL - GetAllProfiles: Couldn't load family members for [1238591]

    2009-01-11 07:13:59,459 6 ERROR com.rtp.web.ecom.estore.actions.GetProducts - Can not determine a display category code.

    2009-01-11 07:14:18,506 6 ERROR com.rtp.web.ecom.estore.actions.GetProducts - Can not determine a display category code.

      ===================================================================================================

    -
    Richard Ray
    Jackson Hole Mountain Resort
    Teton Village, WY USA

  •  01-11-2009, 1:23 PM 50041 in reply to 50040

    Re: Parsing logs for errors NOT matching a list.

    Reply Quote

    With multiline enabled if matching against all lines as one string:

    Raw Match Pattern:
    ^(?=.*(\bERROR\b|FATAL))(?=.*(GetLineItemRule|Unable to execute forward)).*

  •  01-11-2009, 5:03 PM 50045 in reply to 50041

    Re: Parsing logs for errors NOT matching a list.

    Reply Quote
    ddrudik:

    With multiline enabled if matching against all lines as one string:

    Raw Match Pattern:
    ^(?=.*(\bERROR\b|FATAL))(?=.*(GetLineItemRule|Unable to execute forward)).*

    Maybe I'm misreading the question, I thought the OP was asking to exclude those lines not match them 

    Raw Match Pattern:
    ^(?=.*(?:\bERROR\b|FATAL))(?!.*(?:GetLineItemRule|Unable to execute forward)).*

    $matches Array:
    (
        [0] => Array
            (
                [0] => 2009-01-11 04:44:02,662 1 ERROR com.rtp.acropolis.rtpone.RtpOneInventory - AddInventoryPoolDate : <reqres><req><InventoryPool-CreateInventoryDate><OrderId>22897426</OrderId><LineNumber>10</LineNumber><ProductCode>38450</ProductCode><InventoryPoolInventoryDate>2009-02-19</InventoryPoolInventoryDate></InventoryPool-CreateInventoryDate></req><res></res></reqres>
                [1] => 2009-01-11 08:16:59,647 1 ERROR com.rtp.web.ecom.eprofile.actions.Login - Unable to locate IP.
                [2] => 2009-01-11 08:20:05,709 1 ERROR com.rtp.web.ecom.estore.actions.AssignCustomer - Unable to assign guest to line item: order type = order, lineItemNumber = 1
                [3] => 2009-01-11 08:24:53,756 1 ERROR com.rtp.acropolis.rtpone.RtpOneCustomer - Retrieve Customer unable to retrieve customer: [<reqres><req><Customer-Retrieve><OrderId>22897495</OrderId><CustomerId>1241144</CustomerId></Customer-Retrieve></req><res></res></reqres>]
                [4] => 2009-01-11 08:29:18,943 1 ERROR com.rtp.web.ecom.estore.actions.SellProduct - GetProducts: No start defined.
                [5] => 2009-01-11 01:59:02,865 6 ERROR com.rtp.web.ecom.estore.actions.GetProducts - Can not determine a display category code.
                [6] => 2009-01-11 02:15:32,818 6 ERROR com.rtp.web.ecom.eprofile.bl.CrmBL - GetAllProfiles: Couldn't load family members for [1238591]
                [7] => 2009-01-11 07:13:59,459 6 ERROR com.rtp.web.ecom.estore.actions.GetProducts - Can not determine a display category code.
                [8] => 2009-01-11 07:14:18,506 6 ERROR com.rtp.web.ecom.estore.actions.GetProducts - Can not determine a display category code.
            )

    )

     

    Michael

    "In theory, theory and practice are the same. In practice, they are not."
    Albert Einstein

  •  01-11-2009, 6:23 PM 50048 in reply to 50045

    Re: Parsing logs for errors NOT matching a list.

    Reply Quote

    Bingo - yes, exclude them. Now I just have to figure out why my efforts using the negative lookahead didn't work, because yours works exactly right!

     Many thanks...

     

    -
    Richard Ray
    Jackson Hole Mountain Resort
    Teton Village, WY USA

  •  01-11-2009, 6:30 PM 50049 in reply to 50048

    Re: Parsing logs for errors NOT matching a list.

    Reply Quote
    RichardRay, show your code for input as to why it didn't work.

  •  01-11-2009, 7:40 PM 50052 in reply to 50049

    Re: Parsing logs for errors NOT matching a list.

    Reply Quote

    I started off with (\S*[\d] ERROR|\S*[\d] FATAL\S*)

    That got me all the rows.

    Then I wanted to exclude at least the ones that contained 'GetLineItemRule'. I tried various things starting from the simple version:

    (\S*[\d] ERROR|\S*[\d] FATAL\S*(?!(GetLineItemRule|xxxxxxxx)))

    My problem looks like it's in the formation of the capture group in the negative lookahead. I started throwing syntax around with no success, either I was getting nothing at all back, or I was getting everything including the lines I was trying to exclude. I'm new at this regex stuff and lack fluency in thinking like the regex engine :-). It looks like you guys used a positive lookahead for the first part and then a negative lookahead for the exclusion.

    I think the .* in front of the capture group for the negative lookahead was something I hadn't gotten to - I guess that allows any string of characters to occur between the ERROR or FATAL string and the beginning of the capture. I was thinking that the \S* would get me there, but that's non-whitespace only, so it broke the on the first space character it saw after the original match and probably never even got to the capture group for the negative lookahead.

    More .*'s, and more careful formation of the capture groups, I think, 

    -
    Richard Ray
    Jackson Hole Mountain Resort
    Teton Village, WY USA
View as RSS news feed in XML
Got more questions? Find advice on: ASP | SQL | XML | Windows | About regexadvice.com | Privacy Policy | ASP.NET Tutorials | Advertise
Copyright ©2005 DevAdvice.com