Log Manager policies

Alert Logic allows you to create three types of policies in Log Manager. These policies dictate how Log Manager collects log messages and updates your software.

• A collection policy sets up rules for collection based on the type of log messages you want to collect. Log Manager runs an existing collection policy in the corresponding collection source.

• A correlation policy allows you to create a new log message when Log Manager collects a configured number of log message types during a configured time frame. Then you can set these new messages to trigger alerts.

• An updates policy allows you to schedule hosts to update to the latest version of the agent software at the agent's specified check-in.

Log Manager automatically assigns either a Windows event log or Syslog source to each host in your environment. To edit a default collection source, you must create a new Windows event log or Syslog policy. Also, to collect flat file or S3 log messages, you must create a new collection policy, and then create the corresponding collection source.

Access Log Manager policies

To access the Log Manager policies page, click CONFIGURATION, click Log Management, and then click Policies in the left navigation panel.

Work with correlation policies

A correlation policy allows you to create a new log message when Log Manager collects a configured number of log message types during a configured time frame.

Create a correlation policy

To create a correlation policy:

1. Access the Log Management Policies page and click the Correlation tab.
2. Click the Add icon ().
3. In Name, type a descriptive name.
4. Select Active to activate this policy.
5. (Optional) Select Automatically create an incident to automatically create an incident when this policy is triggered.
6. (Optional) To apply this policy to child customers, select Apply policy to child customers.
7. In the Trigger this policy when there are and messages seen within fields, type a number value. Then, select a time interval from the Display Format drop-down menu.
8. In Message Types, click the empty field and select a listed message type. You can add multiple message types.
9. In Properties and Fields, click the empty field, and then select a listed property or field. You can add multiple properties and fields.

   • From Correlate by Property or Field Value, select one or more options from the drop-down menu.

   • Click Add condition. Complete the configuration area that appears. (Optional) You can add a condition for each selection in Properties and Fields.
10. In the Log Message Template field, type a log message.

11. Select a field value and click Insert Field to insert a field in the log message template.
12. In the empty box below Insert Field, type a descriptive message to correspond with the Log Message Template message. This message can briefly describe the alert so that users can quickly read the message and understand the problem.

13. Select Automatically create an Alert to automatically create a correlation alert rule when this policy is triggered.
    • In Alert Rule Name, type a descriptive name.
    • In Time Between Alert Occurrences, type a number value expressed in minutes.
    • Select Include message text to enable the message typed in Log Message Template for this template.
14. In the Email Addresses field, select targets to receive emails, or create a new target.
15. Click Save.

Update a correlation policy

To update a correlation policy:

1. Access the Log Management Policies page and click the Correlation tab.
2. In the list of correlation policies, click the pencil icon ( ) for the correlation policy to edit.
3. Make the changes you wish to make. For information about the fields, see Create a correlation policy.
4. Click Save.

Delete a correlation policy

To delete a correlation policy:

1. Access the Log Management Policies page and click the Correlation tab.
2. Click the trash icon ( ) for the correlation policy to delete.
3. Click Delete.

Work with flat file policies

Overview

Flat file log messages (also known as text-based log messages) are a common log message format and can be collected, stored, and normalized similarly to Windows event log messages and Syslog messages. A flat file policy lets you collect flat files for Log Manager to review.

Flat file collection

Log Manager can collect flat file log messages with or without an agent.

Collect flat file log messages with the agent

To collect flat file log messages with the agent, you must create a flat file collection policy and create a flat file collection source in the Alert Logic console. After you create the flat file collection source, the collection source executes the flat file collection policy.

Collect flat files without the agent

To collect flat file log messages without the agent, you must use a physical appliance. Log Manager flat file log collector uses a CIFS file share connection from the appliance to the log file data. In cases where a CIFS file share cannot be enabled on a specific log host computer, you must copy the affected log file to a separate computer where the appliance can access a CIFS file share.

Required configurations

No industry standard exists to structure flat file log messages. As a result, log formats vary by computer device.

To enable collection and to parse these logs in the Alert Logic console, you must:

• Define the rotation schema.
  The rotation schema (or pattern) is the order that the date appears within the log message. For example, a pattern may be MM.DD.YYYY (month, day, year) or DD.MM.YYYY (day, month, year).

• Choose a single or multi-line log.
  By default, Log Manager assumes a single-line flat file log message format. For multi-line file formats, you must:
  • Define a fixed number of rows per log message.
  • Use a known pattern. This pattern can be used at the beginning, middle, or end of the log message. Also, this pattern can be a Perl Compatible Regular Expression (PCRE).
• Pick the desired time stamp method.
  Three options exist to configure the time stamp for each flat file:
  • You can choose the local time zone and settings of the log source.
  • You can choose one of several predefined rules.
  • You can create a custom time format.

Supported flat file rotation formats

For any of the rotation schemes below, Log Manager supports gzip, bzip2, and zip compressed logs:

1. YYYYMMDD (IIS Native Method):
   • Typically rotated files are given the form: <name>YYYYMMDD.log

     For example: ex20091230.log

   • Newest log file is <name>.log

     For example: ex.log

   • Files ordered based on YYYYMMDD value
2. YYYYMMDD (append method):
   • Rotated files are given the form: <name>.YYYYMMDD
   • Newest rotated log has the highest epoch time
3. Epoch Timestamp
   • Rotated logs get epoch time appended, in the form: <name>.<epoch>

     For example:  access.log.1757392910

   • Newest rotated log is one with highest epoch time
4. Incrementing Integer Method (logrotate)
   • Newest rotated log named <name>.1
   • Older logs increased in count:
     • syslog+
     • syslog.1 (newest)
     • syslog.2 (2nd newest)
     • syslog.3 (oldest)
5. Other formats

YYMM	MM-DD-YY	MM_DD_YYYY	DD_MM_YY
YYMMDD	MM.DD.YYYY	DD-MM-YY	DD-MM-YYYY
YYMMDDhh	MM.DD.YY	DD.MM.YYYY	DD_MM_YYYY
MM-DD-YYYY	MM_DD_YY	DD.MM.YY	YYYY-MM-DD
YY-MM-DD

Create a flat file collection policy

The collection policy determines which flat file log messages to collect, how to separate log messages within a flat file, and how to read the time of each log message. Also, the collection policy can specify the flat file log message collection times.

To create a flat file collection policy:

1. Access the Log Management Policies page and click the Flat-File tab.
2. Click the Add icon ().
3. In Flat File Policy Name, type a descriptive name.
4. In Source File Path, type the path information.
   

   To use an agent for collection, specify the local file system path to the log files. Otherwise, specify the network share path to the log files.

5. In File Name or Pattern, type the file name or date pattern of the flat file log messages. Log Manager can only collect flat file log messages that match the pattern.
   

   htaccess.* is a file name with a pattern. The * represents the time stamp of the flat file log. Log Manager accepts a variety of date formats.

6. From File Name Rotation Scheme, select a file name rotation scheme. The format must match the format of your flat file log messages.
   

   The default Auto-Detect identifies many rotation schemes. Alert Logic recommends you specify the rotation scheme format of your flat file log messages. If you are unsure of the format, or if you do not find the specific format from the drop-down menu, select Auto-Detect.

7. In Multi-line Handling, select a multi-line handling option:
   • If all of your flat file log messages contain a single line, keep the selection: File contains single line log messages.
   • If all of your flat file log messages do not contain a single line, select File contains log messages with multiple lines. Also, select and enter a configuration:
     • If the length of your log messages are consistent:

       Keep the selection: Each log message spans a fixed number of lines, and then in Number of lines, type the number of lines.

     • If the length of your log messages are not consistent:

       Select Each log message follows a known pattern, select the appropriate Pattern application, type the Pattern that takes place in the log message, and then if your pattern is a Perl Compatible Regular Expression (PCRE), select Regular expression.

       

       Pattern application options:

       • At the beginning of message: A line that matches the specified pattern marks the beginning of a new message; non-matching lines are lumped into the prior message.
       • In the middle of message: A line that does not match the specified pattern marks the beginning of a new message; matching lines are lumped into the prior message.
       • At the end of message: A line that matches the specified pattern marks the end of a message; non-matching lines prior to that are lumped into this message.
8. In Timestamp Rule, select a timestamp rule option:
• To use the timestamp from the collector, keep the selection: Set message time as collect time.
• To use an existing timestamp, select Parse times from messages using a pre-defined timestamp format, and then select a format from Format a date string.
• To use a custom timestamp, select Parse times from messages using a custom timestamp format, and then enter a format for the date string in the expanded configuration area. In the Format of date string field, type a format for the date string, and follow the on-screen instructions.
9. In Host Credential, select or create a credential:
   

   If you use the Alert Logic agent for log collection, do not select or create host credentials.

   • To use an existing credential, keep the default selection: Use an existing credential, and then from Existing Credential, select a credential.
   • To create a new credential, select Create a new credential, and then enter new credentials. In the corresponding configuration fields, type a Credential Name, Host Username, and Host Password. In the Retype Password field, retype the host password.
10. In Collection Schedule, select or create a collection schedule:
• To select a collection schedule, keep the default selection: Use an existing schedule, and then from Existing schedule, select a schedule.
• To create a new collection schedule, select Create a new schedule, and then enter and select the schedule options. Type a Schedule Name, select a Schedule Time Zone, and select Enabled to enable blackout periods. Also, you can add or remove extra blackout periods.
11. Click Save.

Update a flat file collection policy

To update a flat file collection policy:

1. Access the Log Management Policies page and click the Flat-File tab.
2. In the list of flat file collection policies, click the pencil icon ( ) for the collection policy to edit.
3. Make the necessary updates. For more information, see Create a flat file collection policy.
4. Click Save.

Delete a flat file collection policy

To delete a flat file collection policy:

1. Access the Log Management Policies page and click the Flat-File tab.
2. Click the trash icon ( ) for the flat file collection policy to delete.
3. Click Delete.

Work with syslog policies

Syslog is a way for network devices to send event messages to a logging server – usually known as a syslog server. A syslog policy lets you collect syslog files for Log Manager to review.

You must create a collection policy before you can create a collection source.

Create a syslog collection policy

To create a syslog collection policy:

1. Access the Log Management Policies page and click the Syslog tab.
2. Click the Add icon ().
3. In Syslog Policy Name, type a descriptive name.
4. In Syslog Listen Port, type the port where the agent receives Syslog messages.
5. In Local Syslog Cache Disk Limit (MB), type the amount of disk space you want to allow to store Syslog messages.

6. In Collection Schedule Blackout Periods, select or create a collection schedule:

   • To select a collection schedule, keep the default selection: Use an existing schedule, and then from Existing schedule, select a schedule.

   • To create a new collection schedule, select Create a new schedule, and then enter and select the schedule options. Type a Schedule Name, select a Schedule Time Zone, and select Enabled to enable blackout periods. Also, you can add or remove extra blackout periods.

7. Click Save.

Update a syslog collection policy

To update a syslog collection policy:

1. Access the Log Management Policies page and click the Syslog tab.
2. In the list of syslog collection policies, click the pencil icon ( ) for the syslog collection policy to edit.
3. Make the necessary updates. For more information, see Create a syslog collection policy.
4. Click Save.

Delete a syslog collection policy

To delete a syslog collection policy:

1. Access the Log Management Policies page and click the Syslog tab.
2. Click the trash icon () for the syslog collection policy to delete.
3. Click Delete.

Work with Windows Event Log policies

Windows event log files track significant events on a Windows server, such as user login or a program error. A Windows event log policy lets you collect event log files for Log Manager to review.

You must create a Windows event log collection policy before you set up a Windows event log collection source.

Create a Windows event log collection policy

To create a Windows event log collection policy:

1. Access the Log Management Policies page and click the Windows event log tab.
2. Click the Add icon ().
3. In Windows event log Policy Name, type a descriptive name.
4.  In Host Credential, select or create a credential:
   
   

   If you use the Alert Logic agent for log collection, do not select or create host credentials.

   • To use an existing credential, keep the default selection: Use an existing credential, and then, from Existing Credential, select a credential.
   • To create a new credential, select Create a new credential and then enter new credentials. In the corresponding configuration fields, type a Credential Name, Host Username, and Host Password. In the Retype Password field, retype the host password.

9. In Collection Schedule, select or create a collection schedule:
   • To select a collection schedule, keep the default selection: Use an existing schedule. Next, from Existing schedule, select a schedule.
   • To create a new collection schedule, select Create a new schedule, and then enter and select the schedule options. Type a Schedule Name, select a Schedule Time Zone, and select Enabled to enable blackout periods. Also, you can add or remove extra blackout periods.
10. Choose one of the following:
    • To collect all Windows event log streams, keep the default selection Collect All Available event log Streams selected.
    • To collect specific Windows event log streams, deselect Collect All Available event log Streams and select your desired streams under Alert and Collect on Selected Streams.
11. Click Save.

Update a Windows event log collection policy

To update a Windows event log collection policy: 

1. Access the Log Management Policies page and click the Windows event log tab.
2. In the list of Windows event logs, click the pencil icon ( ) for the Windows event log to edit.
3. Make the necessary updates. For more information, see Create a syslog collection policy.
4. Click Save.

Delete a Windows event log collection policy

To delete a Windows event log collection policy:

1. Access the Log Management Policies page and click the Windows event log tab.
2. Click the trash icon () for the Windows event log policy to delete.
3. Click Delete to confirm.

Work with S3 collection policies

S3 collection policies set guidelines for collecting Amazon Simple Storage Service (S3) access logs, which provide details about a single access request, such as the requester, bucket name, request time, request action, response status, and error code, if any. A S3 policy lets you collect S3 logs for Log Manager to review.

You must create an S3 collection policy before you can create an S3 collection source.

Create a S3 collection policy

To create a S3 collection policy:

1. Access the Log Management Policies page and click the S3 tab.
2. Click the Add icon ().
3. In the Name field, enter a name for the new S3 policy.
4. In Policy Template, select Customized.
5. In Multiline Handling, select a multiline handling option:
   • If all of your flat file log messages contain a single line, select File contains single line log messages.
   • If all of your flat file log messages don't contain a single line, select File contains log messages with multiple lines. Also, select and enter a configuration:
     • If the lengths of your log messages are consistent, select Each log message spans a fixed number of lines and then type the number of lines in Number of lines,.
     • If the lengths of your log messages are not consistent, select Each log message follows a known pattern, select the appropriate Pattern application, type the Pattern that takes place in the log message, and then select Regular expression to use a Perl Compatible Regular Expression (PCRE).

6. Select a Timestamp Rule option:

   • To use the timestamp from the collector, select Set message time as collect time.
   • To use an existing timestamp, select Parse times from messages using a pre-defined timestamp format, and then select a format from Format a date string.
   • To use a custom timestamp, select Parse times from messages using a custom timestamp format, and then enter a format for the date string in the expanded configuration area. In the Check Format field, type a format for the date string, and follow the on screen instructions.
7. Click Save.

Update an S3 collection policy

To update a S3 collection policy:

1. Access the Log Management Policies page and click the S3 tab.
2. In the list of S3 policies, click the pencil icon ( ) for the S3 collection policy to edit.
3. Make necessary changes to:
   • The policy Name
   • The selected Policy Template. For for information about settings for a Customized policy template, see Create a S3 collection policy.
4. Click Save.

Delete a S3 collection policy

To delete a S3 collection policy:

1. Access the Log Management Policies page and click the S3 tab.
2. Click the trash icon ( ) for the S3 collection policy to delete.
3. Click Delete.

Work with updates policies

An updates policy schedules hosts to update to the latest version of the agent software at the agent's specified check-in. By default, Alert Logic assigns the Default Update Policy, which sends software updates, as they become available, to your hosts. If the maintenance strategy for your organization requires a scheduled maintenance window, you can specify the time frame in Updates.

Create an updates policy

To create an updates policy:

1. Access the Log Management Policies page and click the Updates tab.
2. Click the Add icon ().
3. In the Updates Name field, type a descriptive name.
4. Under Updates Frequency, select one of the following:
   • Automatic
   • Scheduled
   • Never
5. Specify your update options (if any), and then click Save.

Modify an updates policy

To modify an updates policy:

1. Access the Log Management Policies page and click the Updates tab.
2. In the table of updates, click the pencil icon ( ) next to the update you want to edit.
3. Select New Updates.
4. In the Updates Name field, type a descriptive name.
5. Under Updates Frequency, select one of the following:
   • Automatic
   • Scheduled
   • Never
6. Specify your update options (if any), and then click Save.

Delete an updates policy

To delete an updates policy:

1. Access the Log Management Policies page and click the Updates tab.
2. Click the trash icon () for the updates policy to delete.
3. Click Delete.

Related topics