Siebel Administration >  Audit Trail does not work sometimes

If some operations do not show in the audit trail, it could be because
of a defect in version 8.x in Siebel. This is what Siebel says about this bug

Applies to:
Siebel Audit Trail - Version 8.0 [20405] and later
Siebel CRM - Version 8.0 [20405] and later
Information in this document applies to any platform.
Description
Audit Trail stops working for a period of time.
Audit Records cease being recorded for one or more users in an environment.

Users may experience a situation with Audit Trail where:
No error messages are reported in the Siebel logs.
No audit records are captured during the time when Audit Trail has stopped.
The error will be noted as being resolved when the Enterprise is restarted;
however, the behavior may repeat later.
Occurrence
This problem occurs after Audit Trail has been functioning correctly,
without any change(s) having been made to the Audit configuration.

Internally there are two levels of cache for Audit:
 Process level
 Session level.

When the first user logs in to the application, the Process level cache is loaded;
the Session level cached is then filled on an "as needed" basis.

As part of this basis, the cache version number is checked to determine whether the cache needs to be reloaded.
In the case of this specific issue, while checking for the cache version number, a database connection error occurs.
On receiving this error, the Process global cache and Session level cache are cleared.

Internally, however, the version number and flag, which indicates the cache status, are not cleared.

The next time a user tries to access the Audit cache, by performing some operation that
would cause an auditable event, the information is returned from an empty cache, as if nothing has been enabled for Audit.

The problem will then exist for all users connecting to the same Object Manager process, unless a workaround is invoked.
Symptoms
Most users only experience this issue in their Production environment.

The reason for this problem being limited to Production is primarily due to volume.
This issue will only occur if a database connection error happens during a very small code window,
giving a low likelihood of occurrence for each individual transaction.

Highly stressed, high volume systems are much more likely to hit the combination of instructions needed to provoke the issue.

There is no single consistent log message that can be leveraged to indicate this problem has been encountered.
Workaround
To clear this error condition, the Audit cache can be explicitly set to clear.

Note the following three workarounds:
1. Press the Update Audit Cache button on the Audit Administration screen.
2. Call the UpdateAuditCache method of the Audit Trail Configuration Service Business Service.
3. Execute an automated script, as documented in the following::
Automation To Resolve Missing Audit Trail problem Document 1350649.1

Workaround 1:
Periodically, manually press the "Update Audit Cache" button,
located on the 'Administration - Audit Trail' screen. In most cases, an Administrative user performs
this function since "standard" users would generally not have access to the Audit Trail Administration view.

Steps:
1. Go to Site Map > Administration - Audit Trail.
2. Press the "Update Audit Cache" button.

***  Caution * **  Workaround 2 can cause database contention problems when used in a high volume Production environment.

Workaround 2:  DO NOT USE THIS WORKAROUND IN AN ENVIRONMENT WITH MANY USERS LOGGING INTO THE APPLICATION
Call the UpdateAuditCache method of the Audit Trail Configuration Service Business Service during login.

Steps:
1. Create a new Action Set to call the Business Service Audit Trail Configuration Service with method UpdateAuditCache:
a. Go to Site Map > Administration - Runtime Events > Action Sets
b. Select the 'New' button, entering the following:
Action Type=BusService
Business Service Name=Audit Trail Configuration Service
Business Service Method=UpdateAuditCache.

2.  Create a new Runtime Event to execute on Application Login:
a. Go to Administration - Runtime Events > Events
b. Select the 'New; button, entering the following:
Object Type=Application
Object Name=Object Manager (Example: Siebel Financial Services)
Event=Login
Action Set Name=Action Name created in Step A.

3.  Reload Runtime Events.

4. To Test:
a. Enable client side logging in a test environment.
b. Check the log for the "Audit Trail Configuration Service, UpdateAuditCache Method Execute as part of the login process.
c. Repeat the same test with logging enabled on your Object Manager(s) in test.
d. You should also enable logging on your Workflow Process Manager component to test the impact of this on workflow processes executed.

Workaround 3:
Implement the script that automates the updating of the cache, as documented in the following:
Automation To Resolve Missing Audit Trail problem Document 1350649.1

Result:
If any of these workaround methods is invoked, the cache version number on the database
will increment and global cache flag will be set to dirty. When a user subsequently tries
to invoke an auditable operation, the Process level global cache is emptied.

Both Process level and Session Level caches are then filled with the next request to access the Audit cache.

This workaround of updating the cache has been shown effective, regardless of the method used to invoke it.
The required frequency appears to be dependent on the customer's implementation, but generally ranges between 4 and 24 hours.

Patches
The fix changes the behavior by explicitly resetting the version number and setting the audit flag to dirty,
in the case of a database error. Thus, the next time a user requires access to the Audit cache, it will be reloaded.

The fix has been built into base Siebel version 8.2.2.
Fixes are also available under the following fix pack versions/patches:

Bug:11721518 Version 8.1.1.5 Patch 12545956
Bug:11734994 Version 8.0.0.12 Patch 12884885

This issue is fixed in the 8.1.1.5 and 8.0.0.12 Fix Packs.

If you are using the Siebel Audit Trail, make plans to install the appropriate Fix Pack for your release. 

Automating this process:

Automation:

Here is the script example to invoke "UpdateAuditCache" method of "Audit Trail Configuration Service":
var BS = TheApplication().GetService("Audit Trail Configuration Service");
var oIn = TheApplication().NewPropertySet();
var oOut = TheApplication().NewPropertySet();

BS.InvokeMethod("UpdateAuditCache", oIn, oOut);

Work Flow:
************
To invoke the business service from workflow process, please add a business service step in the process and set the following:

Business Service Name: Audit Trail Configuration Service
Business Service Method: UpdateAuditCache

Repeating Component Request:
****************************
Invoke the above Work flow through RCR
Schedule this RCR Daily or weekly

If you need assistance on creating repeating component,
please refer to Siebel System Administration Guide > Administering Siebel Server Run-Time Operations >
Administrating Component Jobs > Starting a Repeating Component Job.
Automation of Audit Trail Problem : Audit Trail Stops Tracking Auditable Operations Intermittently