| Siebel Administration > Async Requests Are Crashing In Prod Daily with errors SBL-OSD-02006, SBL-SRB-00061
Async Requests Are Crashing In Prod Daily with errors SBL-OSD-02006, SBL-SRB-00061
APPLIES TO:
Siebel Workflow - Version 15.6 [IP2015] and later
Information in this document applies to any platform.
SYMPTOMS
On : 15.6 [IP2015] version, System Admin
Few of the asynchronous requests are failing in prod daily with below error message
SBL-SRB-00061: Process WfProcMgr on Siebel Server PRODCRM22 terminated.
The requests are getting successful on resubmitting .
ERROR
SBL-SRB-00061
SBL-OSD-02006
Call stack:
PROCESS 3218 CRASHED AFTER RECEIVING SIGNAL 11 IN THREAD 35183472!!!
Current Time: 03/20/2018 16:04:16.467
/siebelcrm/Siebel/siebsrvr/lib/libsscfom.so(_ZN8CSSModel15TriggerCTEventsE12CTObjectTypePKDsS2_S2_P10CSSBusComptRK14CCFPropertySet+0x6f)[0x1694b6f]
/siebelcrm/Siebel/siebsrvr/lib/libsscfom.so(_ZN10CSSBusComp12RuntimeEventEijRK8SSstringRK14CCFPropertySet+0x124)[0x16d4c34]
/siebelcrm/Siebel/siebsrvr/lib/libsscfom.so(_ZN10CSSBusComp12RuntimeEventEijRK8SSstring+0x6f)[0x16d500f]
/siebelcrm/Siebel/siebsrvr/lib/libsscfom.so(_ZN10CSSBusComp12RuntimeEventEij+0x4c)[0x16d508c]
/siebelcrm/Siebel/siebsrvr/lib/libsscfom.so(_ZN10CSSBusComp16EventWriteRecordEi+0x58)[0x16d6b68]
/siebelcrm/Siebel/siebsrvr/lib/libsscfom.so(_ZN10CSSBusComp11WriteRecordEbP13CSSObjectBase+0x3aa1)[0x16fa9b1]
FDR:
225117 1521561856 35183472 Fdr_OBJMGR Object Manager FdrSub_OBJMGR_SQLOBJ_DSTR SqlObj Destruction -757637432 0 EQX WOF Action
225118 1521561856 35183472 Fdr_SCRIPT Generic Scripting Engine FdrSub_SCRIPT_FREE Object Deallocation -549180512 -750451160
225119 1521561856 35183472 Fdr_OBJMGR Object Manager FdrSub_OBJMGR_CACHE_QUERY Cache Query Parse Tree -572988924 0 :"
225120 1521561856 35183472 Fdr_FDR Fdr Internal FdrSub_FDR_CRASH ** CRASHING THREAD ** 0 0
CAUSE
The call stack is related to runtime event triggered by WriteRecord.
It is shown in the WfProcMgr log that workflow step 'Update Order POF Status' of workflow process 'EQX OF Update Sales Order POF Status Sub WF' there are the following SQLs for changing data which is WriteRecord:
1st is UPDATE SIEBEL.S_ORDER_ITEM SET STATUS_CD = Closed
2nd is INSERT INTO SIEBEL.S_AUDIT_ITEM
After the SQLs, the log shows runtime event WriteRecord with EventId = '1-1B51OEI4' was triggered. The action for the runtime event is to execute the workflow process 'EQX Z Side Closed Notification Wrapper WF'. Here is extracted from the log:
ObjMgrBusServiceLog InvokeMethod 4 00006d2c5a93da6b:0 2018-02-26 23:46:57 Begin: Business Service 'Workflow Process Manager' invoke method: 'RunProcess' at 9eca6f50
EngInv EngInv 3 00006d2c5a93da6b:0 2018-02-26 23:46:57 Workflow engine requested to perform method 'RunProcess'.
EngInv Arg 4 00006d2c5a93da6b:0 2018-02-26 23:46:57 Input: @0*0*4*0*0*0*7*EventId10*1-1B51OEI49*Sub Event0*11*ProcessName41*EQX Z Side Closed Notification Wrapper WF10*Event Name11*WriteRecord
PrcExec PrcExec 3 00006d2c5a93da6b:0 2018-02-26 23:46:57 Executing process definition 'EQX Z Side Closed Notification Wrapper WF'.
From workflow process 'EQX Z Side Closed Notification Wrapper WF', there are 12 other workflow processes got executed:
PrcExec Create 4 00006d2c5a93da6b:0 2018-02-26 23:46:57 Instantiating process definition 'EQX Z Side Closed Notification WF'.
PrcExec Create 4 00006d2c5a93da6b:0 2018-02-26 23:46:59 Instantiating process definition 'EQX Get IBX Lang codes for Sales Order'.
PrcExec Create 4 00006d2c5a93da6b:0 2018-02-26 23:46:59 Instantiating process definition 'EQX BIP Generic Email Notification Framework'.
PrcExec Create 4 00006d2c5a93da6b:0 2018-02-26 23:46:59 Instantiating process definition 'EQX Get Notification Framework Details WF'.
PrcExec Create 4 00006d2c5a93da6b:0 2018-02-26 23:46:59 Instantiating process definition 'EQX OM Get Contact List'.
PrcExec Create 4 00006d2c5a93da6b:0 2018-02-26 23:46:59 Instantiating process definition 'EQX OM Notifications Multi Lang Code Update'.
PrcExec Create 4 00006d2c5a93da6b:0 2018-02-26 23:46:59 Instantiating process definition 'EQX BIP Send EMail Async'.
PrcExec Create 4 00006d2c5a93da6b:0 2018-02-26 23:46:59 Instantiating process definition 'BIP Report Generation'.
PrcExec Create 4 00006d2c5a93da6b:0 2018-02-26 23:46:59 Instantiating process definition 'BIP Create Report Output'.
PrcExec Create 4 00006d2c5a93da6b:0 2018-02-26 23:47:25 Instantiating process definition 'EQX OCM BIP Notification'.
PrcExec Create 4 00006d2c5a93da6b:0 2018-02-26 23:47:25 Instantiating process definition 'EQX BIP Read File to String'.
PrcExec Create 4 000005615a92e50f:0 2018-02-26 23:47:25 Instantiating process definition 'EQX BIP Create Email Activity Attachment'.
And after workflow process 'EQX Z Side Closed Notification WF' execution was finished, the crash happened, and the log stopped there. So executing of the workflow process 'EQX Z Side Closed Notification Wrapper WF' was not completed, and it is the workflow process the runtime event set up to execute. .
SOLUTION
Suggest to look into the runtime event EventId = '1-1B51OEI4', if it should be inactivated, and workflow processes being executed by runtime event if they should stay as enabled.
To help examine this runtime event and workflow processes, please review the following Doc that is based on one example that you can use to work on yours:
Workflow Process Error SBL-BPR-00158 In Upgraded Siebel 8.1.1.10 (Doc ID 1667957.1)
Suggest to look into 13 workflow processes being executed related to this runtime event, and verify if they exist and if they should stay active. If they should not stay active, then they should be inactivated. The above Doc explains how you can look up these workflow processes in Administration - Business Process.
If the runtime event should be inactivated, then inactivate it. The above Doc also covers this part.
If you have another environment that is production like environment with production data, then test the solution of inactivating runtime event action(s) and workflow processes in non-production environment first before deploying the solution to production.
| 
