Encryption madness

My colleagues got faced with ridiculous security settings implemented by EMC, so I decided to write this blogpost. Starting from Documentum 7.2 EMC decided to switch from resolving security issues to treating symptoms (first attempt, second attempt). The security issue is: if attacker is able to hijack aek.key file from Content Server filesystem he is able to get superuser access to repository (the basic scenario is: we can decrypt i_ticket_crypto_key from docbase config using aek.key and after that we can issue any login ticket). To “mitigate” such attack EMC proposes two options:

As we already know, both options have nothing in common with security, but, as it turned out, cause a lot of pain in behind. The problem is: if you protect aek.key by passphrase or use lockbox, DFC is unable to use aek.key anymore to decrypt passwords (the basic example is: the file with password for ldap server stored in config directory on Content Server, ldap synchronisation job needs to decrypt this password), so it is required to invent another approach for decrypting passwords on DFC side, but EMC did something weird. Starting from 7.2 DFC following behaviour takes place:

  • installer adds dfc.crypto.repository parameter into dfc.properties (it is worth to mention, that installer does it in wrong way: installation of second repository overrides this parameter)
  • when encrypting text/password, if DFC sees dfc.crypto.repository parameter in dfc.properties it tries to establish trusted session to this repository to call ENCRYPT_TEXT/ENCRYPT_PASSWORD RPC command and adds repository name to the end of encrypted text/password:
    Connected to Documentum Server running Release 7.2.0060.0222  Linux64.Oracle
    Session id is s0
    API> encryptpass,c,mycoolpassword
    ...
    DM_ENCR_PASS_V2=AAAAEKfc...RYjoGt::DCTM_DEV
    API> encrypttext,c,mycoolpassword
    ...
    DM_ENCR_TEXT_V2=AAAAELQ...O+x5g::DCTM_DEV
    
  • when decrypting text, if DFC sees dfc.crypto.repository parameter in dfc.properties it tries to establish trusted session to this repository or, in case of failure, to repository mentioned at the end of encrypted text
  • when decrypting password DFC uses old technique based on reading aek.key file

I believe, the behaviour described above is extremely logical, unfortunately, I’m unable to understand this logic. For example, let imagine that I installed two repositories, in this case dfc.crypto.repository points to the second one. Now I’m going to setup LDAP synchronisation, in order to encrypt ldap password Documentum Administrator calls replicate_setup_methods docbase method, and passwords will be encrypted using the second repository, now, what will happen if second repository goes down? It’s obvious that ldap synchronisation will stop working for the first repository – DFC is unable to connect to the second repository to decrypt password (actually, all DFC encryption/decryption operations on CS side will fail because repository configured in dfc.properties is down). Also, I have noticed that some users (and my colleagues too) have the imprudence to copy dfc.properties file from Content Server to Application Server, after that some things stop working because copied dfc.properties file contains dfc.crypto.repository parameter, but Application Server has no trusted access to repository, below are some examples from ECN: