Encryption madness. Part II

Do you remember a story about dfc.crypto.repository parameter? Below is a continuation.

Case: I want to perform upgrade from 6.7SP2 to 7.2. What should I expect? Upgrade procedure will fail because AES became a standard in 2001, but it seems that EMC got to know about that only in 2013 and decided to break upgrade procedure:

- We happy?
- Vincent! We happy?
- Yeah, we happy.

Don’t you see anything strange? I do not understand what “crypto_mode = AES128_RSA1024_SHA256” and “crypto_mode = 3DES_RSA1024_SHA256” parameters do mean, actually, I do know what the abbreviations “AES”, “RSA”, “DES” and “SHA” do mean, but I have no idea what do they mean combined together. So, let’s check documentation:

Cool, now I know even less than I knew 5 minutes ago 😦 What is my problem? I’m too smartcurious, and I do know that 3DES and AES are symmetric ciphers, RSA is a asymmetric encryption algorithm which is primarily used for key exchange, and SHA is a MAC algorithm (message authentication code). And being brought together all these abbreviations make sense only for SSL/TLS, let me demonstrate:

Andreys-MacBook-Pro:~ apanfilov$ openssl ciphers -v
DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-DSS-AES256-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(256)  Mac=SHA1
AES256-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA1
EDH-RSA-DES-CBC3-SHA    SSLv3 Kx=DH       Au=RSA  Enc=3DES(168) Mac=SHA1
EDH-DSS-DES-CBC3-SHA    SSLv3 Kx=DH       Au=DSS  Enc=3DES(168) Mac=SHA1
DES-CBC3-SHA            SSLv3 Kx=RSA      Au=RSA  Enc=3DES(168) Mac=SHA1
DES-CBC3-MD5            SSLv2 Kx=RSA      Au=RSA  Enc=3DES(168) Mac=MD5 
DHE-RSA-AES128-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-DSS-AES128-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(128)  Mac=SHA1
AES128-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-RSA-SEED-SHA        SSLv3 Kx=DH       Au=RSA  Enc=SEED(128) Mac=SHA1
DHE-DSS-SEED-SHA        SSLv3 Kx=DH       Au=DSS  Enc=SEED(128) Mac=SHA1
SEED-SHA                SSLv3 Kx=RSA      Au=RSA  Enc=SEED(128) Mac=SHA1
RC2-CBC-MD5             SSLv2 Kx=RSA      Au=RSA  Enc=RC2(128)  Mac=MD5 
RC4-SHA                 SSLv3 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=SHA1
RC4-MD5                 SSLv3 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=MD5 
RC4-MD5                 SSLv2 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=MD5 
EDH-RSA-DES-CBC-SHA     SSLv3 Kx=DH       Au=RSA  Enc=DES(56)   Mac=SHA1
EDH-DSS-DES-CBC-SHA     SSLv3 Kx=DH       Au=DSS  Enc=DES(56)   Mac=SHA1
DES-CBC-SHA             SSLv3 Kx=RSA      Au=RSA  Enc=DES(56)   Mac=SHA1
DES-CBC-MD5             SSLv2 Kx=RSA      Au=RSA  Enc=DES(56)   Mac=MD5 
EXP-EDH-RSA-DES-CBC-SHA SSLv3 Kx=DH(512)  Au=RSA  Enc=DES(40)   Mac=SHA1 export
EXP-EDH-DSS-DES-CBC-SHA SSLv3 Kx=DH(512)  Au=DSS  Enc=DES(40)   Mac=SHA1 export
EXP-DES-CBC-SHA         SSLv3 Kx=RSA(512) Au=RSA  Enc=DES(40)   Mac=SHA1 export
EXP-RC2-CBC-MD5         SSLv3 Kx=RSA(512) Au=RSA  Enc=RC2(40)   Mac=MD5  export
EXP-RC2-CBC-MD5         SSLv2 Kx=RSA(512) Au=RSA  Enc=RC2(40)   Mac=MD5  export
EXP-RC4-MD5             SSLv3 Kx=RSA(512) Au=RSA  Enc=RC4(40)   Mac=MD5  export
EXP-RC4-MD5             SSLv2 Kx=RSA(512) Au=RSA  Enc=RC4(40)   Mac=MD5  export

Doesn’t it look familiar? So, how does the crypto_mode parameter relate to “mode based on the algorithm used to generate the AEK key”? Nohow, documentation and implementation are misleading. So, what is stored in aek.key file? Nothing but a sequence of bytes which represent either 3DES or AES secret key.

No about the problem.

I have not idea why, but in D7, when fallback settings take place, encrypttext API commands got an extremely weird behaviour:

Session id is s0
API> encrypttext,c,xxx
...
DM_ENCR_TEXT_V2=AAAACHYQW2Ab8FTGW8gul3tK6Q8M+9RKuRSGgxypEcVmqaJLt2JE0+7tuKzVQzh78QTCxS6gcWAq7sOx
API> encrypttext,c,xxx
...
DM_ENCR_TEXT=W8gul3tK6Q8M+9RKuRSGgzw1veDUWIeDnWK6wT7gFEaVCVEGGORiHu/uzJNPhsAh

moreover:

~]$ iapi -X -Sapi
Running with non-standard init level: api
API> encrypttext,a,xxx
...
DM_ENCR_TEXT_V2=AAAACHYQW2Ab8FTGW8gul3tK6Q8M+9RKuRSGg63EjkDYL8tY2+Ox0El+nLbK+UgeDk0lAKELAnUx2Rnu
API> encrypttext,a,xxx
...
DM_ENCR_TEXT=W8gul3tK6Q8M+9RKuRSGg5J4DiemgxHojpL7UY6GbwClg7osvnn1GTnEmC672QgY
API> 

What does such behaviour mean? Now, you are unable to set ldap password from DA: when setting ldap password from DA, it executes replicate_setup_methods docbase method (something like “execute do_method with method=’replicate_setup_methods’,arguments=’mkfile_encrypt_text password /u01/documentum/dba/config/DCMT_DEV/ldap_08002b8f801ccabb.cnt'”), this method executes encrypttext API command and gets wrong password.

A FATAL error has occurred

Have you ever seen such error:

?

I believe everybody who tried to deploy Documentum in large enterprise had faced with such spontaneous errors but never paid much attention because error message is completely misleading: “[DM_SESSION_E_AUTH_FAIL]error: “Authentication failed for user” – dumb user enters invalid password, so it’s not our issue. But actually, this error reveals a lot of problem related to session management in Documentum.

Root cause

If take a careful look at stacktrace it becomes clear that the error originates not from login page but somewhere from WDK (put any other application here) intestines:

at com.documentum.web.formext.privilege.PrivilegeService.getUserPrivilege(PrivilegeService.java:57)
at com.documentum.web.formext.config.PrivilegeQualifier.getScopeValue(PrivilegeQualifier.java:69)
at com.documentum.web.formext.config.BoundedContextCache.retrieveQualifierScopeValue(BoundedContextCache.java:153)
at com.documentum.web.formext.config.ScopeKey.<init>(ScopeKey.java:57)
at com.documentum.web.formext.config.ConfigService.makeScopeKey(ConfigService.java:1482)
at com.documentum.web.formext.config.ConfigService.lookupElement(ConfigService.java:527)

this fact means that user was already successfully logged in and this is, obviously, not a user’s mistake. So, what does really happen there? The basic description of the problem is: user’s dfc session got either reused by another user or released by application and when application tries to acquire new dfc session it fails. So, the first question is why application fails to acquire new dfc session. I believe there are a lot of reasons, but the most common for large enterprises is following: ldap authentication in Documentum is unstable: most time it works as expected but sometimes it fails and causes a hardly diagnosable issues.

Mitigation options

  1. disable D7 session pooling in dfc (i.e. set dfc.compatibility.useD7SessionPooling to false) – the most of customers noticed that error has started occurring more frequently after moving to new version of dfc, actually, it is a true, because new pooling implementation tends to keep the amount of dfc sessions as small as possible, so the amount of authentication requests increases
  2. if you use bind_type=bind_search_dn in ldap config switch to bind_by_dn – it will decrease the amount of ldap round-trips
  3. use as nearest ldap servers as possible
  4. put a blame onto EMC – authentication is not a kind of thing which must occur every 5 seconds due to poor application design