Security through guessing

About three weeks ago I made a decision to stop posting about security vulnerabilities in Documentum, but on Friday I have faced with amusing behaviour of webtop and I was unable to leave that fact without blogpost. Nine months ago I wrote a post about how EMC fails to read documentation. Actually, I was never considering the ability to read webtop’s configuration files through HTTP requests as vulnerability because I always follow my own best practices and never put environment-specific configuration into web application, unfortunately this point is not obvious for some developers and we may get something like:

What did cause me to write this post? On Friday I was trying to merge some changes implemented in webtop 6.8 to customised webtop 6.7 and I had noticed new weird changes in web.xml:

   <context-param>
      <param-name>StaticPageIncludes</param-name>
      <param-value><![CDATA[(\.bmp|\.css|\.htm|\.html|\.gif|\.jar|\.jpeg|\.jpg|\.js|\.properties|\.xml|\.xml;|\.png)$]]></param-value>
   </context-param>

note, that EMC added “\.xml;” to protect config files from reading:

The problem is EMC still fails to read documentation:

Silent UCF. HowTo

As was promised in previous post, I describe how make webtop customers happy.

What is the problem?

It seems that when Oracle acquired Sun it also acquired a bunch of security issues in JRE, for example, it’s pretty obvious that even if applet is signed by valid certificate it’s not enough to consider it as trusted – benign applet could be potentially used for malicious purposes (for example UCF applet is capable to launch any program on user’s machine, so attacker may exploit this capability). Security measures for this issue are obvious: just bind applet to specific website, and Oracle did it: JAR File Manifest Attributes for Security. Unfortunately these “innovations” are barely suitable for self-hosted applications – URLs vary from customer to customer, which makes it impossible to create applet suitable for every customer, though, I suppose EMC’s customers pay enough support fees to have an ability to request “custom” UCF applet from EMC, “custom” means that only manifest changes are required and such procedure could be easily automated – why do not create special service on customer portal? Instead of doing correct things EMC suggests something weird:

Actually, I double-checked Oracle’s documentation and was unable to find a place where they suggest to delegate security-related questions to business users, I suppose that EMC guys think that some posts in forums/blogs/etc have a status of official documentation, that, in fact, is not surprising: for example, IIG’s director of security prefers to get inspiration from this blog instead of performing code review:

Possible solutions

Anyway, Oracle’s suggestions related to security prompts are:

Though rule sets option is straightforward I do not think that it is helpful due to following reasons:

  • it costs money: “The JAR file must be signed with a valid certificate from a trusted certificate authority”
  • it is dangerous: “The Deployment Rule Set feature is optional and shall only be used internally in an organization with a controlled environment. If a JAR file that contains a rule set is distributed or made available publicly, then the certificate used to sign the rule set will be blacklisted and blocked in Java”, i.e. you risk to loose money spent for your certificate
  • if you already have a “valid certificate from a trusted certificate authority” why do not sign all applets in enterprise? So, this option is more suitable for cases when applets are embedded in hardware devices like KVMs, iLOs, etc and you unable to replace those applets
  • I think the best case for rule sets is enable certain known applets and block all other

Second option costs money too because it requires a “valid certificate from a trusted certificate authority”, but it’s the only disadvantage of this option. So, what did I do to disable all security prompts in previous video? Obviously, at first, we bought code-signing certificate.

Solution

Let’s take a look at security prompts raising when UCF applet gets loaded:

  • Java’s standard prompt asking whether user is going to trust applet’s certificate:
  • UCF’s prompt aimed to prevent malicious usage of applet (whitelisting):
  • Java’s prompt about not following security best practices:

For the third option EMC, as we already know, suggests to tick “Do not show this again for this app and web site” checkbox before accepting security warning. Besides the fact that delegating security-related questions to end-users is not a good idea, this options does not work properly. The problem is WDK generates special URLs for applets to prevent them from caching by JRE:

here “19gmgko28” and “98th” parts of URL are just encoded values of last modified time and size of applet file:

-bash-4.1$ stat -c "Last Modified: %Y, size: %s" \
> app/webapps/wp/wdk/system/ucfinit.jar
Last Modified: 1426684797, size: 304049
-bash-4.1$ groovysh
Groovy Shell (2.4.1, JVM: 1.6.0_45)
Type ':help' or ':h' for help.
groovy:000> import com.documentum.web.util.StringUtil
===> com.documentum.web.util.StringUtil
groovy:000> StringUtil.toUnsignedString(1426684797000,5)
===> 19gmgko28
groovy:000> StringUtil.toUnsignedString(304049,5)
===> 98th

Such behaviour of WDK has following disadvantages:

  • redeploying/updating webtop changes last modified time or/and size of applet file
  • in clustered environment it’s expected that applet file has different last modified time on different nodes

also we already know that the cornerstone of UCF troubleshooting is to delete everything related to Java and Documentum, reinstall JRE, press ctrl+alt+reset, etc – these activities obviously wipe user’s settings, it’s clear that ticking “Do not show this again for this app and web site” checkbox is just a temporary workaround, so, we decided to sign applet by our certificate. The procedure is straightforward:

  • edit META-INF/MANIFEST.MF, you should get something like (note Caller-Allowable-Codebase, Application-Library-Allowable-Codebase and Codebase headers):
    Manifest-Version: 1.0
    Built-By: dmadmin
    Application-Name: Documentum
    Created-By: 1.5.0_11-b03 (Sun Microsystems Inc.)
    Copyright: Documentum Inc. 2001, 2004
    Caller-Allowable-Codebase: docum-at-app
    Build-Date: January 17 2015 09:07 PM
    Ant-Version: Apache Ant 1.8.4
    Title: Documentum Client File Selector Applet
    Application-Library-Allowable-Codebase: docum-at-app
    Bundle-Version: 6.7.2220.0231
    Build-Version: 6.7.2220.0231
    Permissions: all-permissions
    Codebase: docum-at-app
    
  • remove old signature files, i.e. META-INF/DOCUMENT.RSA and META-INF/DOCUMENT.SF
  • run jarsigner to sign applet – do not forget to specify TSA URL

after that applet is ready for deployment in production. What about another two security prompts? I suppose it is obvious that whitelisting capability is now useless because now JRE performs the same checks, below is an example of security notice when applet gets loaded from non-trusted URL:

because of these considerations, I decided to disable whitelisting capability, unfortunately the only option to disable it in UCF is decompile com.documentum.ucf.client.install.installer.security.impl.Whitelist class: methods isHostAllowed(String) and isHostAllowed(String, IHostVerifier) must always return true (I bet it doesn’t worth to mention that after replacing class in applet you must sign applet again).

And finally, to remove the first security prompt I read Deployment Configuration File and Properties article and decided that manipulating by User Level configuration files is not a good idea, but System Level configuration files could be easily distributed among users’ machines in enterprise using various deployment tools, so, I did following (note naming convention of certificate alias):

C:\Windows\system32>md C:\Windows\Sun\Java\Deployment

C:\Windows\system32>cd C:\Windows\Sun\Java\Deployment

C:\Windows\Sun\Java\Deployment>type CON > deployment.config
deployment.system.config=file\:C\:/Windows/Sun/Java/Deployment/deployment.properties

C:\Windows\Sun\Java\Deployment>type CON > deployment.properties
deployment.system.security.trusted.certs=C\:\\Windows\\Sun\\Java\\Deployment\\trusted.certs

C:\Windows\Sun\Java\Deployment>keytool.exe -importcert -keystore trusted.certs -file G:\Users\andrey\work\cert.crt -alias deploymentusercert$tsflag$loc=http//docum-at-app:8280##docbase:http//docum-at-app:8280##from:http//docum-at-app:8280
Enter keystore password:
Re-enter new password:

...

Trust this certificate? [no]:  yes
Certificate was added to keystore

Voilà.

Security enhancements in Webtop 6.8

CVE-2014-4637 quote:

Open redirect vulnerability in EMC Documentum Web Development Kit (WDK) before 6.8 allows remote attackers to redirect users to arbitrary web sites and conduct phishing attacks via an unspecified parameter.

Real behaviour of webtop 6.8 (note how it sends login ticket to remote site):

CVE-2014-4636 quote:

Cross-site request forgery (CSRF) vulnerability in EMC Documentum Web Development Kit (WDK) before 6.8 allows remote attackers to hijack the authentication of arbitrary users for requests that perform Docbase operations.

Real behaviour of webtop 6.8:

Power Pivot

To investigate performance problems in WDK applications I typically use capabilities of com.documentum.web.env.IFormRenderListener interface, the code below demonstrates a basic idea – when page starts rendering I put date into form return value and when page rendering gets finished I gather and log required information:

/**
 * @author Andrey B. Panfilov <andrew@panfilov.tel>
 */
public class FormRenderLogger implements IFormRenderListener {

    private static final String FORM_RENDER_LOGGER_START_DATE = "__FORM_RENDER_LOGGER_START_DATE__";

    public FormRenderLogger() {
        super();
    }

    public void notifyFormRenderStart(Form form) {
        form.setReturnValue(FORM_RENDER_LOGGER_START_DATE, new Date());
    }

    public void notifyFormRenderFinish(Form form) {
        Date startDate = (Date) form
                .getReturnValue(FORM_RENDER_LOGGER_START_DATE);
        form.removeReturnValue(FORM_RENDER_LOGGER_START_DATE);
        long diff = new Date().getTime() - startDate.getTime();
        String containerId = null;
        String componentId = null;
        ArgumentList initArgs = null;
        Context context = null;
        if (form instanceof Container) {
            containerId = ((Container) form).getId();
            componentId = ((Container) form).getContainedComponentId();
            initArgs = ((Container) form).getInitArgs();
            context = ((Container) form).getContext();
        } else if (form instanceof Component) {
            componentId = ((Component) form).getComponentId();
            initArgs = ((Component) form).getInitArgs();
            context = ((Component) form).getContext();
        }
        String userName = null;
        try {
            if (ComponentDispatcher.isRepositoryAccessRequiredComponent(form
                    .getId())) {
                IDfSessionManager sessionManager = SessionManagerHttpBinding
                        .getSessionManager();
                if (sessionManager != null
                        && SessionManagerHttpBinding.getCurrentDocbase() != null
                        && sessionManager.hasIdentity(SessionManagerHttpBinding
                                .getCurrentDocbase())
                        && form instanceof Component) {
                    userName = ((Component) form).getDfSession()
                            .getLoginUserName();
                }
            }
        } catch (DfException ex) {
            throw new WrapperRuntimeException(ex);
        }
        StringBuilder message = new StringBuilder(50);
        if (containerId != null) {
            message.append("Container: ").append(containerId).append(", ");
        }
        if (componentId != null) {
            message.append("Component: ").append(componentId).append(", ");
        }
        if (userName != null) {
            message.append("User: ").append(userName).append(", ");
        }
        if (initArgs != null) {
            message.append("InitArgs: ").append(initArgs).append(", ");
        }
        if (context != null) {
            message.append("Context: ").append(context).append(", ");
        }
        message.append("Remote IP: ")
                .append(form.getPageContext().getRequest().getRemoteHost())
                .append(", ");
        message.append("Render time: ").append(diff).append("ms");
        DfLogger.debug(this, message.toString().replaceAll("\\{", "[")
                .replaceAll("\\}", "]"), null, null);
    }

}

app.xml:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<config>
 <scope>
  <application extends="webtop/app.xml">
   ...
   <listeners>
    ...
    <formrender-listeners>
     <listener>
      ...
      <class>FormRenderLogger</class>
     </listener>
    </formrender-listeners>
   </listeners>
  </application>
 </scope>
</config>

Something similar was described earlier by Stephane Marcon in Documentum Monitoring – platform logs’ centralization using Logfaces – Part 2, but the problem is logs accuracy of webtop logs collected by Stephane completely suck. Though, code, given above, does not provide information about delays caused by wdk actions, you can take advantage of IFormRenderListener, IApplicationListener and IRequestListener interfaces to implement more robust solution (see example of usage in Dynamic groups. Advances. Part IV).

But yesterday I have faced with a weird problem. Typically I parse performance logs into tab-delimited file, then load this file into excel and build some reports to figure out what wdk components are slow. But yesterday I have found that customer’s daily logs contain more than 3 million rows and Excel does not support such a amount of data. What to do? Power Pivot comes to the rescue!

How to provide backward compatibility with old java clients during UCF updates

This post was originally published on ECN.

If you are not familiar with apache httpd’s mod_rewrite and regexps, please do not read this post.

Problem

When EMC releases new UCF fixes, aimed to provide compatibility with corresponding JRE security fixes, they completely miss the fact, that customers are unable to perform JRE updates for all business users at the same time, and, so, IT personal should choose the lesser of two evils: either some users should suffer or all users should stay with vulnerable JRE, moreover, sometimes I can’t understand what EMC is doing (see also: Dumb UCF applet, Dumb UCF applet. Part II, UCF applet’s certificate expired… ORLY?)

Solution

At first we need to understand what files EMC changes from one point fix to another (it’s very simple for me because I download every new pointfix to check that EMC still not fixed 30 security issues ). Typically they change wdk/system/ucfinit.jar and wdk/fileselector/fileSelector.jar files, but ucfinit.jar contains checksums for some files in wdk/contentXfer directory, so, ucfinit.jar file in general is not interchangeable between point fixes. Major changes in webtop 6.7SP2 was (actually I double checked 1.7_45 and 1.7_25 versions I can certainly say that patch notes lie: for both versions it’s required to relax security settings to get working UCF):

  • P11 – JRE1.7_51 support
  • P07 – JRE1.7_45 support
  • P05 – JRE1.7_25 support
  • P02 – JRE1.7_21 support

that means that following filesets are consistent:

  • wdk/system/ucfinit.jar, wdk/fileselector/fileSelector.jar, wdk/contentXfer from P11 for JRE1.7_51
  • wdk/system/ucfinit.jar, wdk/fileselector/fileSelector.jar, wdk/contentXfer from P10 for JRE1.7_45
  • wdk/system/ucfinit.jar, wdk/fileselector/fileSelector.jar, wdk/contentXfer from P06 for JRE1.7_25
  • wdk/system/ucfinit.jar, wdk/fileselector/fileSelector.jar, wdk/contentXfer from P04 for JRE1.7_21

for filesets 2-4 I created following structure inside wdk directory:

JRE17_21  
├── contentXfer  
│  ├── All-MB.jar  
│  ├── ES1_MRE.exe  
│  ├── ExJNIAPI.dll  
│  ├── ExJNIAPIGateway.jar  
│  ├── jacob.dll  
│  ├── jacob.jar  
│  ├── libMacOSXForkerIO.jnilib  
│  ├── MacOSXForker.jar  
│  ├── mac_utilities.jar  
│  ├── ucf-ca-office-auto.jar  
│  ├── ucf-client-installer.zip  
│  └── UCFWin32JNI.dll  
├── fileselector  
│  └── fileSelector.jar  
└── system  
    └── ucfinit.jar  
  
JRE17_25  
├── contentXfer  
│  ├── All-MB.jar  
│  ├── ES1_MRE.exe  
│  ├── ExJNIAPI.dll  
│  ├── ExJNIAPIGateway.jar  
│  ├── jacob.dll  
│  ├── jacob.jar  
│  ├── libMacOSXForkerIO.jnilib  
│  ├── MacOSXForker.jar  
│  ├── mac_utilities.jar  
│  ├── ucf-ca-office-auto.jar  
│  ├── ucf-client-installer.zip  
│  └── UCFWin32JNI.dll  
├── fileselector  
│  └── fileSelector.jar  
└── system  
    └── ucfinit.jar  
  
JRE17_45  
├── contentXfer  
│  ├── All-MB.jar  
│  ├── ES1_MRE.exe  
│  ├── ExJNIAPI.dll  
│  ├── ExJNIAPIGateway.jar  
│  ├── jacob.dll  
│  ├── jacob.jar  
│  ├── libMacOSXForkerIO.jnilib  
│  ├── MacOSXForker.jar  
│  ├── mac_utilities.jar  
│  ├── ucf-ca-office-auto.jar  
│  ├── ucf-client-installer.zip  
│  └── UCFWin32JNI.dll  
├── fileselector  
│  └── fileSelector.jar  
└── system  
    └── ucfinit.jar

then I put urlrewritefilter-4.0.3.jar (http://tuckey.org/urlrewrite/) into WEB-INF/lib directory and added following lines to web.xml:

<filter>  
    <filter-name>UrlRewriteFilter</filter-name>  
    <filter-class>org.tuckey.web.filters.urlrewrite.UrlRewriteFilter</filter-class>  
    <init-param>  
        <param-name>modRewriteConf</param-name>  
        <param-value>true</param-value>  
    </init-param>  
</filter>  
  
<filter-mapping>  
    <filter-name>UrlRewriteFilter</filter-name>  
    <url-pattern>/*</url-pattern>  
    <dispatcher>REQUEST</dispatcher>  
    <dispatcher>FORWARD</dispatcher>  
</filter-mapping>

and finally put following .htaccess file into WEB-INF directory:

RewriteCond  %{HTTP_USER_AGENT}  Java/1\.7\.0_21  
RewriteRule  ^/(_[^/]/[^/]*?-[^/]*?/)?wdk/(system/ucfinit\.jar|fileselector/fileSelector\.jar|contentXfer/(All-MB\.jar|ES1_MRE\.exe|ExJNIAPI\.dll|ExJNIAPIGateway\.jar|jacob\.dll|jacob\.jar|libMacOSXForkerIO\.jnilib|libUCFLinuxGNOME\.so|libUCFLinuxJNI\.so|libUCFLinuxKDE\.so|libUCFSolarisGNOME\.so|libUCFSolarisJNI\.so|MacOSXForker\.jar|mac_utilities\.jar|ucf-ca-office-auto\.jar|ucf-client-installer\.zip|UCFWin32JNI\.dll))$  /wdk/JRE17_21/$2  [PT,L]  
      
RewriteCond  %{HTTP_USER_AGENT}  Java/1\.7\.0_25
RewriteRule  ^/(_[^/]/[^/]*?-[^/]*?/)?wdk/(system/ucfinit\.jar|fileselector/fileSelector\.jar|contentXfer/(All-MB\.jar|ES1_MRE\.exe|ExJNIAPI\.dll|ExJNIAPIGateway\.jar|jacob\.dll|jacob\.jar|libMacOSXForkerIO\.jnilib|libUCFLinuxGNOME\.so|libUCFLinuxJNI\.so|libUCFLinuxKDE\.so|libUCFSolarisGNOME\.so|libUCFSolarisJNI\.so|MacOSXForker\.jar|mac_utilities\.jar|ucf-ca-office-auto\.jar|ucf-client-installer\.zip|UCFWin32JNI\.dll))$  /wdk/JRE17_25/$2  [PT,L]
      
RewriteCond  %{HTTP_USER_AGENT}  Java/1\.7\.0_45
RewriteRule  ^/(_[^/]/[^/]*?-[^/]*?/)?wdk/(system/ucfinit\.jar|fileselector/fileSelector\.jar|contentXfer/(All-MB\.jar|ES1_MRE\.exe|ExJNIAPI\.dll|ExJNIAPIGateway\.jar|jacob\.dll|jacob\.jar|libMacOSXForkerIO\.jnilib|libUCFLinuxGNOME\.so|libUCFLinuxJNI\.so|libUCFLinuxKDE\.so|libUCFSolarisGNOME\.so|libUCFSolarisJNI\.so|MacOSXForker\.jar|mac_utilities\.jar|ucf-ca-office-auto\.jar|ucf-client-installer\.zip|UCFWin32JNI\.dll))$  /wdk/JRE17_45/$2  [PT,L]

And now I have webtop build that is compatible with four JRE security baselines, what about yours?!

I fixed it!

grounded Got a bad news for inhabitants of EMC Support Portal: today CERT disclosed a bunch of security vulnerabilities presented in EMC Documentum products, the most interesting thing here is a fact that CERT’s list primarily contains vulnerabilities previously announced by EMC as remediated. The list of contested CVEs is: CVE-2014-2518, CVE-2014-2514, CVE-2014-2507, CVE-2014-2513, CVE-2014-4618, CVE-2014-4626, CVE-2014-2515, CVE-2014-2504, CVE-2014-4629

I perceive the world: request.getRequestURI()

In Webtop 6.7SP2P05 EMC made some weird changes:

these changes completely removes usefulness of “componentlist” component:

after clicking on any component:

Initially I thought that this is a new security feature, but webtop configuration files should not contain any environment-sensitive information, otherwise webtop deployment gets into a mess. After some research I have found that webtop performs checks using following way:

HttpServletRequest request;
HttpServletResponse response;
// these regexps come from web.xml
Pattern staticPages = Pattern.compile(
        "(\\.bmp|\\.css|\\.htm|\\.html|\\.gif|\\.jar|\\.jpeg|\\.jpg|\\.js|\\.properties|\\.xml|\\.png)$",
        Pattern.CASE_INSENSITIVE);
Pattern configs = Pattern.compile("/app\\.xml|/config/.*\\.xml", Pattern.CASE_INSENSITIVE);
if (staticPages.matcher(request.getRequestURI()).find()) {
    if (configs.matcher(request.getRequestURI()).find()) {
        response.sendError(HttpServletResponse.SC_UNAUTHORIZED, 
                "The URL is unauthorized in WDK");
    }
}

What does getRequestURI() method return? Documentation:

Actually, this part of documentation is not clear and RFC 1808 comes to the rescue:

RFC 1808           Relative Uniform Resource Locators          June 1995


      <scheme>://<net_loc>/<path>;<params>?<query>#<fragment>

   each of which, except <scheme>, may be absent from a particular URL.
   These components are defined as follows (a complete BNF is provided
   in Section 2.2):

      scheme ":"   ::= scheme name, as per Section 2.1 of RFC 1738 [2].

      "//" net_loc ::= network location and login information, as per
                       Section 3.1 of RFC 1738 [2].

      "/" path     ::= URL path, as per Section 3.1 of RFC 1738 [2].

      ";" params   ::= object parameters (e.g., ";type=a" as in
                       Section 3.2.2 of RFC 1738 [2]).

      "?" query    ::= query information, as per Section 3.3 of
                       RFC 1738 [2].

      "#" fragment ::= fragment identifier.

Now: