JMeter vs UCF

Imagine that you need to perform load tests, but you have no money to buy HP LoadRunner licences, so you think that Apache JMeter is a good opensource alternative to HP LoadRunner – from my perspective this “genius” idea sounds weird, because the main advantage of HP LoadRunner is it’s analysis engine, though, my colleagues think it is possible to collect JMeter logs and process them using MS Excel, on the other hand I had never seen a proper HP LoadRunner report – performance guys always like to provide something like:

and I have no idea how to interpret such drawings πŸ™‚

Well, we need to implement load scenarios using Apache JMeter and the main “challenge” is emulate content transfer “facilities” (did you know that UCF is acronym for “Unified Client Facilities”?). Actually it is not a challenge at all, all what you need is following:

  • if your JMeter instance is on UNIX you need to modify wdk/contentXfer/ucf.installer.config.xml on webtop side and remove dots before “ucf” in ucfHome and ucfInstallsHome elements – you need to get “$java{user.home}/Documentum/ucf” instead of “$java{user.home}/Documentum/.ucf”
  • download ucfinit.jar to JMeter host from webtop and execute
    java -cp ucfinit.jar com.documentum.ucf.client.install.TestInstall \
    http(s)://<host>:<port>/webtop /wdk/contentXfer/ucf.installer.config.xml
  • copy all files from ~/Documentum/ucf/<host>/shared/bin/<ucf version>/ directory to JMeter’s lib/ext directory
  • set CookieManager.save.cookies=true in jmeter.properties
  • create following groovy (JSR223) sampler:
    import com.documentum.ucf.client.transport.impl.ClientSession;
    import com.documentum.ucf.client.transport.requesthandlers.spi.BaseRequestHandler;
    import com.documentum.ucf.client.transport.IClientRequest;
    import com.documentum.ucf.client.transport.IResultPackage;
    import com.documentum.ucf.common.transport.IParameter;
    import com.documentum.ucf.client.transport.impl.BaseResultPackage;
    import com.documentum.ucf.common.transport.spi.BaseResponse;
    import com.documentum.ucf.client.transport.impl.RequestProcessor;
    import com.documentum.ucf.client.transport.requesthandlers.spi.RequestHandlerMismatchException;
    import org.apache.jmeter.threads.JMeterVariables;
    
    class LaunchApplicationHandler extends BaseRequestHandler {
    
      public IResultPackage process (IClientRequest request) throws RequestHandlerMismatchException {
        BaseResultPackage result = init(request);
        BaseResponse response = (BaseResponse) result.getResponse();
        return result;
      }
    	
    }
    
    class UCFHelper {
    
      static {
        RequestProcessor requestProcessor = ClientSession.s_requestProcessor;
        requestProcessor.addHandler("LaunchApplication", new LaunchApplicationHandler());
      }
    
      public static String getCookie(JMeterVariables vars) {
        String cookie = vars.get("COOKIE_JSESSIONID");
        return "JSESSIONID=" + cookie; 	
      }
    
      public static String getUrl(JMeterVariables vars) {
        return vars.get("WebtopProtocol") + "://" + vars.get("WebtopHostName") + ":" + vars.get("WebtopPort") + vars.get("WebtopContextRoot"); 	
      }
    
      public static void createUcfSession(JMeterVariables vars) {
        String cookie = getCookie(vars);
        ClientSession session = vars.getObject("UCFSESSION");
        boolean valid = session != null && session.isAlive();
        if (valid) {
          valid = cookie.equals(session.getSessionId());	
        }
        if (valid) {
          return;	
        }
        if (session != null) {
        	 session.interrupt();
        }
        session = new ClientSession(getUrl(vars), "Gair", cookie);	
        session.startSession();
        vars.putObject("UCFSESSION", session);
        vars.put("UCFKEY", session.getUID());	
      }
    	
    }
    
    UCFHelper.createUcfSession(vars);
    
    

voilΓ :

Silent UCF. HowTo

As was promised in previous post, I describe how make webtop customers happy.

What is the problem?

It seems that when Oracle acquired Sun it also acquired a bunch of security issues in JRE, for example, it’s pretty obvious that even if applet is signed by valid certificate it’s not enough to consider it as trusted – benign applet could be potentially used for malicious purposes (for example UCF applet is capable to launch any program on user’s machine, so attacker may exploit this capability). Security measures for this issue are obvious: just bind applet to specific website, and Oracle did it: JAR File Manifest Attributes for Security. Unfortunately these “innovations” are barely suitable for self-hosted applications – URLs vary from customer to customer, which makes it impossible to create applet suitable for every customer, though, I suppose EMC’s customers pay enough support fees to have an ability to request “custom” UCF applet from EMC, “custom” means that only manifest changes are required and such procedure could be easily automated – why do not create special service on customer portal? Instead of doing correct things EMC suggests something weird:

Actually, I double-checked Oracle’s documentation and was unable to find a place where they suggest to delegate security-related questions to business users, I suppose that EMC guys think that some posts in forums/blogs/etc have a status of official documentation, that, in fact, is not surprising: for example, IIG’s director of security prefers to get inspiration from this blog instead of performing code review:

Possible solutions

Anyway, Oracle’s suggestions related to security prompts are:

Though rule sets option is straightforward I do not think that it is helpful due to following reasons:

  • it costs money: “The JAR file must be signed with a valid certificate from a trusted certificate authority”
  • it is dangerous: “The Deployment Rule Set feature is optional and shall only be used internally in an organization with a controlled environment. If a JAR file that contains a rule set is distributed or made available publicly, then the certificate used to sign the rule set will be blacklisted and blocked in Java”, i.e. you risk to loose money spent for your certificate
  • if you already have a “valid certificate from a trusted certificate authority” why do not sign all applets in enterprise? So, this option is more suitable for cases when applets are embedded in hardware devices like KVMs, iLOs, etc and you unable to replace those applets
  • I think the best case for rule sets is enable certain known applets and block all other

Second option costs money too because it requires a “valid certificate from a trusted certificate authority”, but it’s the only disadvantage of this option. So, what did I do to disable all security prompts in previous video? Obviously, at first, we bought code-signing certificate.

Solution

Let’s take a look at security prompts raising when UCF applet gets loaded:

  • Java’s standard prompt asking whether user is going to trust applet’s certificate:
  • UCF’s prompt aimed to prevent malicious usage of applet (whitelisting):
  • Java’s prompt about not following security best practices:

For the third option EMC, as we already know, suggests to tick “Do not show this again for this app and web site” checkbox before accepting security warning. Besides the fact that delegating security-related questions to end-users is not a good idea, this options does not work properly. The problem is WDK generates special URLs for applets to prevent them from caching by JRE:

here “19gmgko28” and “98th” parts of URL are just encoded values of last modified time and size of applet file:

-bash-4.1$ stat -c "Last Modified: %Y, size: %s" \
> app/webapps/wp/wdk/system/ucfinit.jar
Last Modified: 1426684797, size: 304049
-bash-4.1$ groovysh
Groovy Shell (2.4.1, JVM: 1.6.0_45)
Type ':help' or ':h' for help.
groovy:000> import com.documentum.web.util.StringUtil
===> com.documentum.web.util.StringUtil
groovy:000> StringUtil.toUnsignedString(1426684797000,5)
===> 19gmgko28
groovy:000> StringUtil.toUnsignedString(304049,5)
===> 98th

Such behaviour of WDK has following disadvantages:

  • redeploying/updating webtop changes last modified time or/and size of applet file
  • in clustered environment it’s expected that applet file has different last modified time on different nodes

also we already know that the cornerstone of UCF troubleshooting is to delete everything related to Java and Documentum, reinstall JRE, press ctrl+alt+reset, etc – these activities obviously wipe user’s settings, it’s clear that ticking “Do not show this again for this app and web site” checkbox is just a temporary workaround, so, we decided to sign applet by our certificate. The procedure is straightforward:

  • edit META-INF/MANIFEST.MF, you should get something like (note Caller-Allowable-Codebase, Application-Library-Allowable-Codebase and Codebase headers):
    Manifest-Version: 1.0
    Built-By: dmadmin
    Application-Name: Documentum
    Created-By: 1.5.0_11-b03 (Sun Microsystems Inc.)
    Copyright: Documentum Inc. 2001, 2004
    Caller-Allowable-Codebase: docum-at-app
    Build-Date: January 17 2015 09:07 PM
    Ant-Version: Apache Ant 1.8.4
    Title: Documentum Client File Selector Applet
    Application-Library-Allowable-Codebase: docum-at-app
    Bundle-Version: 6.7.2220.0231
    Build-Version: 6.7.2220.0231
    Permissions: all-permissions
    Codebase: docum-at-app
    
  • remove old signature files, i.e. META-INF/DOCUMENT.RSA and META-INF/DOCUMENT.SF
  • run jarsigner to sign applet – do not forget to specify TSA URL

after that applet is ready for deployment in production. What about another two security prompts? I suppose it is obvious that whitelisting capability is now useless because now JRE performs the same checks, below is an example of security notice when applet gets loaded from non-trusted URL:

because of these considerations, I decided to disable whitelisting capability, unfortunately the only option to disable it in UCF is decompile com.documentum.ucf.client.install.installer.security.impl.Whitelist class: methods isHostAllowed(String) and isHostAllowed(String, IHostVerifier) must always return true (I bet it doesn’t worth to mention that after replacing class in applet you must sign applet again).

And finally, to remove the first security prompt I read Deployment Configuration File and Properties article and decided that manipulating by User Level configuration files is not a good idea, but System Level configuration files could be easily distributed among users’ machines in enterprise using various deployment tools, so, I did following (note naming convention of certificate alias):

C:\Windows\system32>md C:\Windows\Sun\Java\Deployment

C:\Windows\system32>cd C:\Windows\Sun\Java\Deployment

C:\Windows\Sun\Java\Deployment>type CON > deployment.config
deployment.system.config=file\:C\:/Windows/Sun/Java/Deployment/deployment.properties

C:\Windows\Sun\Java\Deployment>type CON > deployment.properties
deployment.system.security.trusted.certs=C\:\\Windows\\Sun\\Java\\Deployment\\trusted.certs

C:\Windows\Sun\Java\Deployment>keytool.exe -importcert -keystore trusted.certs -file G:\Users\andrey\work\cert.crt -alias deploymentusercert$tsflag$loc=http//docum-at-app:8280##docbase:http//docum-at-app:8280##from:http//docum-at-app:8280
Enter keystore password:
Re-enter new password:

...

Trust this certificate? [no]:  yes
Certificate was added to keystore

VoilΓ .

How to provide backward compatibility with old java clients during UCF updates

This post was originally published on ECN.

If you are not familiar with apache httpd’s mod_rewrite and regexps, please do not read this post.

Problem

When EMC releases new UCF fixes, aimed to provide compatibility with corresponding JRE security fixes, they completely miss the fact, that customers are unable to perform JRE updates for all business users at the same time, and, so, IT personal should choose the lesser of two evils: either some users should suffer or all users should stay with vulnerable JRE, moreover, sometimes I can’t understand what EMC is doing (see also: Dumb UCF applet, Dumb UCF applet. Part II, UCF applet’s certificate expired… ORLY?)

Solution

At first we need to understand what files EMC changes from one point fix to another (it’s very simple for me because I download every new pointfix to check that EMC still not fixed 30 security issues ). Typically they change wdk/system/ucfinit.jar and wdk/fileselector/fileSelector.jar files, but ucfinit.jar contains checksums for some files in wdk/contentXfer directory, so, ucfinit.jar file in general is not interchangeable between point fixes. Major changes in webtop 6.7SP2 was (actually I double checked 1.7_45 and 1.7_25 versions I can certainly say that patch notes lie: for both versions it’s required to relax security settings to get working UCF):

  • P11 – JRE1.7_51 support
  • P07 – JRE1.7_45 support
  • P05 – JRE1.7_25 support
  • P02 – JRE1.7_21 support

that means that following filesets are consistent:

  • wdk/system/ucfinit.jar, wdk/fileselector/fileSelector.jar, wdk/contentXfer from P11 for JRE1.7_51
  • wdk/system/ucfinit.jar, wdk/fileselector/fileSelector.jar, wdk/contentXfer from P10 for JRE1.7_45
  • wdk/system/ucfinit.jar, wdk/fileselector/fileSelector.jar, wdk/contentXfer from P06 for JRE1.7_25
  • wdk/system/ucfinit.jar, wdk/fileselector/fileSelector.jar, wdk/contentXfer from P04 for JRE1.7_21

for filesets 2-4 I created following structure inside wdk directory:

JRE17_21  
β”œβ”€β”€ contentXfer  
β”‚  β”œβ”€β”€ All-MB.jar  
β”‚  β”œβ”€β”€ ES1_MRE.exe  
β”‚  β”œβ”€β”€ ExJNIAPI.dll  
β”‚  β”œβ”€β”€ ExJNIAPIGateway.jar  
β”‚  β”œβ”€β”€ jacob.dll  
β”‚  β”œβ”€β”€ jacob.jar  
β”‚  β”œβ”€β”€ libMacOSXForkerIO.jnilib  
β”‚  β”œβ”€β”€ MacOSXForker.jar  
β”‚  β”œβ”€β”€ mac_utilities.jar  
β”‚  β”œβ”€β”€ ucf-ca-office-auto.jar  
β”‚  β”œβ”€β”€ ucf-client-installer.zip  
β”‚  └── UCFWin32JNI.dll  
β”œβ”€β”€ fileselector  
β”‚  └── fileSelector.jar  
└── system  
    └── ucfinit.jar  
  
JRE17_25  
β”œβ”€β”€ contentXfer  
β”‚  β”œβ”€β”€ All-MB.jar  
β”‚  β”œβ”€β”€ ES1_MRE.exe  
β”‚  β”œβ”€β”€ ExJNIAPI.dll  
β”‚  β”œβ”€β”€ ExJNIAPIGateway.jar  
β”‚  β”œβ”€β”€ jacob.dll  
β”‚  β”œβ”€β”€ jacob.jar  
β”‚  β”œβ”€β”€ libMacOSXForkerIO.jnilib  
β”‚  β”œβ”€β”€ MacOSXForker.jar  
β”‚  β”œβ”€β”€ mac_utilities.jar  
β”‚  β”œβ”€β”€ ucf-ca-office-auto.jar  
β”‚  β”œβ”€β”€ ucf-client-installer.zip  
β”‚  └── UCFWin32JNI.dll  
β”œβ”€β”€ fileselector  
β”‚  └── fileSelector.jar  
└── system  
    └── ucfinit.jar  
  
JRE17_45  
β”œβ”€β”€ contentXfer  
β”‚  β”œβ”€β”€ All-MB.jar  
β”‚  β”œβ”€β”€ ES1_MRE.exe  
β”‚  β”œβ”€β”€ ExJNIAPI.dll  
β”‚  β”œβ”€β”€ ExJNIAPIGateway.jar  
β”‚  β”œβ”€β”€ jacob.dll  
β”‚  β”œβ”€β”€ jacob.jar  
β”‚  β”œβ”€β”€ libMacOSXForkerIO.jnilib  
β”‚  β”œβ”€β”€ MacOSXForker.jar  
β”‚  β”œβ”€β”€ mac_utilities.jar  
β”‚  β”œβ”€β”€ ucf-ca-office-auto.jar  
β”‚  β”œβ”€β”€ ucf-client-installer.zip  
β”‚  └── UCFWin32JNI.dll  
β”œβ”€β”€ fileselector  
β”‚  └── fileSelector.jar  
└── system  
    └── ucfinit.jar

then I put urlrewritefilter-4.0.3.jar (http://tuckey.org/urlrewrite/) into WEB-INF/lib directory and added following lines to web.xml:

<filter>  
    <filter-name>UrlRewriteFilter</filter-name>  
    <filter-class>org.tuckey.web.filters.urlrewrite.UrlRewriteFilter</filter-class>  
    <init-param>  
        <param-name>modRewriteConf</param-name>  
        <param-value>true</param-value>  
    </init-param>  
</filter>  
  
<filter-mapping>  
    <filter-name>UrlRewriteFilter</filter-name>  
    <url-pattern>/*</url-pattern>  
    <dispatcher>REQUEST</dispatcher>  
    <dispatcher>FORWARD</dispatcher>  
</filter-mapping>

and finally put following .htaccess file into WEB-INF directory:

RewriteCond  %{HTTP_USER_AGENT}  Java/1\.7\.0_21  
RewriteRule  ^/(_[^/]/[^/]*?-[^/]*?/)?wdk/(system/ucfinit\.jar|fileselector/fileSelector\.jar|contentXfer/(All-MB\.jar|ES1_MRE\.exe|ExJNIAPI\.dll|ExJNIAPIGateway\.jar|jacob\.dll|jacob\.jar|libMacOSXForkerIO\.jnilib|libUCFLinuxGNOME\.so|libUCFLinuxJNI\.so|libUCFLinuxKDE\.so|libUCFSolarisGNOME\.so|libUCFSolarisJNI\.so|MacOSXForker\.jar|mac_utilities\.jar|ucf-ca-office-auto\.jar|ucf-client-installer\.zip|UCFWin32JNI\.dll))$  /wdk/JRE17_21/$2  [PT,L]  
      
RewriteCond  %{HTTP_USER_AGENT}  Java/1\.7\.0_25
RewriteRule  ^/(_[^/]/[^/]*?-[^/]*?/)?wdk/(system/ucfinit\.jar|fileselector/fileSelector\.jar|contentXfer/(All-MB\.jar|ES1_MRE\.exe|ExJNIAPI\.dll|ExJNIAPIGateway\.jar|jacob\.dll|jacob\.jar|libMacOSXForkerIO\.jnilib|libUCFLinuxGNOME\.so|libUCFLinuxJNI\.so|libUCFLinuxKDE\.so|libUCFSolarisGNOME\.so|libUCFSolarisJNI\.so|MacOSXForker\.jar|mac_utilities\.jar|ucf-ca-office-auto\.jar|ucf-client-installer\.zip|UCFWin32JNI\.dll))$  /wdk/JRE17_25/$2  [PT,L]
      
RewriteCond  %{HTTP_USER_AGENT}  Java/1\.7\.0_45
RewriteRule  ^/(_[^/]/[^/]*?-[^/]*?/)?wdk/(system/ucfinit\.jar|fileselector/fileSelector\.jar|contentXfer/(All-MB\.jar|ES1_MRE\.exe|ExJNIAPI\.dll|ExJNIAPIGateway\.jar|jacob\.dll|jacob\.jar|libMacOSXForkerIO\.jnilib|libUCFLinuxGNOME\.so|libUCFLinuxJNI\.so|libUCFLinuxKDE\.so|libUCFSolarisGNOME\.so|libUCFSolarisJNI\.so|MacOSXForker\.jar|mac_utilities\.jar|ucf-ca-office-auto\.jar|ucf-client-installer\.zip|UCFWin32JNI\.dll))$  /wdk/JRE17_45/$2  [PT,L]

And now I have webtop build that is compatible with four JRE security baselines, what about yours?!

Dumb UCF applet

It seems that EMC has changed their release policy, and now we got support of JRE 1.7.0_55 in webtop within two weeks after general availability of JRE 1.7.0_55, or may be it is just a happenstance, judge for yourselves what is more probable, below I provided release dates of JRE and corresponding webtop versions (actually, it’s worth to note that both webtop 6.7SP2P05 and 6.7SP2P07 did not support corresponding JRE versions fully – it was required to relax JRE security settings to get working UCF):
Continue reading