Fix: Groupwise Webaccess error CMC initialization of the GroupWise domain database failed (f107)


Earlier I wrote about loading Groupwise WebAccess 7.0.3HP1 into it’s own memory address space to lessen the effects of the MapSCCErrtoGWDCAErr-SCCErr errors.  My solution worked great for most of my WebAccess servers, but gwinter would fail to load on one of the Netware 6.5.5/Groupwise 7.0.3HP1 with the following error: 

Groupwise Webaccess error CMC initialization of the GroupWise domain database failed (f107)

TID 10063398 describes issues with loading WebAccess into protected memory when the agent is remote from the domain, which is how this server is configured.  I was trying to load WebAccess into it’s own address space, but not protecting it, but nevertheless this solution worked for me:
 
1)  Loading strtweb.ncf from the autoexec.ncf with the following syntax
 
protect strtweb.ncf
 
2)  Adding the following line to strtweb.ncf per TID 10063398
 
load dsapi.nlm
 
3)  adding the following line to stopweb.ncf
 
unload dsapi.nlm
 
My theory is dsapi.nlm needs to be loaded into the same address space as gwinter.nlm whenever the WebAccess is not on the same server as it’s MTA.  I’ll test this theory next time I can take WebAccess down for testing.

Fix: Groupwise Document Viewer error GWDVA MapSCCErrtoGWDCAErr – SCCErr=11


We have recently begun loading our Netware 6.5.5/Groupwise 7.0.3 WebAccess servers into their own memory spaces in order to aleviate the occurance of the following Groupwise Document Viewer Agent error that has caused WebAccess to become unresponsive after a number of occurances:

GWDVA MapSCCErrtoGWDCAErr – SCCErr=11
 
TID 7001663 and TID 7001657 say that this is a known bug in GW 7.0.3 HP1 that has been reported to engineering and development.
 
In order to greatly lessen the occurances of these errors, we did the following to load Groupwise WebAccess into it’s own memory space:
 
1)  Edit sys:\system\strtweb.ncf
 
2)  Replace
 
load sys:\system\gwinter @webac703.waa
with
load address space=webaccmem sys:\system\gwinter @webac703.waa
 
Where webaccmem is an arbitrary name I chose for the memory space and webac703.waa is the name of your webaccess configuration file.
 
3)  Edit sys:\system\stopweb.ncf
 
4)  Replace
 
unload gwinter
with
unload address space=webaccmem
 
5) Unload the old webaccess agent from the server console by typing
 
unload gwinter
 
6)  Load WebAccess into protected memory from the server console by typing
 
strtweb.ncf
 
Make sure to execute stopweb.ncf rather than unloading WebAccess by pressing F7 from the agent screen on the server console when unloading from memory.  Pressing F7 will cause the memory allocated to the webaccmem address space not to be available for re-allocation.

Creating eDirectory SSL certificates with alternate names to use across round robin DNS load balanced web servers


We have three internal Apache web servers that we use for Groupwise webaccess 7.0.3.  Each server will be accessed acrossed our intranet via round robin DNS at https://webaccess/gw/webacc for email.  When users currently access this URL they are getting Internet Explorer Security Alerts, stating:  

The name on the security cerrtificate is invalid or does not match the name of the site.  Do you want to proceed?
 
In order to fix this issue, I need to install SSL certificates on each individual server and configure Apache to use the new certificates.  I also needed to configure my web browser to trust the issuing Certificate Authority.
 
I chose to use our existing Novell Organizational CA to issue the certificates rather than purchase one from Verisign or other Trusted Root Certification Authority since these sites would only be accessed across the corporate intranet.
 
We had one additional requirement – each server still needed to be accessed via https at https://servername for Novell Remote Manager and iManager.  This meant the three servers had to have valid SSL certificates for multiple host names, i.e. both their actual name and the webaccess name.

The Environment

  • Three Netware 6.5.5 server running Apache 2.0.54 for Netware. Servers are named web1, web2, and web3
  • ConsoleOne 1.3.6f
  • Novell Certificate Server Snapin version 2.21 Build 28
  • Internet Explorer 6 web browser
 Creating the server SSL certificates
 
1.  Launch ConsoleOne
 
2.  Browse to the OU that holds the servers you wish to create certificates for.
 
3.  Right click on the server OU
 
4.  Select New – Object – NDSPKI:Key Material – OK
 
5.  Select the server name you want to create the certificate for, and give the certificate a meaningful name.  I named mine intwebaccessweb1
 
6.  Under Creation Method, select Custom – Next
 
7.  Select Organizational Certificate Authority will sign this certificate – Next
 
8.  Accept the defaults of 2048 bit key size, SSL or TLS type, and allow the private key to be exported – Next
 
9.  This is an important part – The subject name must match how you will be accessing your server over https for iManager and NRM.  Click the Edit button, then click the double arrow button to the right of the subject name.  This will move the .CN= portion of the name to the left side of the box.
Replace everything from .CN= to .OU= (or .O=) with the name you will be accessing your server with.  

Since I will be accessing my server at https://web1, I used .CN=web1.O=myOrg.  

If you will be accessing your server for iManager, NRM, or other non-shared services at https://www.yourdomain.com you would enter .CN=www.yourdomain.com.O=yourOrg

10.  Press OK  to accept the subject name.
 
11.  Change the validity period to what ever duration you would like your certificate to be valid for.  I selected maximum, which will make it good until the certificate for my Organizational CA expires.
 
12.  Press the Add Name button – here is where we specify our secondary name we want the SSL certificate to be valid for.
 
13.  Highlight the existing Directory name and press Delete.
 
14.  Click Create – DNS Name
 
15.  Specify the host name you will be sharing amongst your web servers.  This is sometimes referred to as a DNS Subject Alternate Name
 
I specified webaccessOKOKNext.  Again, if you will be accessing your shared web server at https://www.yourdomain.com, specify http://www.yourdomain.com as the DNS name.
 
16.  Select to associate this server certificate with Your organization’s certificate – NextFinish
 
I then repeated these steps for my other two web servers, replacing in steps 5 and 9 ‘web1’ with ‘web2’ and ‘web3’, which are the real host names of my other web servers.  Step 15 remains the same, since this is the common name I want all three web servers to respond to.
 
Configuring Apache to use the new SSL certificates
 
1.  On the first web server edit the sys:\Apache2\conf\httpd.conf file.
 
2.  Replace the line reading
 
SecureListen 443 “SSL CertificateDNS”
 
with
 
SecureListen 443 “intwebaccessweb1”
 
where intwebaccessweb1 is the name of the web server you created in the section above.  Note that the certificate object will be displayed in ConsoleOne as ‘intwebaccessweb1 – web1’.  Do not include the hyphen and server name, i.e. ‘ – web1‘ in the SecureListen statement.
 
3.  Save the httpd.conf file
 
4.  On the web server console, run ap2webdn to unload Apache
 
5.  On the web server console run tc4stop to stop Tomcat
 
6.  On the web server console, run tckeygen to update the keystore data.  Switch to the logger screen to verify the process completes before proceeding to the next step.
 
7.  On the web server console, run tomcat4 to load Tomcat.  Switch to the logger screen to verify the process completes before proceeding to the next step.
 
8.  On the web server console, run ap2webup to load Apache.
 
9.  Browse to the shared name of your web server, https://webaccess/gw/webacc in my case.  Note that you will still receive the Security Alert pop-up until you install the Organizational CA certificate into your Trusted Root Certification Authorities store, which I’ll document tomorrow.
 
10.  On the Security Alert pop-up, you should see the message stating The security certificate has a valid name matching the name of the page you are trying to view.
 
This means your SSL certificate is valid for the host name shared by the web servers.
 
11.  Browse to https://web1, which is the host name of one of your web servers defined in step 9 of Creating the server SSL certificates.  
 
Again, you’ll still receive the Security Alert until you install the Organizational CA certificate into your Trusted Root Certification Authorities store, but you should see The security certificate has a valid name matching the name of the page you are trying to view.  This means your SSL certificate is valid for the host name for this specific web server.
 
Here are the instructions for installing the Organizational CA certificate into your browser’s Trusted Root Certification Authorities store, which is the final thing we’ll need to do to rid ourselves of the Internet Explorer’s Security Alerts.

Howto: Redirect Groupwise 6.5 url /servlet/webacc to Groupwise 7 URL /gw/webacc


Groupwise 6.5 WebAccess is accessed via http://yourserverDNSorIP/servlet/webacc, while Groupwise 7.0 WebAccess is accessed at http://yourserverDNSorIP/gw/webacc.  Here’s how to configure Apache to redirect the old WebAccess url to the updated one:

1) On the Netware server, edit sys:\apache2\conf\httpd.conf 

2) At the bottom of the file paste the following line:

redirect permanent /servlet/webacc http://yourserverDNSorIP/gw/webacc

replacing yourserverDNSorIP with your server’s DNS name or IP address

3) On the Netware console unload Apache by running

ap2webdn

4) On the Netware console load Apache by running

ap2webup

If you access  http://yourserverDNSorIP/servlet/webacc with your web browser, you should now be redirected to http://yourserverDNSorIP/gw/webacc

Fix for “LDAP login failed” error when trying to install Groupwise 7 Webaccess or GWIA on SLES Linux


To fix the LDAP login failed error when trying to install Groupwise 7 Webaccess or GWIA on SLES Linux:

Go to LDAP Group object for the server (not LDAP server object).  On the General tab, uncheck Require TLS for simple binds with Password > OK

Goto LDAP server object for the server, and on the General tab press Refresh NLDAP Server now

Install GWIA or Webaccess, and when installation is complete re-enable Require TLS for simple binds with Password and Refresh NLDAP Server.
 
The reason why is detailed in section 9.3.3 of the Groupwise 7 installation instructions
 

During installation, the WebAccess Installation program requires access to eDirectory by way of LDAP authentication. The LDAP Group object includes an option named Require TLS for Simple Binds with Password, which is enabled by default. With this option enabled, you must provide the LDAP server’s Trusted Root Certificate, which must be exported from the LDAP server, in order for LDAP authentication to take place (typically on port 636) during installation of the WebAccess.

Unless you already have SSL set up, an easier alternative is to disable Require TLS for Simple Binds with Passwords in ConsoleOne, which allows LDAP authentication to take place using clear text (typically on port 389), during installation of WebAccess. After disabling the option, restart eDirectory, install WebAccess, then re-enable Require TLS for Simple Binds with Password and restart eDirectory again.

Fix: Groupwise webaccess hanging with “Request aborted while waiting on locked conversation” in webaccess log file


Users of our external Groupwise 6.5.6 webaccess gateway have been experiencing problems with their sessions hanging when trying to open items. Users access our internal webaccess gateways, which are the same version and configuration as the external gateway, have not been experiencing this problem. All Groupwise servers were running on NetWare 6.5.5. and were patched up to FTF GW 6.5 post SP6 English only Agents Rev 6.

It did not matter which Internet browser the external users were using, the same problem was apparent for users of Firefox 2.x, 3.x, IE6, and IE7. They’d try to open an item, and their browser would appear to hang for anywhere from a few seconds to a few minutes. Rebooting the webaccess server did not have any affect on the problem.

The following message was seen in the webaccess log file:

Error: Request aborted while waiting on locked conversation

The only Novell TID that was relevant to this problem is TID #10023251.  It describes this exact error message, but it specifically states the error only occurs when trying to view an attachment, which wasn’t the case for our users, since the error was logged when they were just navigating though the webaccess client.

The TID referenced above was not helpful, since it stated the only fix is to have the user perform the action again (ie resend the message) and there are no configurable parameters to increase this timeout.

Here’s what we did to troubleshoot this problem:

First thing I did was to verify the amount of free disk space on the sys volume of each server. The two well behaving servers had at least 750MB free, while the failing server only had 500MB free. Java sometimes behaves poorly when it lacks an abundance of free disk space, so I cleared out 500MB of old log files and restarted the server. Unfortunately, no change in performance or error rate was noted.

I loaded config.nlm on the good internal webaccess servers and the problematic external webaccess server. I then used Winmerge to compare the log resulting files to check for differences in versions of drivers, nlms, and configuration files. One of the team members noticed a difference in how webaccess was being loaded in protected mode. We tried duplicating the change on the external server, but that didn’t have any impact on the situation.

Next I checked the versions of java and tomcat on all machines:
To get the Java version number: java – version
To view the running instances of Java: java – show
To see Java instance memory utilization: java -showmemoryID where ID is the ID of the instance listed when performing java – show with no space between showmemory and the ID number.
Note: You have to switch to the NetWare console logger screen to see the output of these commands

I didn’t see anything abnormal in the problem server’s Java configuration, so next I looked at the Sys:\Apache2\logs\mod_jk.log file. Inside it I saw the following messages, repeated frequently:

jk_ajp_common.c (1318)]: Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=ajp13admin failed errno = 54

jk_uri_worker_map.c (620)]: In jk_uri_worker_map_t::map_uri_to_worker, wrong parameters

jk_ajp_common.c (1483): Timeout with waiting reply from tomcat. Tomcat is down, stopped or network problems

jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)

These messages made me think communication was definitely failing somewhere. The server administrator in charge of the NetWare servers replaced the server’s patch cable and moved it to another port on the switch, thinking that may help communication. It didn’t.

My co-worker was poking through the NetWare Console Monitor – LAN/WAN drivers – highlight NIC – press tab for stats, and noticed increasing Rx CRC errors, as well as other errors. He replaced the network card, and all of the webaccess errors went away!

Groupwise Webaccess Loads, but Users Cannot See Webaccess Login Screen


Saturday I received a frantic call from the network administrator of a client.  Their Groupwise 7.0.2 system that runs on a Netware 6.5.7 server had suddenly become inaccessible from both the Internet and local network for Webaccess clients.  Users who used the full Groupwise client reported no issues.

The local admin had performed some troubleshooting prior to contacting me.  She had rebooted the server several times, confirmed Webaccess was loading, verified Webaccess was inaccessible from both the standard URL and IP based URL, i.e. https://mail.domain.com/gw/webacc and https://10.0.0.6/gw/webacc.  She also verified there was nothing odd logged in the Webaccess log files. 

On the Netware server console I typed m ap* to see if the apache2.nlm was loaded.  The .nlm was not listed, so I typed ap2webup.ncf to try to load the Apache web server.  I did not receive an error about Apache not loading, and the logger screen did not show any errors, but when I typed m ap* again, Apache2.nlm was still not listed.

I was fairly certain this was an Apache, not Groupwise Webaccess problem, so I checked out the most recent error_log file in my Sys:\Apache2\logs folder, where I found the following message:

[Sat Apr 26 13:37:23 2008] [crit] (10022)Unknown error: make_secure_socket: for port 443, WSAIoctl: (SO_SSL_SET_SERVER) Configuration Failed

I did some searching and came across TID 3209228 Apache for Netware will NOT load any longer.  This article pointed to a certificate problem being the reason why Apache would not load.

I followed the TID’s advice and performed the following from the Netware server console:

  1. Load pkidiag.nlm
  2. Login as admin (or someone with appropriate rights to create certificates)
  3. Select options 4, 5, 6, 0
  4. run tckeygen.ncf
  5. Rebooted the server

Once the server restarted, Apache loaded succesfully, and the Groupwise Webaccess login screen was once again available for web clients.