EA Server 5.5 comes with a manager application called Jaguar Manager (also called Sybase Central). This is really a Java product, but in this version it’s wrapped in a program called scjview.exe. This is available in %SYBASE%\Shared\Sybase Central 4.3\win32.
I found out this is using an old JRE 1.4. (I use process explorer in Sysinternals. It’s a great tool!). I wanted to switch the EA Server to Java 1.5 , which is the latest version of Java supported in EAS 5.5. With the EA Server itself, this is easy. Just pass -jdk15 to the serverstart batch file.
With Jaguar Manager, this is not as straightforward. There is a batch file called, jagmgr.bat, but this only calls scjview.exe mentioned above. The Java runtime is probably picked up using a DLL named jsyblib142.dll, in %SYBASE%\Shared\win32.
After googling for a bit, I found a solution to this problem. Apparently, scjview.exe has a command line argument -batch.
When you run it like that, the program generates a batch file called sybasecentral.bat in the same directory as scjview.
Now, this batch file runs Java with a bunch of Jar files. You can change the path of the Java command to change the Java version it’s run in. Such a simple solution, completely hidden! And with this available, why did they have to create an EXE like that?? Beats me!
Of course, once you convert to batch file, you can tweak the java parameters such as memory etc to run better.
I recently posted about object life cycle in Sybase EAServer (Jaguar). EAServer provides object pools (cache) where objects are cached when a client lets go of it. Since, the object creation and destroy are so costly, having such an object pool helps with performance immensely.
Last Year, there was a memory leak in one of our EAServers. After debugging the issue, I narrowed it down to couple of methods. Whenever the methods were called, the memory usage seemed to increase. This kept happening until the server ran out of memory and crashed. During our peak season, our web users ended up calling this method more frequently, thus crashing the server sooner.
The real problem was that each time the user called these methods, a new instance of an object (nvo_rtn_processor) was created inside the function and was never let go.
When we looked at the code, the nvo_rtn_processor was being destroyed as expected, but still it was not released for some reason. After analyzing the issue, found that the DESTROY actually caused the issue. Changed to call of_deactivate and it worked out.
A little PB Background
When a new instance of a EA Server component is needed in PB it’s done by calling CreateInstance method in TransactionServer object. This creates an instance on the EA Server side and passes back a reference to it in PB.
For e.g., to create an instance of nvo_rtn_processor, we issued,
Where lnv_rtn_processor is the local reference in PB.
When we are done with the object thus created, we need to let go of it, so it can be returned to pool or garbage collected (see my post on object life cycle). To do this, we could either DESTROY the object or deactivate it.
Typically DESTROY is used for objects that were created using CREATE statement. For objects that were created with CreateInstance, better approach is to use SetComplete on the TransactionServer. (Unless object’s Automatic Deactivation flag is set, then PB will take care of this).
EAServer is distributed environment where distributed components participate in distributed transactions. The components don’t commit/rollback themselves. They merely indicate that they can be committed or rolledback using SetComplete or SetAbort methods. When the Jaguar Transaction Processor sees that, it commits or aborts the component’s changes. This helps Transaction processor to commit or rollback all objects participating in a single distributed transaction.
Solving the mystery
The original developer used DESTROY to get rid of nvo_rtn_processor. This should have worked. Our deactivate and destructor events for this object1 were almost identical. So the issue was not in whether it was destroyed or deactivated. The problem was actually that the component was never “Deactivated”. The key is that we are actually calling a function called of_deactivate which in turn calls SetComplete on the transaction Server. SetComplete actually did the trick – this released the component and thus returned to pool as expected.
// SetComplete to allow instance pooling/destruction
li_result = lts.SetComplete()
The key lesson is that DESTROY only destroys the handle (local variable) to the component, but not the component itself. When nvo_rtn_processor was destroyed, the PB reference and the container reference probably got destroyed, but the actual component was never let go by the TransactionServer object, as we never told it to, thus the memory leak. (If the automatic Demarcation/DeActivation was set for the component, we might not have seen this issue.)
For any component that is created through TransactionServer, we need to call SetComplete (or SetAbort in case of a failure) to complete transaction and deactivate the component. For a lot of components in our application, this is done automatically, by enabling the component’s Automatic Demarcation/DeActivation.
1. The deactivate event for an object will be called when it is deactivated – after SetComplete is called in this case. Destructor event will be called when the object is being destroyed.
2. nvo_rtn_processor itself is not declared transactional, but above discussion is still true irrespective of the transaction type of the component. Automatic Demarcation can be set for components that are not transactional as well. If this is not done, then we must use SetComplete/SetAbort to release the object explicitly.
Fig 1 EA Server Component settings for nvo_rtn_processor.
This post talks about the life cycle of a component, deployed in EA Server, at run time. The component is instantiated on the server by a CreateInstance call from a client (PB) code. The picture below must be self explanatory.
To debug/trace the life cycle, EAServer provides some log/trace flags. By turning on these, we could see the events of a EA Server component life cycle.
Memory allocation is an expensive task in any system. Whether it’s a C program doing malloc or a Java program creating objects on the fly, the system has to constantly go to memory handler and request for more memory. Releasing used memory was also a problem, it took resources, and not releasing properly caused memory leaks. Java came up with Garbage collection (gc), but even this uses a lot of resources, if not setup properly.
In a heavily used system, this means the system will be spending considerable amount of time in object creation and destruction (this will be done by Garbage collection). To alleviate this problem, some type of pooling (or caching) method is often used. Simply put, this is where memory released is held in a pool and reused when the next person asks for it. This way we reduce the number of times objects are created or released.
EA Server offers Object Pooling for components deployed to EA server. When this option is turned on for a component, the object instances are pooled before returning to memory.
EA Server/PB settings that effect Pooling
Our Software uses object pooling option available in EA Server. This option is external to the PB language and is part of EA Server administration. The settings are available in component Properties -> Instances tab in Jaguar Manager.
Below setting enables Pooling for a component in EA Server, if checked.
Fig 1: This defines if pooling needs to be used for a particular object.
Below settings for the component in EA Server define how and when Pooling of the object can take place. As mentioned in the note, an object instance is kept in the pool only if it doesn’t exceed maximum setting. In this case, the pool will have at least 3 instances at any time. By adjusting these settings we can optimize the memory handling for the object. Too low setting for the minimum will cause more frequent garbage collection and too high will keep too many instances in the pool, thus wasting memory.
On the other hand, “Maximum Pooled Instances” setting effects how many can be kept in the pool. In a scenario where this object is constantly requested, increasing maximum setting will help. Again, this also means too many will be sitting in the pool during idle times.
PB settings for Pooling
The below PB settings only decide if the releasing of object reference from PB side is automatic or manual. These settings are set at the component level in EAServer Component Generator (Project Painter).
The actual pooling is effected by the settings on the EA Server side, as mentioned above.
Fig 3: This shows the settings on the PB side.
EA Server object instance life cycle
Below image is a capture of a visio diagram (attached) that attempts to show how pooling plays a role when objects are created from Powerbuilder code. As mentioned, Auto demarcation affects how the object references are released (automatic or manual) on the PB side.
When pooling is enabled for the object on the EA Server side (see Fig1), once the references are released the object instance is typically qualified for garbage collection.
If pooling is enabled, the object instance is then returned to Object Pool in EA Server, if the other conditions are met (see Fig 2).
Fig 3: Object Life Cycle during a typical method call
Sybase EA Server (Ver 5.5) provides 2 interfaces for the admin to work with the Server and the Repository. Sybase Jaguar Manager is the GUI interface, that one can use to manage the Server and the repositories.
Jagtool is the command line equivalent of the above manager. There are a lot of things you can do in Jagtool and with host scripting (batch files in windows), you can make real powerful scripts to execute these in batch mode.
EA Server typically configures all elements using properties files. But, it also allows us to use XML files to configure the server. These are available at various levels such as Server, Package, Component, Connection Cache etc.
This gotcha is about writing comments in an XML file meant for configuring EA Server. Single line comments follow the XML/HTML standard.
<!!-- this is a single line comment -->
When you write multiline comment, make sure the tags are on their own line, thus:
<!!-- This is a multi-line comment -->
If you include any text in the first line, it will error out.
Note: You only need <!– to mark the beginning of a comment; I doubled exclamation mark to avoid losing the comment line in view!! Otherwise, WordPress HTML processor would have hidden it from view.