CA-Clipper 5.x Memory ManagementApproved by Roger Donnay for Indonesian Translation in May 2002
The layout is revised due to consistency with my website, the original link is http://www.dclip.com/memory.htm
CA-Clipper 5.x Memory Management
Approved by Roger Donnay for Indonesian Translation in May 2002
The layout is revised due to consistency with my website, the original link is http://www.dclip.com/memory.htm
Introduction
The CA-Clipper Memory System
Terms and Definitions
Virtual Memory Manager
Symbol Table
Clipper Code
Macros
Code Blocks
Dynamic Paging System
Programming Techniques
Conventional Memory
Symbol Management
Use Constants instead of Memvars
Use Arrays instead of Memvars
Use Same-Name Memvars
Use LOCALs instead of PRIVATEs
Use STATIC Functions
Use Pseudo-Functions
Don't use STATICs to Pass Parameters
The "Myth" of STATICs vs PUBLICs
The Affect of Databases
Declare Fields in Your Code
Manage Code Segment Size
Limit RDD Memory
The Garbage Collector
Managing DGROUP Memory
Monitor Memory with //INFO
Create a .MAP File
Managing the EVAL Stack
Use LOCAL Arrays
Pass Arrays instead of Memvars
Expand the EVAL Stack Size
Using a Lint Checker
Program Architecture
TBrowse Memory
The Runtime Environment
Expanded Memory
Run Memory
Windows and Clipper
Using Video RAM
OS/2 and Clipper
Linking Techniques
Dynamic-Overlaying of Clipper-Code
Internal/External Overlays
Overlaying C/ASM Code
Using Expanded RAM for Overlays
Using UMB for Overlays
Overlaying Modules
Overlaying Libraries
Speed Optimization
Overlay Caching
Overlay Pool Size
Overlay Stack Size
DOS Extenders
Resurrect your 286s
Protected Mode = Protected Code
ExoSpace
Hardware
Memory Requirements
Operating Systems
Blinker
Dual Mode
Dual Mode Caveat
CauseWay
ClipXMS
Hints for CA-Clipper 5.3
Conclusion
Introduction
There is no way to escape from having to deal with memory problems in your CA-Clipper
applications. There are several axioms that I have learned to respect over the years,
the first being "all hard-disks will eventually fail", and the second being "no amount
of memory is sufficient".
In the late 70's I developed terminal emulations using the 8080 and Z80 processors. These processors had the capability of accessing 64k of total memory. I recall wondering why I would possibly ever need this much memory. In fact, our most complicated terminal emulations were accomplished using 16k of ROM and another 16k of RAM.
When the IBM computer was introduced in 1981, it was incomprehensible that the default 512k of memory offered would not be sufficient to meet our needs, but here we go again, fighting this constant battle of wits verses bits.
Managing memory in CA-Clipper can be accomplished by using proper programming techniques, linking techniques, and setup of your DOS, Windows or OS/2 memory environment.
This section describes how the CA-Clipper 5.x Virtual Memory Manager uses conventional memory, expanded memory and disk space to store both data and CA-Clipper code. We will examine the PUBLIC, PRIVATE, LOCAL and STATIC variable classes, storage of memory variable values of different types, and how the dynamic paging system manages CA-Clipper 5.x code at application run time.
The level of expertise of the reader is expected to be medium to high, assuming an in-depth knowledge of CA-Clipper programming, and a good knowledge of PCs, networks and programming techniques in general.
Expanded memory is memory which is also accessible on all PC compatibles. Programs which are to use expanded memory have to be explicitly written to do so. Expanded memory may be provided in the form of hardware or software emulation, and in later versions of the specification, known as EMS, is limited to 32 MB. It is managed in pages, typically 16 kb in size, which may be brought into an area of conventional memory to be accessed by a program. The currently executing program will request a particular page of expanded memory from the expanded memory manager, and will provide an address in conventional memory at which to place the page. On return from the manager, the data contained in the requested page can be read or written to as if it were permanently resident in conventional memory.
Extended memory is memory accessible by the 80286 and later processors, and exists outside of the 1 Mb range of conventional memory. These processors can access extended memory directly when running in one of their enhanced modes. When running in an 8086 emulation mode, however, a programming interface to extended memory is necessary, and a number of these have been specified. The most widely used of these interfaces is known as the XMS specification.
Software memory managers will often manage extended memory and provide both EMS and XMS programming interfaces to it for maximum versatility.
Virtual memory is a technique which has been used for many years to enable programmers to write programs requiring more memory than is directly available on the destination machine. The technique provides a simple interface to memory for storing and retrieving code and data, whilst hiding the fact that the information may be stored on one or more alternative devices until it is needed again.
The virtual memory manager, which may be implemented in hardware, software, or a combination of the two, monitors the frequency and duration of usage of the information, and decides where to keep each piece of information for maximum overall performance of the system. Typically, the least recently used information will be saved out to slower devices, while the more recently or more often used information will be kept in fast, real memory.
CA-Clipper currently makes no direct use of extended memory, so if the application will be running on a 386 PC or above, then obtaining a memory manager such as QEMM, 386MAX or the one supplied with MS DOS 6.0 will be a good investment.
Once a CA-Clipper application has loaded into memory and started executing, the application allocates the remaining real memory according to parameters set with the CLIPPER environment variable or the // command line options. The format of these is the same, and consists of the //, the letter or group of letters denoting the area, e.g. E for EMS, a ':' and a number indicating the size in Kb to be used for that area.
The parameters controlling allocation of memory are X:nnn and E:nnn. The X parameter specifies how much conventional memory to eXclude from use by CA-Clipper, and takes a value from 0 to 256 kb. The E parameter specifies how much expanded memory to allocate to the VMM, and takes a value from 0 to 8192 kb.
For example : TEST //E:1000
which would limit CA-Clipper to using 1 Mb of expanded memory.
It is worth noting in passing that the default of all available EMS up to a maximum of 8MB, or the E value if one is specified, is allocated to the VMM in one block at the start of the application. This means it is not available to any other part of the system until the application terminates and the memory is freed. In addition, the application could run out of conventional memory if there is too much EMS available to it, since a table of proportional size to the amount of EMS used is allocated in conventional memory. Depending on the amount of data manipulated by the application, a suitable maximum value may be 1000 to 2000, representing 1 - 2 Mb of expanded memory.
The other CA-Clipper parameters relevant to the VMM are the SWAPPATH:'path' and SWAPK:nnn parameters. If the application's conventional memory and EMS memory is fully utilised then the VMM will create a temporary swap file in the directory indicated by the SWAPPATH parameter, or in the current directory if no SWAPPATH is specified. This disk file will be used to store the least recently used data owned by the VMM, and will gradually increase in size until either the application has terminated and the file deleted, or the size limit set by the SWAPK parameter has been reached. The default size limit for the swap file if no SWAPK parameter is specified is 8 MB.
Virtual memory as managed by the VMM is allocated in segments, each of which may contain from 1K to 64K of data. When memory is allocated from the VMM, instead of returning a pointer to real memory it returns a form of segment number to identify the segment, in the same way as DOS returns a handle when a file is opened. Whenever the data within the segment is needed, a request is made to the VMM to return the current location of the segment in real memory where it can be read or written to.
Initially all the segments will be located in real memory, and because each segment is movable, real memory can be organised efficiently by filling up the gaps as segments are freed. Once real memory fills up, the VMM will swap out least recently used segments to EMS if it is available, or disk if not, to make room for new segments. If those segments are used at a later stage in the program, the VMM will swap out other segments to make room and bring the original segments back in, in the same way as an overlay manager manipulates code overlays.
CA-Clipper 5.x also contains a special type of memory manager designed to manage complex data values such as character strings and arrays. The CA-Clipper 5.x object memory is called the Segmented Virtual Object Store (SVOS). SVOS uses virtual memory managed by the VMM to store data values, including character strings, arrays, and dynamically created (macro-compiled) code blocks.
SVOS provides two important functions beyond the basic capabilities offered by the VMM, memory compaction and garbage collection.
Memory compaction consists of automatically compacting stored values on an ongoing basis. This eliminates fragmentation of the virtual memory and reduces swapping, since each segment can be fully utilised before requesting further segments.
Some CA-Clipper 5.x values (e.g., arrays) may be referred to by several program variables or array elements at the same time. The garbage collection routine automatically reclaims space occupied by values which are no longer accessible through any variable or array. By default, this occurs in background when CA-Clipper is in an idle state, e.g. waiting for keyboard input.
The real memory remaining to the VMM is set up as a swap area to bring swapped out pages of data into memory for use in the CA-Clipper program. When a RUN command is performed, as much of the top of the swap space as possible is cleared and returned back to DOS to be combined with the X area, and then the RUN command is issued. In this way more memory is freed up for RUN commands than would have been with Summer '87, although the exact amount will depend on the size and usage of the lower end of the swap space.
Because of this dynamic nature, at run time CA-Clipper requires more information about variables and procedures than traditonal lower level languages such as Pascal, C and Modula 2. Some of this information is available at compile and link time, such as the name of the variable, but some of it, such as its type, will only be available once the application has started executing.
For these reasons, each CA-Clipper .OBJ file is created with a symbol table of 16 bytes per symbol, and all code in the .OBJ file refers to that symbol table. At run time the symbol table entry is used to point to the control information and value or code for the symbol. The symbol table is created in its entirety in the root of the application, and can grow to upwards of 64 kb, so it can significantly affect the amount of conventional memory required by the application. This is why even 100% overlayed applications grow when code is added.
CA-Clipper 5.x introduced static and local variables to the language to encourage better and more efficient coding practices. Another important benefit is that these classes of variables do not require a symbol table entry as they cannot be accessed via macros. Changing as many PUBLIC and PRIVATE variables as possible to LOCAL or STATIC variables can therefore significantly reduce the amount of conventional memory required.
The major linkers now available remove the duplicate symbols from the symbol tables in the various .OBJ and .LIB files at link time, creating one large consolidated symbol table. This process, known as symbol table compression, can significantly reduce the run time memory requirement of the .EXE, leaving more memory for the application's data and overlays. All the duplicates are removed except the symbols belonging to procedures declared as static, since these are local to each .OBJ and will have different code associated with each occurrence of the symbol.
It is worth noting that prior to link time symbol table compression, the only way to reduce the number of duplicate symbols was to minimize the number of .OBJ files, but this is no longer necessary.
For example, in the code:
FUNCTION T A = B + C
we would have a symbol table containing:
T A B C
and the tokenised code would consist of (in simplified terms):
Take symbol 2 (B) Take symbol 3 (C) Add them together Store result in symbol 1 (A)
This tokenised approach has a number of advantages over true compiled code, with only a neglible cost in performance. The code produced is very compact, for example taking only three bytes for a procedure call, as opposed to five for a direct call. It is also very self contained. All external references go via the symbol table, so operations such as incremental linking are made significantly easier. This approach also makes it possible to use the dynamic paging system described below for faster overlayed applications with lower memory requirements.
The size overhead of a simple CA-Clipper compiled .EXE is actually made up of the runtime routines from the CLIPPER.LIB which are called by the processing of the tokens. The apparently large size of even a "Hello world" type program is due to the potential for macro operations, which could execute just about any CA-Clipper command from even a two line program.
Instances of variables and their values
In conventional languages the scope, size and type of a named variable is known at
compile time, so the exact amount of space can be reserved for it in memory at run time.
This memory will always be used to store the value, no matter how often the value is
changed.
The remaining memory above the program's .EXE image is usually known as the heap and is managed by a heap manager, which will allocate blocks of memory of varying size to the program as and when requested. Space for data allocated dynamically at run time, for constructs such as linked lists or buffers, whose sizes are not known at compile or link time, will be located from and returned to this heap.
With CA-Clipper, determination of the type and size of all variables and the scope of public and private variables is left until run time, so a more complicated mechanism for storing the values of variables is required.
CA-Clipper 5.x offers several different storage classes for program variables, depending on how they are declared and used in the program. LOCAL and STATIC variables are stored in a dedicated area of real memory, as described below. PRIVATE and PUBLIC variables, known as MEMVAR variables, are created and destroyed dynamically while a program is running, and are stored in VM segments.
For performance reasons, these segments remain locked in real memory during most operations except memory intensive operations and RUN commands. Each MEMVAR uses 20 bytes in a VM segment, so converting PRIVATE and PUBLIC variables to LOCAL and STATIC variables can reduce memory requirements for some applications.
At run time, each instance of a variable is allocated a value, which is represented internally as a data structure called a VALUE. The contents and format of a VALUE differ depending on the type of data it represents. Simple data, such as integers, are stored directly into the VALUE. Larger items, or data of variable length such as strings or arrays, have a "reference" to the string or array stored in the VALUE, and the actual data is stored elsewhere. Internally, CA-Clipper is organized as a stack based machine which uses an area of memory called the Eval Stack to contain temporary variables such as function parameters, intermediate results of expressions and local variables. The Eval Stack is simply a contiguous group of VALUEs that are accessed as a stack, in the same way as the processor stack is used by C programs.
For example, in a CA-Clipper function call, parameters are pushed onto the Eval Stack before the function is executed. The function operates on the top-most items in the Eval Stack and produces a result. After the function completes, the parameter values are popped from the Eval Stack and replaced with the function result.
Each entry in the Eval Stack, i.e. each VALUE, occupies 14 bytes, and for complex data types such as character strings, arrays and code blocks there will be an additional memory requirement handled by the VMM where the actual value is stored.
The Eval Stack is allocated from the default data segment, defined as the start of the group DGROUP, when the program starts executing, so initialisation will fail if DGROUP is too full. This is not usually a problem with pure CA-Clipper applications, but if a number of third party libraries are linked in to the application it may possibly fill up unless they have avoided storing data in DGROUP. The number of kb remaining in DGROUP for CA-Clipper's use can be examined by executing the program, with the //INFO parameter, and the amount of conventional and expanded memory available will be displayed at the same time.
LOCAL variables are the simplest variables, and are allocated as locations within the Eval Stack to store their VALUEs. To manipulate a LOCAL variable, the system simply copies the variable's VALUE from one position in the Eval Stack to another. Local variables are visible only within the current procedure or function, and are created automatically each time the procedure in which they were declared begins executing. When that procedure terminates through a return, all it's LOCALs are removed from the Eval Stack and any associated VMM memory freed up.
STATIC variables are similar to LOCAL variables, but have a duration of the lifetime of the application. Because of their permanence, they are allocated as fixed locations at one end of the Eval Stack, but are manipulated in the same way as LOCAL variables simply by copying their VALUEs. This means that every STATIC variable in the system also requires 14 bytes on the Eval Stack in DGROUP, which is another reason for C and ASM programmers to avoid storing data in DGROUP.
PRIVATE and PUBLIC variables are more complex than LOCAL or STATIC variables because in addition to an associated VALUE they also have a name which may be referred to during execution of the program via a macro or its equivalent. MEMVAR variables are allocated locations for their VALUEs in dedicated VM segments and these locations are stored with their names in the symbol table. When a MEMVAR is manipulated, the symbol table entry is used to point to the VALUE which can then be placed on the Eval Stack in the normal way.
FIELD variables differ from the other storage classes because they have no memory location at all, since their values are stored in a database record buffer. To manipulate a FIELD, the system generates a request to the file's database driver, which then creates an appropriate VALUE to be manipulated on the Eval Stack.
An array VALUE contains a reference to the array rather than an actual value, so when an array is assigned to a variable, the system simply overwrites the variable's VALUE with a new VALUE containing a reference to the array. The array itself is simply a group of VALUEs stored in virtual memory, where each element of the array is a VALUE. Any VALUE can contain another reference, so multidimensional arrays are created by having each element refer to another array rather than have an absolute value. When values are assigned to array elements, the VALUE for that element is updated. When an array is assigned to another variable, only a copy of the VALUE referring to the array is made, and the array data itself is not duplicated.
A character string VALUE contains a reference to the character data, which is stored elsewhere in the VM. As with arrays, assigning a character value to a variable simply overwrites the variable's VALUE with a new VALUE containing a reference to the character data. In a similar way to arrays, assigning a character value from one variable to another simply duplicates the VALUE (i.e., the reference to the data). The character data itself is not duplicated.
This reference-based memory management technique is the same for strings, arrays, and code blocks. CA-Clipper's garbage collector monitors references to objects, and when there are no longer any references to a particular piece of data, the space occupied by that data is automatically reclaimed.
The remaining case of creating a new variable is handled by adding a new entry to the end of the symbol table. This new entry will have the name of the variable filled in, along with a pointer to a VALUE for the symbol, and will be used from then on to refer to the variable.
Both Summer '87 and CA-Clipper 5.x provide other mechanisms to avoid the creation of these dynamically named variables in the majority of circumstances, such as using an array of elements to store the values, or using code blocks in 5.x. These alternative mechanisms should be used wherever possible, if only because macro operations are inherently very slow, as each name in the symbol table has to be checked until a match is found before execution can continue.
If the use of a macro cannot be avoided, but the name to be created will be one of a known set, then these names should be mentioned explicitly somewhere in one of the programs. The code does not ever have to be executed, but just using the names causes them to be added to the symbol table at compile time, thus avoiding the above situation.
Because the code block consists of normal tokens, it will include references to the symbol table, so the equivalent symbol table must be available when the code block is actually evaluated. This is one of the reasons why it will prove difficult (but not impossible) to save code blocks in a database from one application and restore and evaluate them at a later time in the same or another application.
During linking all CA-Clipper modules are broken down into pages of 1 kb in size. These pages are stored either in the executable file or in separate overlay files. The manipulation of overlays in these 1 kb pages removes the effect the size of compiled functions or modules has on the memory required to load the overlay. Large modules are broken into multiple pages, and small functions are grouped together in a single page.
At execution time, CA-Clipper 5.x's dynamic overlay manager loads pages based on information embedded in the .EXE by the linker. The dynamic pages are loaded into VM (Virtual Memory) segments, allowing the VMM to manage the overlay pages on a competitive basis with other uses of memory such as the application data.
The paging architecture allows the system to discard low-use sections of code even if the code is still active, and reload it only when control returns to that piece of code. Code pages which are being heavily used are maintained in memory by the VMM's LRU swapping policy.
When possible, the VMM will place dynamic overlay pages in expanded memory, reducing overlay reads. Overlay pages are never written to the VMM disk swap file, however. If a VM segment containing an overlay page is to be removed from memory altogether, it is simply discarded. If it is needed subsequently, it is re-read from the overlay file. In addition to virtual memory, the dynamic overlay manager uses a dedicated area of real memory to cache the most active dynamic overlay pages.
This page mechanism is made possible by the nature of the CA-Clipper code. As explained before, it is not actually code but a series of tokens which are processed at run time. This means that the __PLANKTON procedure from CLIPPER.LIB which is processing the tokens can detect when it has reached the end of a page and request the next one to be loaded. All CA-Clipper code is therefore overlayable, so there are no restrictions on which CA-Clipper .OBJs can be placed in the overlay area. It should be noted that linkers which use the dynamic paging mechanism of CA-Clipper 5.x automatically overlay ALL CA-Clipper code unless directed otherwise.
CA-Clipper includes an automatic memory manager referred to as the Virtual Memory Manager or VMM. There is very little the CA-Clipper programmer needs to know about the VMM system other than it requires EMS memory or available disk space for the creation of swap files. After CA-Clipper uses up the available EMS for VMM, it will start creating swap files.
In a network situation with no local hard drive or ram drive, this can slow down the application, therefore it is recommended that you set up your environment to allocate from 500k to 1 meg of EMS for average Clipper applications. You may monitor CA-Clipper's usage of EMS during your application by reporting the remaining EMS with the MEMORY(4) function. If the number reported falls below 100, then it is recommended that you increase available EMS by several hundred K.
If no EMS is available, then Virtual memory can be allocated from XMS memory by using a Third-party product named ClipXMS. See the section titled DOS Extenders for more information on this product.
To give the CA-Clipper application more conventional memory, you must start at DOS with ample conventional memory. After you load device drivers, network drivers, mouse drivers, TSR's, etc. your environment just may not have sufficient conventional memory remaining to run your Clipper application. The new generation of expanded memory managers are designed to not only create EMS/XMS memory but they also will load drivers and TSR's into the upper memory blocks (UMB) area of memory above 640k so they will not use up valuable conventional memory. I have used QEMM and 386MAX in the past to accomplish this task but have recently found that, on some systems, the MEMMAKER.EXE utility and the EMM386 driver supplied with DOS 6.x are adequate in creating a better memory environment.
Symbols cannot be overlayed or swapped to VMM, therefore it is important that you program in a manner consistent with producing the smallest number of symbols in your CA-Clipper-compiled objects. Here are some tips for reducing the symbol table size in your applications.
Use Constants instead of Memvars
All PRIVATE, PUBLIC and STATIC CA-Clipper memory variables are treated as "symbols".
Refrain from using a memory variable if a constant is sufficient. For example, an
unnecessary symbol can be eliminated by changing the code:
nEscapeKey := 27 DO WHILE INKEY() # nEscapeKey * CA-Clipper code ENDDO
to:
DO WHILE INKEY() # 27 * CA-Clipper code ENDDO
or:
#define K_ESC 27 DO WHILE INKEY() # K_ESC * CA-Clipper code ENDDO
Use Arrays Instead of Memvars
Every different CA-Clipper PRIVATE, PUBLIC, or STATIC memvar name creates a "symbol",
whereas an array name creates only ONE symbol. The following example shows how to save
considerable memory in a CA-Clipper application by reducing the symbol count with an array.
This code produces 5 symbols:
PRIVATE cName := customer->name PRIVATE cAddress := customer->address PRIVATE cCity := customer->city PRIVATE cState := customer->state PRIVATE cZip := customer->zip @ 1,1 SAY 'Name ' GET cName @ 2,1 SAY 'Address' GET cAddress @ 3,1 SAY 'City ' GET cCity @ 4,1 SAY 'State ' GET cState @ 5,1 SAY 'Zip ' GET cZip READ
This code produces 1 symbol:
PRIVATE aGets[5] aGets[1] := customer->name aGets[2] := customer->address aGets[3] := customer->city aGets[4] := customer->state aGets[5] := customer->zip @ 1,1 SAY 'Name ' GET aGets[1] @ 2,1 SAY 'Address' GET aGets[2] @ 3,1 SAY 'City ' GET aGets[3] @ 4,1 SAY 'State ' GET aGets[4] @ 5,1 SAY 'Zip ' GET aGets[5] READ
Some programmers choose memvars over arrays because it makes their source code more readable. Source code that refers to hundreds of array elements can look very cryptic and can be hard to maintain. To reconcile this problem, use the CA-Clipper pre-processor to create a "symbolic" reference to each array element like in the following example.
STATIC aGets[5] #define cNAME aGets[1] #define cADDRESS aGets[2] #define cCITY aGets[3] #define cSTATE aGets[4] #define cZIP aGets[5] cNAME := customer->name cADDRESS := customer->address cCITY := customer->city cSTATE := customer->state cZIP := customer->zip @ 1,1 SAY 'Name ' GET cNAME @ 2,1 SAY 'Address' GET cADDRESS @ 3,1 SAY 'City ' GET cCITY @ 4,1 SAY 'State ' GET cSTATE @ 5,1 SAY 'Zip ' GET cZIP READ
Use the Same Name Memvars whenever possible
Again, every "different" PUBLIC, PRIVATE or STATIC CA-Clipper memvar in a module creates
a symbol. If an object contains several procedures, use the same name for memvars even
though they may not perform the same or similar functions. For example, procedure A and
procedure B both need 5 memvars. If procedure A declares its memvars with 5 unique names
and procedure B declares its memvars with 5 unique names, then 10 symbols are used in the
linked application. To eliminate 5 symbols, make sure that procedure B assigns the same
name to the memvars as procedure A. This is not possible of course, if the memvars need
to be PUBLIC to both procedures and perform different functions, only if they are PRIVATE.
Use Complex Expressions instead of Memvars
The following three lines of code represents the method that most CA-Clipper programmers
choose to accomplish most programming tasks. It makes sense to code this way for
readability and debugging, but if you are writing a very large application, the complex
expression technique can save some memory. The following three lines of code will read
the disk file READ.ME into a memvar named cReadFile, save the changed code into a file
named cEditFile, then write the changed code back to the disk file READ.ME.
cReadFile := MEMOREAD('READ.ME')
cEditFile := MEMOEDIT( cReadFile )
MEMOWRIT('READ.ME', cEditFile )
These three lines of code can be replaced by one complex expression which uses no symbols at all.
MEMOWRIT("READ.ME", MEMOEDIT(MEMOREAD("READ.ME")))
An additional advantage to coding this way is that less free-pool or VMM memory is used because the process of temporarily storing the text in cReadFile and cEditFile is completely eliminated. If you find that creating complex expressions such as this are unreadable and hard to maintain then use the CA-Clipper pre-processor to accomplish the same task as follows:
#define cEditFile;
MEMOREAD('READ.Me')
#define cReadFile ;
MEMOEDIT(cReadFile)
MEMOWRIT('READ.ME',cEditFile)
The above code is nearly as readable as the original three lines of code but will not create any variables at compile time. Try compiling this code with your CA-Clipper compiler and use the /P switch to write the pre-processed code to a .PPO file, then look at the .PPO file to see what is actually compiled.
Use LOCALS Instead of PRIVATES
Sometimes it is just not possible or practical to write code without using symbols, so if
you find yourself in this situation, CA-Clipper provides the feature of "LOCALIZING"
symbols to the code segment which is currently being executed rather than placing the
symbol in the main symbol table.
LOCAL symbols are effectively "overlayed" because they are treated as part of the code segment rather than given a place in the main symbol table. Not only does this save valuable memory but it also improves speed performance because the public symbol table does not need to be searched each time a LOCAL symbol is referenced in your code. Of course, if the symbol you are referencing is needed for the entire application or is used in a macro, then it must be declared as PRIVATE or PUBLIC.
Symbols which are not declared at all are automatically assumed to be PRIVATE, so make sure you use the LOCAL declaration for all symbols in your code which you do not want to end up in the main symbol table. In the previous code example, the 5 PRIVATE memvars consume 110 bytes of memory, the single PRIVATE array consumes 22 bytes of memory, whereas 5 LOCAL declarations would consume 0 bytes of memory.
Use STATIC functions instead of PUBLIC functions
Every PUBLIC function and procedure name will occupy 22 bytes of the symbol table. Don't
make functions and/or procedures PUBLIC unless they need to be called from anywhere
within your application. STATIC procedures and functions are called only from within the
source code module in which they are declared thereby eliminating the need to place the
function name into the public symbol table.
Use PSEUDO Functions or CONSTANTS instead of PUBLIC Functions
A pseudo-function is a function that does not really exist in the compiled code but
exists only in your source code. Many programmers will use a large number of functions
with different names even when each function may do something very similar like a
conversion or a table lookup. I have seen a lot of code that looks like this:
SetColor( blueonwhite() + "," + redongreen() + "," + black() ) FUNCTION blueonwhite RETURN 'B/W' FUNCTION redongreen RETURN 'R/G' FUNCTION black RETURN 'N/N'
Programmers will use this technique to make their code easy to read and maintain without understanding how this can bloat the size of the application. Probably the best technique to replace the above code would be to #define constants for each as follows:
SetColor( BLUEONWHITE+ "," + REDONGREEN + "," + BLACK ) #define BLUEONWHITE "B/W" #define REDONGREEN "R/G" #define BLACK "N/N"
Your existing source code may have "many" references to public functions and you don't want to risk changing all the function names to constants, or maybe the functions return something a little more complex than can be handled by a simple constant. In this case, you can #translate the functions into psuedo-functions as follows:
BEFORE:
FUNCTION BLOBget( nPointer, nStart, nCount )
RETURN dbInfo( BLOB_GET, { nStart, nCount } )
FUNCTION BLOBput( nPointer, xBlob )
RETURN dbInfo( BLOB_PUT, { nPointer, xBlob } )
FUNCTION BLOBExport( nPointer, cTargetFile, lMode )
RETURN dbInfo( BLOB_EXPORT, { nPointer, cTargetFile, lMode } )
AFTER:
#xTranslate BLOBget( a, b, c ) => dbInfo( BLOBGET, {a,b,c} )
#xTranslate BLOBput( a, b ) => dbInfo( BLOBPUT, { a, b } )
#xTranslate BLOBExport( a, b, c ) => dbInfo( BLOBEXPORT, { a, b, c } )
In the BEFORE example above, three PUBLIC functions were created simply to provide a better way to call the same dbInfo() function, whereas in the AFTER example, no PUBLIC functions were needed to accomplish the exact same task. Not only did we save 3 symbol table entries but we also eliminated 3 LOCAL variables that get pushed onto the EVAL stack. Of course it must be remembered that since psuedo-functions don't actually exist they cannot be used in macros or index keys.
Don't use STATICs or PRIVATEs to pass parameters
Many CA-Clipper programmers will assign a variable as PRIVATE or STATIC so it can be
accessed and changed within multiple procedures in the same source code module. Variables
should be STATIC only if their value needs to be maintained throughout the program, not
for the convenience of eliminating the need to pass parameters. By using the
pass-by-reference symbol "@" you can change the value of LOCAL variables anywhere within
your calling program as shown by the following example.
LOCAL cName, cAddress
MyFunction( @cName, @cAddress )
Return( { cName, cAddress } )
STATIC FUNCTION MyFunction ( cName, cAddress )
cName := "CA-Clipper"
cAddress := "New York, NY"
Return( nil )
The "Myth" of PUBLICs versus STATICs
There is no memory advantage to using a STATIC variable in lieu of a PUBLIC variable.
Both types of variables use up permanent space in conventional memory.
A STATIC variable will use space in DGROUP, whereas a PUBLIC variable will use space in the Symbol Table. In fact, there are situations in which a good case can be made for using a PUBLIC array instead of a STATIC array. Take the example of a system-wide color system in which the colors are defined in an array. If the array is STATIC, then extracting the information from the array requires a call to a PUBLIC function which exists in the same source code module as the array definition. This would actually add more symbols to the symbol table and take more processor time than if the color array were defined as PUBLIC thereby allowing direct access to the array without the need for the public function. A STATIC array may be a wiser choice in cases where the application is a library which is incorporated into another CA-Clipper application, thereby eliminating any possibility of symbol conflicts with the application.
The Affect of Databases on Memory Usage
Many programmers have taken an affection to "data-driven" programming to speed up the
development of custom applications. Since CA-Clipper evolved as an X-Base type language,
the common approach to data-driven programming is to create an "engine" or "kernel" .EXE
program that retrieves "custom" information about the application from a set of database
files. The more sophisticated the system, the more data files are required for
configuring the application. It is important to understand the impact of databases on
memory usage when designing applications that use many databases.
When a database is opened with the USE command, all the field names are placed into the public symbol table. This allows database field names to be used in expressions in the same manner as memvars. Because this symbol table is PUBLIC, the field names will remain in the symbol table, even after the database is closed. CA-Clipper employs no mechanism to remove symbols from the symbol table, only to add them. As each database is opened the symbols are added, thereby reducing the remaining available root memory ( MEMORY(0) ). It is conceivable that an application could run out of memory if many databases are opened and closed. Fortunately, symbols of the same name are "reused" in the symbol table, so if you open and close the same database many times, the symbol memory space will not be reduced. Keeping this in mind, it is a good practice to use databases with fields of the same name.
To improve speed operation, some programmers will open data-dictionary databases at the start-up of the program, load the custom program configuration into arrays, then close the databases. This action will reduce the amount of memory needed for data structures and file buffers and lower the number of DOS handles required, but the symbols will still be added to the symbol table even though they may never be accessed by the program. A data-driven application in which the databases are used only during the start-up of the application could be re-designed to convert the database information to a text file to generate a "runtime" version of the application that will load the array(s) from a text file rather than databases, thereby eliminating the symbol-table problem.
Declare FIELDS in your Compiled Code
To prevent the need to add Field Names to the symbol table at runtime, it is always a good
idea to declare fields to the Compiler by using the FIELD <fieldname> statement in
your source code. This will insure that the symbol table memory is allocated at the time
the application is linked into an executable program rather than during the running of the
program.
CA-Clipper-compiled P-Code is loaded in fixed-size pages so this is not a requirement when programming in CA-Clipper.
When a large segment of memory is required by the application and is not available, Clipper has to try to move around segments that are not "locked" so it can get sufficient memory for the routine that's being called. This is "garbage collection". Unfortunately, symbol memory and other memory allocation are "locked" and cannot be moved. Applications should take into consideration reducing the amount of fixed heap, stack, and VMM memory that gets allocated during the running of the program, and also to insure that garbage collection is forced on a methodical basis to prevent the fragmention.
CA-Clipper uses a technique called "scavenging" in its memory garbage collector. The garbage collector is automatically invoked during writes to the screen and "wait" states while waiting for keyboard input. Programs that have routines which have few screen-write or input routines can fragment memory badly particularly if a lot of file opening and closing, reindexing, appending, etc. is going on with little operator interaction.
It is recommended that you put calls to Devout("") in your code to invoke the garbage collector in the event that you experience "Conventional Memory Exhausted", "Memory Low", or "Stack Eval" errors. My own experience with this method has not always given me desirable results, however, and I have found that placing calls to MEMORY(-1) in strategic locations in my code, such as just before opening databases, or creating large arrays actually produces better results. This method is not supported by CA or many other developers so use this recommendation with caution. Another popular method of perfoming garbage collection is to use the FT_IDLE() function from the public domain NANFORUM TOOLKIT available on many BBS's including the CLIPPER forum of COMPUSERVE. Garbage collection should be forced especially before opening databases and indexes.
Assembly-language programmers like to store data in the DGROUP area to improve performance of their application. Data in this area doesn't get overlayed and it is insured that it will always be accessible regardless of the state of the application. Interrupt handlers always store data in DGROUP to insure that the data will be accessible in the event of an interrupt. CA-Clipper stores it's stacks and static values in DGROUP.
The eval stack is where local and private variable VALUEs are stored. The CA-Clipper static area is where CA-Clipper static VALUES are stored. ITEMs generated by programs using ITEM.API also create VALUE entries in the CA-Clipper static area. Each VALUE entry uses 14 bytes. If the eval stack grows into the CA-Clipper static area (or vice versa), you get a UE 667 (stack fault) error. In low memory situations, the VMM will allocate "space" to the conventional memory pool. After this happens, if the eval stack grows into the allocated space, you get a UE 668 error. Likewise, if the CA-Clipper static area needs to grow, you get a UE 669 error. When the eval stack or CA-Clipper static area expands, and later retracts, the system maintains "watermarks" to indicate the farthest expansion of them. The VMM allocates only the space between the "watermarks". So one way to control things is to make sure the "watermarks" are set to allow your program to execute normally.
There is very little that a CA-Clipper programmer can do to resolve DGROUP problems other than to avoid using third-party products and/or C/ASM code that uses SMALL MODEL rather than LARGE MODEL programming techniques and replace LOCAL variables with LOCAL arrays. CA-Clipper is a LARGE MODEL programming language and it is recommended in the CA-Clipper API that extensions to the language should also be compiled as LARGE MODEL. Unfortunately, many C/ASM programmers develop libraries designed for speed performance rather than memory performance. I refer to these products as "DGROUP Hogs". These libraries will consume so much of the C/ASM static space that there will be literally nothing left to run the application. If you find that you cannot limit the usage of third-party libraries, then and alternative solution is to use one or more of the methods described below under "Managing the EVAL Stack".
There are several methods to determine how much DGROUP a library uses.
Monitor memory with //INFO
When you start your CA-Clipper application with the //INFO option, CA-Clipper reports the
condition of memory at the start of the application. The reports looks like this:
* DS=4F4E:0000 DS avail=30KB OS avail=255KB EMM avail=960KB * DS=is the starting address of the DGROUP segment. * DS avail= KB reports the amount of DGROUP available. * OS avail= KB reports the amount of conventional memory available. * EMM avail= KB report the amount of expanded memory allocated to the current application.
When the DS avail (DGROUP) is less than 15k at the start-up of an application, some large applications could experience stack eval errors at runtime. This number is arbitrary and your application may actually run just fine with less available DGROUP, however it is important that you monitor DS especially if your application uses third-party libraries or lots of LOCAL and STATIC memvars. It's always a good idea to make note of the amount of DS avail in your current application before and after you add new functions or libraries to the application. If you suddenly notice a dramatic decrease in DS avail after adding new code, then you should take note that you may be using a function from a library that uses excessive DGROUP.
Create a .MAP file
Blinker supports a MAP=<file> command that creates a map file with information about
segment usage. This map file will show you which routines use DGROUP and exactly how much
they consume.
Use LOCAL ARRAYS instead of LOCAL VARIABLES
Every time a new function is called, all LOCAL variables are "pushed" onto the EVAL stack.
They are "popped" off the stack when returning to the calling program. If a procedure or
function is called recursively, then it can be quite easy to blow up the eval stack with
a 667 Stack Eval Error.
For example, I had this problem in one of my larger applications due to the fact that programs were deeply nested via calls to a common set of menu functions. I found that my main menu function could blow up the stack if called recursively as few as 5 times. Each time this function was called, 85 LOCAL variables were pushed on to the eval stack. I completely eliminated the problem with about 2 hours of programming and debugging by replacing the 85 separate LOCAL variables with 1 LOCAL array. A LOCAL array uses the same amount of stack space as any other LOCAL variable, therefore I reduced the amount of values pushed on the stack from 85 to 1. After doing this, I could not blow up the stack even if I called the menu system recursively more than 30 times. This was accomplished quite easily with very minor code changes.
Here is an example of the BEFORE and AFTER code.
BEFORE:
FUNCTION DC_MenuMain() LOCAL cColor, cMenuScrn, nInkey, ; cTempStr, nTempNum, nItemLen, ; cOldColor, nItems, nMsgLen, ; nHotStart, cBoxColor, ; cMenuColor, cHotColor, ; cSelColor, cBuffer, nRow, nCol, ; lMouseHit, nSaveRow, nSaveCol, ; lMessage, cGrayColor, ; nMouseStat, nStRow, nStCol, ; nEnRow, nEnCol, lMouVisible, ; nPass, aSubMenu, aSubBlock, ; nElement, aMenuItems, ; aHotKeys, aMenuBlocks, ; aSubItems, aBlockItems, cType, ; nStart, cTitle, lBar, ; lReturnVal, lShadow, aMouseKeys, .... ..code...
AFTER:
FUNCTION DC_MenuMain() LOCAL aMenu := Array(85) #define cColor aMenu[ 1]; #define cMenuScrn aMenu[ 2] #define nInkey aMenu[ 3]; #define cTempStr aMenu[ 4] #define nTempNum aMenu[ 5]; #define nItemLen aMenu[ 6] #define cOldColor aMenu[ 7]; #define nItems aMenu[ 8] #define nMsgLen aMenu[ 9]; #define nItems aMenu[10] #define nHotStart aMenu[11]; #define cBoxColor aMenu[12] #define cMenuColor aMenu[13]; #define cHotColor aMenu[14] #define cSelColor aMenu[15]; #define cBuffer aMenu[16] #define nRow aMenu[17]; #define nCol aMenu[18] #define lMouseHit aMenu[19]; #define nSaveRow aMenu[20] #define nSaveCol aMenu[21]; #define lMessage aMenu[22] #define cGrayColor aMenu[23]; #define nMouseStat aMenu[24] #define nStRow aMenu[25]; #define nStCol aMenu[26] #define nEnRow aMenu[27]; #define nEnCol aMenu[28] #define lMouVisible aMenu[29]; #define nPass aMenu[30] #define aSubMenu aMenu[31]; #define aSubBlock aMenu[32] #define nElement aMenu[33]; #define aMenuItems aMenu[34] #define aHotKeys aMenu[35]; #define aMenuBlocks aMenu[36] #define aSubItems aMenu[37]; #define aBlockItems aMenu[38] #define cType aMenu[39]; #define nStart aMenu[40] #define cTitle aMenu[41]; #define lBar aMenu[42] #define lReturnVal aMenu[43]; #define lShadow aMenu[44] #define aMouseKeys aMenu[45]; .... ...code...
Pass ARRAYS instead of LOCAL Variables as parameters
Passed parameters are automatically defined as LOCAL by Clipper and are treated like any
other LOCAL parameter, therefore functions that pass a lot of parameters will eat up the
EVAL stack very quickly if they are called recursively. Use an array of parameters rather
than separate parameters if at all practical.
Expand the EVAL stack size
The linker supports a STACK or PROCEDURE DEPTH command to determine the size of the CPU
"Call Stack". If your application is structured with few recursive calls, then this stack
size can be lowered to give more memory to the EVAL stack. If your application generates
a 667 or 668 "Stack Eval Error", then you will want to decrease the CPU stack size. If
your application starts generating a 650 "Stack Eval Error", then you will want to
increase the CPU stack size.
You can also use fix_UE668(), a small procedure written by Robert Montgomery and placed into the public domain. The code has been included on the conference diskette (if you received this booklet at a software conference), is posted on the DONNAY Software BBS, or is available from the CLIPPER forum on COMPUSERVE in a file named FIX668.PRG. This code will expand the eval stack prior to allocation of "space" as conventional memory.
Examples of lint that will use unwanted memory are:
A popular lint checker for CA-Clipper is a product named "GROK" by John Kaster.
A procedural or hierarchical program is one in which the program consists of a main loop or main menu that parses control to sub loops or sub-menus then returns control back to the main menu. In this type of program, "temporary" memory is allocated during the invocation of the sub-routines and deallocated upon return back to the menus. The design of the menus and sub-menus will force the user of the program to return back to the menus before invoking a new task.
An event-driven or object-oriented program is one in which there is no main control loop but instead an "event manager". In this type of program, new instances of events or windows will create objects that allocate "permanent" memory because the memory is not deallocated when the object is "out of focus" but instead remains in suspension until the object is destroyed or the application is terminated. I have seen many poorly-designed event-driven programs in which the programmer failed to limit the "instantiation" of windows and events and the program eventually runs out of memory when the operator gets more prolific with his or her tasks.
Do not attempt to write a CA-Clipper program that is fully event driven and/or object oriented until you have a good grasp of CA-Clipper's usage of memory. The concept of multiple intantiation and inheritance is useful and powerful, but it has its drawbacks, mostly in the area of speed and memory performance.
If the user is simply paging up and down through a database, but not panning left or right, all the undisplayed columns in the TBrowse are updated anyway. It can be seen that designing a TBrowse system that has a hundred columns or more can allocate and access a considerable amount of SVOS (object) memory. In addition it takes a considerable amount of time to refresh the buffers in all the undisplayed columns, thereby slowing down the painting of the data in the TBrowse window. Programmers will design their TBrowse systems this way, however, because it is much simpler than trying to design a system that allocates memory better and also because it improves the speed of "panning" left or right. Since the undisplayed columns are always refreshed with the current data, they can be displayed very quickly when moving left or right. You should consider whether or not you want better performance when paging up and down or whether you want better performance when panning left and right. Most programmers I have discussed this with prefer the former model, so they have redesigned their Tbrowse systems to "dynamically" ADD new TBrowse columns to the right of their TBrowse windows when panning to the right and DELETE old TBrowse columns on the left. This of course, should be reversed when panning to the left.
The affect of this "dynamic" design is to dramatically reduce the amount of SVOS memory that is allocated. Columns will be created with TBColumnNew() and added to the TBrowse object only when they are scrolled into the visible window. Columns that are scrolled out of the visible window will be deleted from the TBrowse object and their memory reclaimed. A design like this may take a little more time to write and maintain, but it will provide great improvements in memory performance. If your application often returns a Memory Low error when running your TBrowse system, then you may need to make this design change.
I have discovered also, that TBrowse systems will fail whenever you try to add more than 200 TBColumn objects to a TBrowse object. Try opening a database with more than 200 fields and using the DBEDIT() function. You will most likely get a Limit Exceeded error.
If it is impractical to limit the amount of EMS memory on your computer, then you can at least limit the amount that the CA-Clipper application will use by the E parameter in your SET CLIPPER environment variable. For example, if your computer has 10 meg of EMS available and you want CA-Clipper to use only 2 meg of EMS, place the following command in your AUTOEXEC.BAT:
SET CLIPPER=E:2000
An alternative is to run the CA-Clipper program with the //E option as follows:
MYPROG //E:2000
The CA-Clipper MEMORY(4) function may be used to monitor the amount of remaining EMS that has not been used by the VMM system. You can use this function in your applications to see how efficiently CA-Clipper allocates memory when creating large strings, arrays, etc. You may want to set a hot key in your application to monitor memory while running sub-systems or menus to see how much memory is really being used by CA-Clipper. For example, if MEMORY(4) always reports more than 1000 (1meg) available, you may want to reduce the amount of EMS memory allocated to the application so this memory is available for other applications. This "memory-tuning" starts to play an important role when running in a multi-tasking environment like Windows, DesqView or OS/2.
If your application has been designed to require calls to other DOS programs, then you should invest in a third-party product that has been designed to give you the maximum amount of memory possible. The two most common products are the Overlay() function from the product OVERLAY or the SwpRunCmd() function from the product BLINKER. Since Blinker has already been discussed a lot in this book, it should be noted that this is another good reason to purchase this fine product. Both functions work nearly identically, however using the Overlay() function also requires linking with the OVERLAY.LIB library whereas using the SwpRunCmd() function does not require any changes at all to your link scripts.
These "swapping" functions basically capture the "conventional memory" image of the CA-Clipper application that is running and and save it to a temporary file. The memory is "released" and allocated to the new DOS command shell thus giving the called program up to 550k (or more) of available memory. The amount of memory that can be allocated to a called DOS program is usually about 20k less than the amount of conventional memory that was available before the CA-Clipper program started. When returning the CA-Clipper program, the memory image is restored from the temporary file and the file is deleted from the disk. Both products even provide the option of saving the memory image to EMS or XMS memory instead of a file.
Memory swappers have the responsibility of saving and restoring conventional memory and interrupts however they cannot reassign EMS or XMS memory. If you intend to use one CA-Clipper program to call another CA-Clipper program then you must make sure that the first program does not grab all the EMS memory for its VMM pool or there will be none available for the called program and it will run much slower. For example, let's say you have 4m of total EMS on your system and you want to make sure that both the parent and the child CA-Clipper programs have the same amount of memory. You would start up the parent program by using the //E option to insure it only uses 2m of EMS and leaves 2m available for the child program.
Example:
PARENT //E:2000
Memory Requirements
KB Required: -1 KB Desired: -1
EMS Memory :
KB Required: 0 KB Desired: 1024
XMS Memory:
KB Required: 0 KB Desired: 0
I am currently writing this document using a DOS editor in an OS/2 box on a network drive accessed via a PCMCIA network adapter and have my local hard drives "stacked" via Stacker for OS/2 and DOS. When I boot up under DOS (using QEMM 7.5), the same configuration gives me only 450k of conventional memory, whereas booting up under OS/2-Warp gives me 710k of conventional memory. This translates to an increase in Clipper MEMORY(0) from 50k to 300k for an average Clipper application.
All my Clipper applications are CUA-compliant and fully moused and look great when running under OS/2. I have found that OS/2 gives new life to Clipper legacy applications and basically eliminates the need to convert them to Windows. OS/2 provides conventional memory, DPMI memory, VCPI memory, XMS memory and EMS memory to each DOS window, therefore Clipper applications which have been designed to run under DOS will most likely run even better and more reliably under OS/2.
The subjects of this seminar are the three most commonly-used linkers available on the market today for linking CA-Clipper 5.3 applications. They are:
BLINKER
A "Real Mode" Third-Party linker published by Blink, Inc. This is the linker that is
bundled with CA-Clipper 5.3. A "Dual-Mode" version of Blinker is also available from
Blink, Inc.
EXOSPACE
A "Protected Mode" linker developed by CA and bundled with CA-Clipper 5.3.
CAUSEWAY
A third-party "Protected Mode" linker developed by Michael Devore for use with Clipper
5.2 and 5.3.
The CA-Clipper compiler has the responsiblity of converting source code to "machine-readable" code. This .OBJect code can not be "executed" by the machine however until it is "linked" to other .OBJect code segments in the CA-Clipper libraries. This is the responsibility of the linker. Linking or "fixing up" references made by your code to external variables and functions is necessary to insure that your compiled program will address or call other programs, including functions in the CA-Clipper libraries and DOS functions.
A linker's basic job is to combine objects which have been compiled separately. Objects contain symbols of three types:
The linker helps prepare the application to use this capability of CA-Clipper 5.2 by splitting the CA-Clipper-compiled application code into fixed-size "pages" that share a pre-allocated area of memory. These pages of code are then written to a seperate .OVL file or to the end of the .EXE file as determined by your linker options. When a procedure or function is called, the "dynamic-overlay manager" first checks to see if it is in memory. If it is, then it jumps to the called function. If it is not in memory, it loads the function from disk into any "free" location available in the overlay pool. If no free space can be found in the overlay pool, the overlay manager discards pages that have not been called recently to free space to load the called procedure.
The larger the overlay pool, the less often the overlay manager needs to load from disk and the faster your program will run. The size of the dynamic overlay pool needed for your application is automatically determined by CA-Clipper.
The great thing about CA-Clipper 5.x's dynamic overlaying system is that you, the programmer, don't need to do anything to create overlays. It is all done automatically by the linker. Only CA-Clipper-compiled modules, not C/ASM modules, are overlayed "dynamically", although since much of the CA-Clipper libraries has been compiled in CA-Clipper, even large portions of the CA-Clipper libraries will be dynamically overlayed.
Application libraries and third-party libraries written in CA-Clipper will also be automatically overlayed by BLINKER. In many cases it is more beneficial to write functions in CA-Clipper (using LOCAL memvars) than in C or ASM because it will not increase the executable memory model.
EXTERNAL overlays are sets of dynamic pages which are written to a seperately designated file more commonly referred to as an overlay file. You may wish to create a seperate overlay file when your application .EXE is too large to fit on a distribution disk.
Examples:
SECTION INTO myprog.ovl
SECTION INTO myprogs.ovl ;
MYPROG1, MYPROG2
SECTION INTO myfuncs.ovl ;
MYFUNC1, MYFUNC2
Modules which CANNOT be dynamically-overlaid are:
Dynamic overlaying of third-party libraries can be an aggravating learning experience if you don't get support from the third party vendor in trying to accomplish this task. Most of the third-party vendors now break their libraries into separate overlayable and non-overlayable .LIB files to make the task much easier. Once you have determined which .OBJ files and/or .LIB files are acceptable for overlaying, the task of telling the linker to do this is quite simple.
Example of a BLINKER script for overlaying C/ASM:
# Allocate 50k to the overlay pool BLINKER OVERLAY OPSIZE 50 BEGINAREA # CA-Clipper-compiled application code FILE myprog1.obj, myprog2.obj # C/ASM files FILE ccode.obj, asmcode.obj # C/ASM libraries LIB ccode.lib, asmcode.lib ENDAREA
EMS Memory
Many computers have EMS memory. EMS memory managers such as QEMM, 386MAX, and DOS's own
EMM386 create a memory area above conventional memory referred to as the "page frame".
This 64k block of memory is mapped to an area of ram that can execute 8086 code. Blinker
takes advantage of this feature by allocating the 64k EMS page frame rather than
conventional DOS memory for their runtime overlay pool. This frees an additional 64k of
memory for use by CA-Clipper because the overlays are loaded into the page frame rather
than conventional RAM. If no EMS is present when the application starts, then the overlay
pool will be established in normal (conventional) RAM. To enable this feature, use the
command:
BLINKER OVERLAY PAGEFRAME ON
UMB Memory
Expanded memory managers will also create an area between the 640k and 1MB memory
address called UMB (upper memory blocks). This is the area where you would normally load
your memory-resident programs such as DOS, network-drivers, mouse-drivers, etc. If your
enviroment gives you about 30k - 60k of contiguous memory available in the UMB area,
Blinker will load the overlay pool into the UMB area rather than conventional area, thus
freeing more conventional memory for your CA-Clipper application. To enable this feature,
use the Blinker command:
BLINKER OVERLAY UMB ON
Note: In today's modern environments, this UMB area is usually consumed by so many TSRs that the possiblity of finding a contiguous block larger than 30k is virtually impossible, so it recommended that you do not rely on the availability of this memory.
The MODULE command is very useful in that it allows your linker to place "modules" from libraries into overlays. You can organize your projects by placing all your C/ASM objects in libraries with a library manager such as Microsoft's LIB.EXE then by using the SECTION MODULE <module> command in your link file you can place any module into any overlay area. Blinker allows you to define a specific module from a specific library to be linked into the application, thus you don't need to worry about the module being unique to one library. For example, if you have two libraries, both containing an ERRORSYS module, you can decide which ERRORSYS you want linked into the program.
LIB mylib BEGINAREA LIB grump MODULE errorsys FROM grump MODULE myfileA,myfileB MODULE myfileC MODULE myfileD ENDAREA
BLINKER OVERLAY OPSIZE 50 BLINKER PROCEDURE DEPTH 50 FILIB OUTPUT BEGINAREA FILE ALLOCATE # Modules from CLIPPER.LIB MOD accept, acopy, adel, ; aeval, ains, appinit, ; atail, box, cmem, date MOD dbcmd2, dbcmd3, dbcmd4, ; dbcmd5, dbcreate, dbf0, ; dbfdyn, dbjunct MOD dbstruct, dtx0, dtx1, ; dtxdyn, dynina, errsys0, ; errsys1, fget MOD getenv, gets0, gets1, ; gets2, gx, joinlist, ; lupdate, memory ALLOCATE extend ENDAREA LIB clipper, terminal ,dbfntx
There are a standard set of link-scripts included with Blinker that make it very simple to overlay the Clipper 5.2 libraries. They are as follows:
CL520MIN.LNK - Used to overlay the smallest portion of the Clipper libraries for maximum speed performance but minimum memory performance.
CL520MID.LNK - Used to overlay a moderate portion of the Clipper libraries for moderate speed performance and moderate memory performance.
CL520MAX.LNK - Used to overlay the maximum allowable portion of the Clipper libraries for minimum speed performance by maximum memory performance.
These are sub link script files that can be simply included in your master link script file like so:
BLINKER OVERLAY OPSIZE 50 BLINKER PROCEDURE DEPTH 50 FILIB OUTPUT @CL520MAX.LNK
Blinker has become the standard for linking Clipper applications, so much of the third-party community also will include "ready-to-go" link-scripts with their libraries to make it easy to add their library to your application. For example, if you are already using a Blinker link script and decide to add the Six Driver (RDD) to your application, you would simply add one line of code to your link-script:
@SIX3.LNK
These third-party link-scripts will have the commands for deciding which modules in the library should be overlayed and which should remain in the root.
Simply overlaying an entire library is not always the best choice when speed performance is critical. For example, a function that skips a record pointer through a database must be called once for each record. If this function is in an overlay then the speed of browsing a database, locating a record, or reindexing a file can be greatly affected because the function must be called via the overlay manager. Speed is not usually a problem when overlaying code that has been compiled by CA-Clipper, because the CA-Clipper-code overlay manager is actually part of the CA-Clipper libaries and is very efficient in the manner in which it dynamically overlays pages of P-CODE (CA-Clipper .OBJs). C and ASM native code on the other hand must be overlayed by the linker's overlay manager and can greatly affect performance if an often-used function is not in the "root" memory area.
In my opinion, there is no good rule-of-thumb for determining which modules should be overlayed other than the time-tested method of "trial and error". I have spent hours and hours testing different overlay strategies to end up with the just-right balance of speed vs memory optimization. This investment of time has always payed me great dividends because from then on I was confident that I done my homework and that my application was "perfectly tuned".
The linkers included in this discussion offer some other fine features that can provide improved speed performance without sacrificing memory performance, however, they require that you utilize some of the advanced features of your computer hardware to get the benefit.
Overlay Caching
BLINKER supports the feature of "caching" overlays to improve runtime speed on computers
with EMS or XMS memory. A "cache" is an area of memory that has been pre-allocated for
storing code segments that have been overlayed and need to be swapped out to make room in
memory for new code. Rather than discarding the code segment and then reloading from disk
each time it is needed, it can be reloaded from the "cache" memory. Obviously, memory to
memory loading is much faster than disk to memory loading.
Blinker 2.x and later uses the following commands for caching with XMS (2.0 and higher):
BLINKER CACHE XMS niMax[%], niMinLeave[%]
For example the command BLINKER CACHE XMS 50%, 1024 will use a maximum of 50% of all available XMS, but leave at least 1024KB available for other programs. Even faster caching is available by using XMS (expanded memory) drivers because memory is handled in large contiguous blocks rather than the smaller 64k "pages" such as the EMS (expanded memory) drivers. Use the following command for caching using EMS (4.0 and higher)
BLINKER CACHE EMS niMax[%], niMinLeave[%]
Overlay Pool Size
Blinker provides for control of the overlay pool size to allow more code segments to
remain in memory and prevent excessive "swapping" in and out of the pool. Increasing the
size of the overlay pool will usually have the affect of improving speed performance, but
not always. This is another of those "trial and error" options. The disadvantage of
increasing the pool size is that the application memory will be reduced, so don't use
more overlay pool than is needed for the application.
Blinker: BLINKER OVERLAY OPSIZE <nMemoryK>
Procedure Stack Size
A portion of the DGROUP memory segment is allocated to the "procedure stack". This is the
stack that determines how many function calls can be sub-nested in the application. When
this stack overflows, a 650 STACK EVAL error will occur. The stack size is usually set by
the linker and defaults to a decimal size of 5120 or a procedure depth of 50. This means
that the application can nest calls to up to 50 subroutines. Many third-party products,
especially RDD's, require a larger stack size because they nest more calls. They will
include a command like STACK 7160 or BLINKER PROCEDURE DEPTH 70 in their link script. It
should be noted that when the procedure stack is increased in size to prevent 650 errors,
the EVAL stack is reduced in size thereby increasing the risk of 667 or 668 errors.
DOS Extenders eliminate the DOS 640K barrier, since in protected mode, extended memory is directly available without requiring memory swapping of any kind. Protected-mode applications can be loaded in as little as 256K (or less) of low DOS memory. As long as enough extended memory (XMS) is available, your application will run fine. You will have no problems even if available DOS memory is being squeezed by network drivers and the like.
Similarly, overlaying of code is no longer an issue. Most DOS extenders will automatically overlay code if necessary, however, if you have enough RAM, your .EXE will be loaded into memory, as it is used, until the entire .EXE is resident in RAM. Standard linkers use overlay pools in which overlaid code has to be swapped in and out of a relatively small overlay area as it is used, so a large application can never be completely resident in RAM. This has obvious performance implications that are not inherent when using a DOS extender.
"VM Integrity Failures" (a.k.a. 5333 errors) are a thing of the past. While it is still possible for a buggy program to cause an application to crash, many of the situations which used to lead to 5333 errors will no longer cause problems. This is because the virtual memory system swaps VM blocks automatically, as they are accessed, and does not require the programmer to remember to lock a block before accessing it.
Without all the swapping that CA-Clipper is forced to do within the 640K DOS limit, the performance of many applications will improve visibly and substantially, depending on processor and memory configurations.
What about 386 & 486? Applications running on 386 and 486 machines also benefit, of course. Instead of continually having to swap data from expanded memory (EMS) or having to load overlaid code from disk, applications can now use all of a machine's memory directly. Swapping only occurs when all available memory is full, instead of when the first 640K is full. In effect, the 640K RAM ceiling is raised to include all available memory. This will benefit almost any application on any supported machine.
CA-ExoSpace is compatible with most, but not all, third-party libraries for CA-Clipper. Incompatibilities can arise if a third-party library (3PL) performs operations which are not valid in protected mode. General purpose function libraries, and libraries which consist mostly of CA-Clipper code, should be compatible with CA-ExoSpace. Only libraries which perform very low-level or hardware-dependent operations may have problems.
Hardware Compatibility
CA-ExoSpace works on all IBM PC/AT-compatible computers. There are, however, some machines
that claim AT compatibility, but fail in certain important respects an 80286 CPU does not
make a machine compatible. The BIOS code must also be compatible.
CA-ExoSpace will run on any PC that:
Memory Requirements
In its default configuration, the CA-ExoSpace loader takes up about 90KB, 18KB of which is
in low memory. Under VCPI, the VCPI page tables are also stored in low memory, adding
about 15KB to the low memory footprint. Under DPMI, the stack is also locked in low
memory. Under some circumstances, you can reduce the amount of memory that CA-ExoSpace
takes. CA-ExoSpace uses extended memory, which is memory addressed by the CPU above one
megabyte. CA-ExoSpace does not work with expanded memory, or EMS, as defined by the
Lotus/Intel/Microsoft bank-switching scheme, although it can use memory managed by
products that use the 386 Virtual Control Program Interface (VCPI) standard.
Operating Systems
When a CA-ExoSpace application starts up, CA-ExoSpace searches the environment for memory
managers. In most cases, CA-ExoSpace uses the first memory manager that it finds. Device
drivers that get their memory from a DPMI, VCPI, or XMS pool pose no problems for
CA-ExoSpace. In addition, CA-ExoSpace works with many device drivers, even when they do
not get memory from a standard memory manager.
Create Dual mode Programs
A major advantage of Blinker 3.x is its ability to create dual mode programs simply by
adding a command to the link script file. Dual mode programs will be able to detect at
startup whether there are enough system resources to run in protected mode, and will do
so if there are. If there are insufficent resources then the same program will run as a
normal dynamically overlaid program. The .EXE still only consists of one program capable
of operating in either mode, it is not two separate copies joined together in some way.
Therefore, there is negligible increase in EXE size over a single mode program, and a
similarly negligible overhead in execution speed, since the while program is loaded into
memory when running in protected mode. The implication of this is that Clipper
applications linked with Blinker 3.0 will not necessarily have to rule out 8086 and small
80286 machines.
Dual Mode Caveat
For this dual mode to work and be effective, all C and ASM code will have to be written to
execute in both real and protected mode. In the worst case it should be able to test for
the processor mode each time it is executed and then jump to the appropriate code. Blink,
Inc. STRONGLY suggests that you DO NOT create a protected mode library separate from the
real mode library, since both types of code will have to be linked in at link time.
The benefits of writing dual mode code and creating a dual mode library include:
When you link in ClipXMS, as long as the computer's CONFIG.SYS loads a memory manager such as HIMEM.SYS, QEMM, or 386MAX, your Clipper application can use any available extended or expanded memory. If no memory manager is present, ClipXMS will not be able to use any available extended memory. In that case, ClipXMS does include the ClipXMS() function which allows your program to alert the user if they have extended memory available, but have not installed a memory manager.
An important side effect of using ClipXMS is that your users no longer have to configure their computers for EMS (expanded) memory. This frees up 64K of high memory allowing your users to load more device drivers and TSRs into high memory with the LOADHIGH command. Protected and Dual Mode ClipXMS is compatible with ExoSpace and Blinker 3's protected mode, but ClipXMS does not benefit protected mode programs because protected mode programs are able to use extended memory directly by themselves. If you do link ClipXMS into a protected mode program, your program will run exactly as it did without ClipXMS. Compatibility with protected mode is important if you are using Blinker 3's dual mode. When you link ClipXMS into a Blinker 3 dual mode program, ClipXMS will not interfere when the program runs in protected mode, but ClipXMS will kick in when the program runs in real mode allowing your real mode program to use XMS memory directly.
In order to add more features to a language, programmers usually find themselves faced with having to learn new programming and linking techniques to deal with the new "bloat" or other runtime problems. Fortunately, most CA-Clipper programmers have already had to deal with some of these problems under CA-Clipper 5.2 when using a variety of third-party libraries. Programs that would not run because of insufficient memory would run faster and more reliably when using a protected mode linker like CA-Exospace, Blinker 3.0, or Causeway. CA-Clipper 5.3 uses about 30k more conventional memory (MEMORY(0)), than CA-Clipper 5.2, therefore applications linked in real-mode that are already experiencing errors like "Conventional memory exhausted", will probably not run under 5.3. There is probably no reason to convert them to 5.3 unless one or more of the new 5.3 features is required in your application.
I highly recommend that if you intend to use Clipper 5.3 in the future, that you become familiar with protected-mode linking and start linking your Clipper 5.2 applications with one of the linkers described under the section titled DOS Extenders.
The other new features of CA-Clipper 5.3 also add an additional hit on conventional memory usage so if you plan to use the new screen-graphics driver or the new version of the DBFCDX driver, you will most likely find yourself needing to link with the CA-Exospace linker that comes in the 5.3 box. Unfortuately, CA-Exospace is notoriously slow during the linking cycle, so you had better be prepared to purchase a third-party linker like Blinker 3.x or Causeway.
An additional side-affect of upgrading to Clipper 5.3 will be the increased amount of DGROUP that is consumed due to the new larger base. This means that about 3k less DGROUP is available for the EVAL stacks, therefore you should be prepared to start modifying any of your Clipper functions that declare a large number of LOCAL or STATIC memvars to replace these declarations with LOCAL or STATIC arrays. If your CA-Clipper 5.3 application crashes often with STACK EVAL errors (667, 668 or 669) then see the section titled Managing DGROUP Memory for more information about how to do this.