This book is for Joanju's (now open source) 4GL/ABL parser.
C++ DLL, Java jar
There are two flavors of Proparse. The original (2001-2008) is the C++ DLL. The new port of to Java (July 2008) is tightly associated with the ProRefactor project, and its source is in that repository.
Download
You can find pre-built binaries (DLL and .jar) on Joanju's Proparse page at
joanju.com/proparse/.
Documentation
The documentation is available online here:
www.joanju.com/dist/index.html
Related Links
Prolint
ProRefactor
Code Parsing Utilities
PRO/Dox
Joanju Software
If you have everything all set up perfectly, then Proparse can be compiled with a simple Gnu make command. Getting everything set up can take a lot of effort though, so if you are adventurous and want to try compiling it yourself, be prepared for some challenges. The source requires Gnu C++, Gnu Make, perl, and probably other things I've forgotten. For performance reasons I usually compile the Windows DLL with MSVC++, but I found that by installing the right packages with Cygwin, I'm able to compile with that. Doing a build on Linux isn't too hard. The Gnu and other tools are usually easy to find and install for other unix platforms too.
Boost
Compiling Proparse also requires the C++ Boost libraries, which are large, so I haven't included them in the SVN repository. Download it from www.boost.org (I'm using version 1.33.1), unzip it somewhere, then copy its 'boost' subdirectory into the openproparse directory (alongside antlr, java2, spider, etc).
Antlr
The Antlr libraries are included, and the sources are under version control because I've had to hack a few of the C++ source files.
JDK
Proparse compiles with a built in Java-native interface, so it requires a couple of C header files from the JDK. The Makefile, by default, looks for the jdk in /progfile/jdk/, but of course you can change that.
Sample Session
Here is an example session compiling Proparse on Linux. If you aren't a 'vi' user, use your own editor. Note the necessary copy of the 'boost' directory. See the Makefile for necessary (but configurable) jdk directories and files.
$ svn export svn://oehive.org/proparse/trunk/openproparse $ cp -a boost openproparse $ cd openproparse $ vi Makefile $ export LD_LIBRARY_PATH=. $ make
Proparse is ported to Java and can be launched as a little server which listens on a TCP port. It receives a program name, finds and parses the program's source code, and then returns a BLOB containing records of all the objects in the syntax tree and symbol tables.
The blobs will be read by ABL routines to generate ABL objects (temp-tables, data sets, Objects) as needed.
This new Proparse API will be used by another project right away. In the coming days you can look for “GUI BOM” here on the hive.
There isn't any timetable for a new version of Prolint using this new API.
API Goals
The API will be a facade for the byte data in the blob, and the API should generate ABL objects only on demand. For example, consider node:firstChild(). This should return a Node object. The internals of the firstChild() function will check to see if the object has been created yet, create it if necessary, and then return it.
As another example, node:getText() (or just node:text using PROPERTIES) won't actually return data from the Node object itself. The method will look up the necessary records in the blob to return the node's text.
Another example: Functions which query the syntax tree (for example: find all DEFINE nodes) should not operate by recursive-descent of all the objects in the tree, because that would force the creation of a Node object for every node in the tree. Instead, such query functions should work by examining the bytes in the blob directly, and creating those Node objects as needed for the result set. (The API will have to keep track of which Node objects have already been created.)
OpenEdge Version
It is probably reasonable to point out that you can use older versions of Prolint with older versions of OpenEdge. We will write some of the new API using ABL classes, but it wouldn't be a big task if anyone wanted to back-port the API so it works with earlier releases. We'll keep in mind that, when there's no significant disadvantage, we'll use a plain old .p so that back-porting would be a little bit less work.
I created a new repository called 'codeparse', added a directory called 'wip' for shared efforts that are work-in-process or experimental, and added a subdirectory called 'proparseclient'.
svn://oehive.org/codeparse/wip/proparseclient
For those interested in building and using the API, you should be able to run client.p in that directory, which reads the two binary files 'header.bin' and 'blob.bin'. The program 'tree2text.p' is small and reasonably easy to understand as a starting point. If you don't know which API we're talking about, see http://www.oehive.org/node/1247
(SVN is running slow for me today, which is unusual. I hope it's just my local ISP.)
As my next task, I think I'll add a data transfer from Proparse so that we can fetch its internal data about node types and keywords.
Most developers working with Proparse would use an existing API, and these notes would not be of interest. These notes are only for developers working directly with the bytes from the blob, for example, when developing a new API.
The layout for the blob had one over-riding design consideration: It had to be indexed for fast random access of the records and fields within it. Loading all of the records into ABL objects would take too long for an interactive tool like Prolint.
As a result, the blob ended up looking somewhat like a miniature read-only database.
In order to understand what is inside the blob, it is probably easiest to start by understanding the goals.
Fixed Length Records (mostly)
One goal was to be able to access record fields (examples: node.text, node.type, node.line) by their offset in the record. As a result, most types of records are fixed length. The exceptions are strings, lists, and maps (i.e.: 'collections'). Collection records all have a similar layout: the collection's size, followed by the data.
So how does a 'node' record, for example, have a fixed length if it contains variable length data like strings? A string field in the record doesn't contain the string. It only contains a reference to a string record. The same is true for references to lists, maps, and any other record. The only fields that actually have their data there at the field position in the record are boolean and integer fields.
The Index
Since records (other than collections) are fixed-length, we couldn't use the usual serialization technique of writing one record right at a field position within a referencing record. But since we don't know the byte offset of each record within the blob until the record is written, how do we reference the record? Each record is referenced with an integer index number.
The index is an array of integer offsets. So, if we know that we want the record with index number 2, we get the offset by looking at position number two in the index.
The index is written at the end of the blob, after all records have been written, and their index numbers (and offsets) have been tallied.
The Record Schema
A special problem was presented by the fact that the Java code inside Proparse might change. We might want to rename fields. Fields might be inserted into the middle of field lists – especially troublesome in cases where fields get added to super classes.
To avoid problems with fields names and positions changing over time, each blob will contain a 'schema' of the records inside it. The schema is safe to re-use from one blob to the next, as long as the same version of proparse stays running as the server. (I.e. restart your clients if you upgrade the server.)
The schema for a class is very simple: an index to the name of the class in Proparse (ex: “org.prorefactor.core.JPNode”), followed by a pair of numbers for each field. One number is the field's record type, and the other number is the index of the string record of the field's name.
This project is for maintenance and continued enhancements to Joanju's (now open source) 4gl/ABL parser.
Example Subversion repository access:
svn checkout svn://oehive.org/proparse/trunk
Proparse.jar may be launched as a server which listens on a TCP socket. For example:
rem proparse.bat set JAVA_PATH= proparse.jar set JAVA_OPTS= -cp %JAVA_PATH% -Xss2M java %JAVA_OPTS% proparse.Server
As with scripting proparse.jar from JVM languages, the configuration files must be in the server process working directory, in a subdirectory named 'prorefactor', and created by the Project Configuration Dump Utility.
As of early August 2008, the New API for using this server is still work-in-process.
There are a few server server options which can be specified in an optional file named 'proparseserver.properties' in the server's working directory:
# Configuration file for running proparse.jar as a server. # This file must be in your current working directory. # (i.e.: ./proparseserver.properties) # Remove the leading hash mark '#' to uncomment a setting # for property = value. # port (optional) # The port to listen on. If no port is specified, then 55001 is used. # port = 55001 # project (optional) # The project settings directory to load from ./prorefactor/projects/. # If no project is specified, then Proparse will simply load # from the first directory it finds. # There does not have to be a project configured before launching the # server, because the client can take care of that. # project = sports2000
Download
You can find a pre-built proparse.jar on Joanju's Proparse page at
joanju.com/proparse/.
Configuration for Project Settings
See Project Config. Dump for a script which creates a prorefactor project settings directory.
Disc Space
Proparse may write very large amounts of data to into the 'prorefactor' sub-directory of your working directory. If your application is very large, then this could be as much as a few GB. See PUB Files for more information.
This section assumes you will be using Groovy for scripting, but other JVM scripting languages will be similar.
groovy --version
set CLASSPATH=src;bin;prorefactor.jar
assuming proparse.jar is in the working directory. (On Linux or unix, use ':' instead of ';' to separate classpath entries.)