ProRefactor

ProRefactor is a Java library for extending Proparse (joanju.com/proparse). At its core, it slurps Proparse's syntax tree into a tree of plain old Java objects, and further processes that tree to add symbol tables and other useful information.

ProRefactor has been used for building tools for code analysis, lint, documentation, search and replace, and more.

ProRefactor is available as an Eclipse plug-in and as a standalone Java archive (.jar) file.


About

Automated Code Transforms (Automated Refactoring)

Refactoring is the act of improving your source code without changing its behavior. So - your business logic stays intact, but something in the source code gets improved which makes it easier to work with, more portable to other platforms, easier to translate...

Refactoring deals with things that are tedious, time consuming, and ultimately, expensive. These are things that should be automated! We can build you the automated transform tools that you need to save you weeks or months of effort.

Review Changes DialogReview Changes Dialog

Here we automate the transformation from string concatenation to the use of a single string with SUBSTITUTE. This is done because the full sentence in a single string is easier to translate (for i18n) than the sentence fragmented into two strings.

Proparse enables complex code transformations, where context and semantics must be examined. These kinds of refactoring cannot be done with simple search and replace tools. Refactoring Toolkits such as ProRefactor are critical for making software "soft" again, and are key to making agile development really work, maintenance programming less expensive, etc.


Benefits

SUBSTITUTE Refactoring

Automates this necessary step towards making your application translatable. If you need to make your application translatable, ProRefactor can save you weeks of effort, and pay for itself with this one refactoring alone.

NO-UNDO Refactoring

Automates the process of finding and fixing all DEFINE VARIABLE statements which are missing the NO-UNDO option. Most 4gl programmers today already know to do this, but there is a lot of really old code out there. This refactoring can represent a fundamental step in fine tuning your application source for better overall application performance.

Names Refactoring

Automates the process of finding and fixing all database table and field names which are either abbreviated or else use the wrong case (UPPERCASE/lowercase). Abbreviated names are known to be problematic, especially in a system with an evolving schema. Changing the case on table and field names can help accelerate the integration of a third party's source code base into your own source base, or could help accelerate code "cleanup" projects.

Qualify Field Names Refactoring

Automates the process of adding table/buffer names to unqualified field names. Unqualified field names can lead to to bugs which are difficult to find.

Rename Schema

Finds and changes all hard-coded references to schema table and field names. This saves you from a large portion of the manual effort required when renaming schema elements.


Download & Install

Download and Install Using Eclipse's Update Manager

See Install Instructions.

The latest release is 1.7.2, as of June 28, 2007. Be sure to see the special update instructions, below.

The URL that you will require for installing ProRefactor via Eclipse's Update Manager is: 

http://prorefactor.oehive.org/update/

Standalone JAR

http://prorefactor.oehive.org/prorefactor.jar

See Standalone Jar File for details.

OpenEdge Architect

OE Architect 10.1A ships with (and points its shortcuts to) Java 1.4, but ProRefactor requires Java 5. Download and install the latest Java, then simply copy and change the OE Architect shortcut: remove or change the -vm argument. (Simply removing it lets Eclipse find the default Java on your computer.) After that, the usual Help->Install routine for installing ProRefactor seems to work fine. (Tested with the 10.1a01 patch installed.)

Adding ProRefactor Configuration to Existing Projects

If you create a ProRefactor Project, then this step is not necessary. If you create a project of any other sort (such as Simple or OpenEdge) then open the project's .project file, and add the following line to the natures section:

<nature>org.prorefactor.eclipse.ProRefactorNature</nature>

Special update instructions for specific versions

Version 1.6.0, April 13, 2007

You must re-dump your schema, using the Project Config Dump version 002. Field extent has been added to the schema dump layout, so old schema dump files will not work with ProRefactor 1.6.

Version 1.4.0, March 29, 2006

Version 1.3.0, Feb 2006

Version 1.2.3, Jan 2005

If you currently have the old Joanju.com.prorefactor plugin installed, please follow these steps to upgrade:


Install Instructions

ProRefactor is an Eclipse plug-in, and Eclipse provides a user interface to make the installation and updates of plug-ins easy. If you have ever used Eclipse's Feature Updates wizard in the past, then you probably do not need to read this section at all.

Java

ProRefactor and Eclipse require a recent version of Java. Java can be freely downloaded and installed from java.sun.com.

Eclipse

ProRefactor requires a recent version of Eclipse. Eclipse can be freely downloaded and installed from www.eclipse.org.
Note:ProRefactor creates a prorefactor subdirectory in your Eclipse working directory. Your project settings are stored there, and ProRefactor writes various additional files and directories there.

ProRefactor

Start Eclipse and choose Help > Software Updates > Find and Install... and use the following URL: http://prorefactor.oehive.org/update/.

Linux and Unixes

The Proparse shared libraries are not distributed with ProRefactor, and those must be installed separately. Proparse comes with its own install notes, but there are a few additional items worth mention with regards to ProRefactor. The shared libraries and the DLL symlink may be installed into one of two places: either into the org.prorefactor.core plug-in directory, or else into your LD_LIBRARY_PATH (or equivalent). Since there is a new org.prorefactor.core plug-in directory with each new ProRefactor release, it probably only makes sense to install Proparse's shared libraries into your LD_LIBRARY_PATH.

Note: Linux provides a shared library loading mechanism via the /etc/ld.so.conf file. This does not (at present) work for loading shared libraries into Java or Eclipse.

All Done

You should now be able to select the ProRefactor perspective from Eclipse's Window menu.


javadoc

Joanju maintains a build of the javadoc on their web site here:
joanju.com/analyst/javadoc/.
The org.prorefactor.* packages are what you need, ignore the com.joanju.* packages.

Of course, you can always svn checkout the source for ProRefactor and build your own complete set of javadoc for it.


Project Config. Dump

This small package makes it a little bit easier to get started with the configuration of a project for ProRefactor.

It is the equivalent of the schema and configuration dump programs that can be found in ProRefactor's documentation. It is easier because there is only one program to run, and it prompts for which directory to write your settings to.

The zip file contains a directory named prorefactor. Extract it somewhere into your existing PROPATH.

Start your Progress development session, as usual. Your PROPATH, configuration settings, database connections, database aliases, etc., must all be fully set up as if you were going to compile the entire application. If you don't have these settings all correct so that you can compile your application, then ProRefactor won't be able to parse it either.

Then,

RUN prorefactor/configure.p.

You can simply follow the prompts in order to create your ProRefactor working directory project configuration(s).

See the zip file attachment, below.

Version History


ProRefactor project

This project is for maintenance and continued enhancements to the ProRefactor libraries for working with Proparse's syntax tree. Includes tree parsing, symbol tables, persistent storage of the syntax trees, project configuration, Eclipse integration, etc.


Contact Project Admin

Browse with WebSVN

Example Subversion repository access:

svn checkout svn://oehive.org/prorefactor/trunk

Scripts

Scripting opens up endless possibilities for working with ProRefactor and the syntax tree.

There are various scripting languages available for running on the Java Virtual Machine (JVM). Those include Groovy, Jython, JRuby, Rhino Javascript, and others. Any of those should allow you to access ProRefactor as a library, and run the parsing and analysis programs it contains. I chose Groovy because I found that Groovy had the cleanest interfaces between its scripts and existing Java applications and libraries like ProRefactor.

See Standalone Jar File for setting up ProRefactor for use from a scripting environment.


Code to XML

This script allows you to generate XML for each of your compile units. The XML contains the entire syntax tree, as well as the symbol tables. This XML is a very raw dump of Java objects from memory.

What is the benefit of this? It allows you to use OpenEdge scripts to gain access to all of the useful information that is generated by ProRefactor. For example, you might use the XML as input for ABL scripts to generate code documentation, to augment your XREF indexes, or to allow you to perform ad-hoc search and analysis.

The XML is unlikely to be self explanatory, so please feel free to post questions in the CodeParse Forum.

Ideally, this will inspire a new open-source project, written in ABL, which will read the XML and convert it to TEMP-TABLEs or some other ABL data structure. That would make the data much easier for other ABL programmers to understand it and use.

This script may require a lot of CPU, memory, and disc space. If your application is large, then the XML that gets written may require a very large amount of disk space, even though it is compressed. (For example, it might take as much as a few GB for an enormous application.)

The script writes to the prorefactor/xml directory. The script is easy to change if you want it to write its output elsewhere.

See the comments at the top of the script for usage notes.

Comparison to the Proparse API for ABL

Proparse is a DLL (.so on unix), with an API for using it from ABL(4GL) programs.

The XML generated here contains all of the data that is available in the syntax tree generated by Proparse. In addition, it contains all of the symbol tables generated in ProRefactor. Those symbol tables represent a wealth of information about your source code, far beyond what is available from Proparse alone.

This XML output could (should?) be considered a replacement for the old Proparse API. For the most part, new ABL programs should use this, rather than use Proparse directly.


Fix Windows file references for unix

FileRefs.groovy
May 2008 by John Green.

Find and fix code developed on Windows so it has file references which are portable to unix.

Does the following for RUN *.p, RUN *.w, and for include file references:

  • Makes sure forward unix-style slashes '/' are used rather than Windows back-slashes '\'.
  • Makes sure that letter-case in the RUN or include ref matches the letter-case of the actual file name.

USAGE

  • NOTE that ProRefactor writes potentially large amounts of data, and that this script may use large amounts of CPU, memory, and disc space.
  • This script requires Proparse 3.1C or later, and a recent (May 2008) release of prorefactor.jar.
  • You must have ProRefactor and Groovy scripting set up. See: http://www.oehive.org/prorefactor/standalone_jar_file
  • Configure filerefs.properties in your current working directory.
  • Run the script: groovy FileRefs.groovy

Copyright (c) 2008 Joanju Software (www.joanju.com). All rights reserved.
Released under the terms of the Eclipse Public License v1.0:
http://www.eclipse.org/legal/epl-v10.html


Hendrik.groovy

This Groovy script finds variables which use the same name as a database table or field name. It writes its findings, as it runs, to the console. It also accumulates a sorted report which it writes at the end of its run.

I wrote this as a test case for Groovy scripting for ProRefactor. A better place for it would be as a rule in Prolint.

I decided to write this after seeing a post on the PEG from Hendrik Demol looking for a way to do this. Assuming he doesn't object, Hendrik gets the Groovy script named after him. :)

The file download can be found at the bottom of this page.

Requirements

  • Latest version of the ProRefactor standalone jar file.
  • Latest version of Proparse.
  • Java 5 or later.

Setting up standalone ProRefactor

See this page, and follow the instructions there, for getting and using the ProRefactor standalone jar file:
Standalone Jar File
See this page, and follow the instructions there, to generate the project settings files necessary for using ProRefactor:
Project Config. Dump

When launching the script, the prorefactor directory, with its project settings files, must be found in the current working directory.

Note that ProRefactor generates large volumes of data files in its prorefactor directory.

Command line

The script expects the source code to be found in the current working directory. That can easily be tweaked if necessary, find this:

    // Look from the current (working) directory down.
    new File('.').eachFileRecurse{ file ->

and change '.' to '/your/directory/' if you don't want to run the script from the top of your source tree.

The basic command line to launch a Groovy script like this via ProRefactor's standalone jar is:

java -Xss2M -Xmx128M -cp prorefactor.jar groovy.ui.GroovyMain hendrik.groovy

You might want to redirect the output to a file, or tweak the output destination of the sorted report output at the end of the run() method in the script.

ProRefactor API documentation

If you want to tweak this script for your own use, you might want to see the ProRefactor API documentation. See the javadoc page for how to find ProRefactor javadoc online.

Peeking at the script

Much of the script is scaffolding, for example, finding and loading the ProRefactor project settings files, iterating through compile unit source files and parsing them, etc. There are methods for building sorted lists of table and field names, and looking up names to see if they are unique in those lists.

The run() method is what gets run when the Groovy script is launched. It iterates through all of the .p, .w, and .cls files in the directory.

The checkSymbols() method does the really interesting work of iterating through all of the symbols (variables, frames, streams, etc) in the compile unit, and checking their names to see if they are also valid table or field names.


TreeUtils.groovy

Utilities for working with the syntax tree from ProRefactor. (Download below)

static printTree(JPNode n)

Prints the tree under the given node, output to stdout.

NOTE: At time of writing, this is simply dumping the token names from Proparse's API. This script should be changed to be sure to get the valid TokenTypes.TOKEN_NAME. See the notes at the top of http://www.joanju.com/dist/index.html.

static boolean matchNode(JPNode firstNode, List specList)

Test if part of the AST matches a given specification.

The specification is a list. Each element in the list may be an integer node type, a closure, or another nested
list describing the previous node's children.

Example:

import org.prorefactor.core.TokenTypes as T
[
T.RUN, [
  T.VALUE, [
    T.LEFTPAREN,
    T.Widget_ref, [
      T.THISPROCEDURE,
      T.OBJCOLON,
      {it.getText().equalsIgnoreCase('PRIVATE-DATA')} ],
    T.RIGHTPAREN ],
  T.PERIOD ]
]

Note how the example looks a lot like a simple printout from printTree():

RUN
  VALUE
    LEFTPAREN
    Widget_ref
      THISPROCEDURE
      OBJCOLON
      ID "private-data"
    RIGHTPAREN
  PERIOD

Also note that in the example, we are using a Groovy closure to test a node's text, rather than the node type. Anything from the node could be tested, like filename, line, etc. (The "it" variable is the implicit single argument in a Groovy closure, see the Groovy User's Guide for details.)

Once "import static" is supported in Groovy, we won't have to write "T." for every node type, and the specifications will be easier to read.

A null in the list will match any node. An empty sub-list will cause a node's children to match without checking.

The first element of any list must not be a sub-list.

Typically, the input spec list will contain two elements: a spec for a head node, and a sub list containing the specs for its children. But, it's also perfectly valid to specify n sibling nodes in the spec list, and maybe their children.


Standalone Jar File

ProRefactor can be used as a standalone library, entirely independently of Eclipse.

This Java archive (jar) file contains ProRefactor, as well as the third party libraries it requires.

prorefactor.oehive.org/prorefactor.jar
Last updated: June 17, 2008. About a dozen MB.

Setting Up ProRefactor

  1. See Project Config. Dump for a script which creates a prorefactor project settings directory.
  2. Proparse must be downloaded separately: joanju.com/proparse/. The Proparse shared library (.dll or .so) must be in your library path. On Windows, this can simply be your working directory. On most unixes, you would set an environment variable. For example, on Linux you might use: 

    export LD_LIBRARY_PATH=.

    ProRefactor.jar always requires the latest release of Proparse.

  3. Remember that ProRefactor may write very large amounts of data to into the 'prorefactor' sub-directory of your working directory. If your application is very large, then this could be as much as a few GB. See PUB Files

Setting Up Scripting

This section assumes you will be using Groovy for scripting, but other JVM scripting languages will be similar.

  1. Install the latest version of Groovy (assuming you have Java installed): groovy.codehaus.org
  2. Open a command line prompt. Change to your prorefactor working directory. Test that groovy runs OK with: groovy --version
  3. Make sure your CLASSPATH environment variable contains prorefactor.jar. For example, on Linux you might use:

    export CLASSPATH=prorefactor.jar

    assuming prorefactor.jar is in the working directory. (If you need additional jar files, don't forget on unix, use ':' instead of ';' to separate classpath entries.)


Console

Using the standalone jar file, you can launch a command line console which contains one or two examples of how you can use the ProRefactor libraries. The console (org.prorefactor.Console) is the default class for the standalone jar file, so you can launch it simply with:

java -jar prorefactor.jar

Here's what its menu looks like as of Jan 2007:

h) How-used report
l) Load Settings for a project
p) Parse a directory
q) Quit
v) show proparse Version
w) Where used report

Enter selection:

Showing the Proparse version is useful for checking that the Proparse shared library can be found and loaded OK. The parse a directory option is useful for checking that your ProRefactor project settings are correct, and that ProRefactor can parse your application OK.


PUB Files

In prorefactor/projects/projectname/pubs in your working directory, ProRefactor writes out a parse unit binary (PUB) file for each compile unit in your application. These will be built as needed, when you do a parse or run a report.

These PUB files contain the full syntax tree and some additional information, so that your programs do not need to be re-parsed each time ProRefactor wants to get information about them. The *.pub files can be thought of a bit like your *.r files.

Here's the important bit: These files contain a fair amount of information, and will probably require more disc space than your .r files. In fact, for a very large enterprise application, the PUB files might require as much as a few gigabytes of disc space.


Where Used Report

This is not the most efficient way to find where a database field is used, but it is a useful example of using ProRefactor.

See this WebSVN link for the full source.

It prompts for a fully qualified db.table.field name, checks that it exists in ProRefactor's configured schema, and splits it into two string variables: fieldName, and dbTableName.

It then finds all parse units in the directory of your choice, and loads or builds the PUB file for each of those, and then checks to see if your field is used in that compile unit. So, for each parse unit pu:

PUB pub = pu.getPUB();
boolean wasCurrent = pub.loadTo(PUB.SCHEMA);
if (wasCurrent==false) pub.build();
if (usesField(pub) == false) return;
reportOut.write(pu.getFile().toString());

Next is the relevant code for usesField(). The PUB object has a record of all tables and fields used in that compile unit. We get a copy of the list of fields of the table that were used in the PUB, then check if it contains the name of the field we are interested in.

pub.copySchemaFieldLowercaseNamesInto(fieldNames, dbTableName);
return fieldNames.contains(fieldName);

How Used Report

This report shows a snippet from each line of code where a database field is used. For example, a how-used report for sports2000.customer.custnum might show:

s2k\gui\bcust.w
define temp-table ...  like Customer.CustNum validate

s2k\gui\dcust.w
define temp-table ...  like Customer.CustNum validate

s2k\gui\dcust_cl.w
define temp-table ...  like Customer.CustNum validate

s2k\gui\orderreport.p
format ...  Customer.CustNum column-label "Cust-Num" format ">>>>9"
display ...  Customer.CustNum

s2k\gui\vcust.w
define temp-table ...  like Customer.CustNum validate

s2k\gui\vcust2.w
define temp-table ...  like Customer.CustNum validate

s2k\gui\vordlne2.w
find ...  customer.custnum eq order.custnum
find ...  customer.custnum eq order.custnum

s2k\gui\worder.w
define variable ...  like customer.custnum

See this WebSVN link for the full source.

This report takes a ProRefactor Field object (reportField) as a parameter, and then walks the syntax tree for each parse unit passed to it, starting at the topmost JPNode node in the tree. It checks to see if each node is a FieldRefNode referencing the reportField we are interested in:

private void walkTree(JPNode node) throws Exception {
      if (node==null) return;
      boolean match = false;
      if (node instanceof FieldRefNode) {
           Symbol symbol = node.getSymbol();
           if (        symbol!=null
               && symbol instanceof FieldBuffer
               && ((FieldBuffer)symbol).getField() == reportField
               )
               match = true;
      }
      if (match) {
           printReference(node);
      } else {
           walkTree(node.firstChild());
      }
      walkTree(node.nextSibling());
}

In printReference(), the beginning of the statement is found, and the first one node or two (DISPLAY, ASSIGN, etc.) is displayed, followed by the ellipses "...". The parent of the FieldRefNode is then found, and everything below it is then printed (including, of course, the FieldRefNode itself).

If you are new to syntax trees, then you might want to have a look at joanju.com/dist/docs/parse_trees to get an idea why print2() has to take an extra step to make sure that binary operators are printed in line.


Support and Discussion

The project page is where you will find developer and support resources, such as the issues database for bug reports, feature requests, and tasks.

The codeparse group is for discussions about all code parsing topics, including ProRefactor. Be sure to subscribe!

Finally, Joanju provides commercial support and service for ProRefactor.


Version History

Version 1.7.2

Version 1.7.1

Version 1.6.1

Version 1.6.0

Version 1.5.6

Version 1.5.5

Version 1.5.3

Version 1.5.2

Version 1.5.1

Version 1.5.0

Version 1.4.0

Version 1.3.0

Version 1.2.3

Version 1.2.2

Version 1.2.1

Version 1.2.0

Version 1.1.10

Version 1.1.9

Version 1.1.8

Version 1.1.7

Version 1.1.6

Version 1.1.5


Getting Started

Working with Projects

Users of most IDEs are familiar with the concept of a project, but there are a few points of interest which are unique to Eclipse projects and ProRefactor.

Project Nature

Eclipse projects may have one or more natures. Because Eclipse is an IDE for many types of development environments, it also recognizes that any individual project could contain source code for two or more different sorts of development environments, for example, both C++ and Java. Each project nature describes various attributes for a project, such as which project builders are available within the project. (i.e. There would be a builder for compiling the C++ code, and a builder for compiling the Java code.)

ProRefactor provides its own project nature. Most of ProRefactor's features are not available unless you are working with a project which has the ProRefactor nature.

Create a ProRefactor Project

You can create a new Eclipse/ProRefactor project to refer to your existing Progress source code by choosing File > New > Project > Progress > ProRefactor Project.

Project Configuration

Select your ProRefactor project, and choose Project > Properties. Choose ProRefactor Environment in the categories on the left of the dialog. Do not enter any settings, because there is a way to have these settings generated automatically for you. But, you can click the Apply button to force ProRefactor to create the prorefactor subdirectories in your Eclipse working directory.

When you save the configuration settings for your project, ProRefactor creates the following directories in your working directory:

prorefactor/projects/your_project_name
prorefactor/settings

See Project Config Dump for a program that can be run from your Progress project environment in order to create the configuration files to go in prorefactor/projects/your_project_name.

Have a look at the Project > Properties > ProRefactor Environment page. Look for any relative path references in your PROPATH setting - they probably need to be changed to fully qualified paths.

Database Aliases

Proparse and ProRefactor need to know about your database aliases in order to parse your source code. In many cases, the configuration dump program will take care of adding those to your settings. If you want to add database aliases into the project settings dialog, the format is just a comma separated list of pairs of alias names and logical database names: aliasname,ldbname,aliasname,ldbname,... .


General Usage

This section reviews the general aspects of the user interface which are not specific to any one refactoring feature. It also reviews user interface features which are unrelated to refactoring.

Eclipse Workbench

It is beyond the scope of this document to describe the Eclipse Workbench in any detail, however, a few quick notes and terms will make the rest of this chapter more understandable if you are new to Eclipse.

Workbench
Eclipse's Workbench is the entire Eclipse window and all of its contents. You may on occasion have more than one workbench active, but normally you would only work in one workbench at a time.
Workspace
A workspace consists of the list of the list of projects and settings that you see in your workbench.
Project
This term has the same meaning as most other IDEs. A project has a single, top-most directory, which in turn contains all of your project's sub-folders and files.
Resource
A resource in Eclipse is a project, a directory, or a file. In simple terms, it is anything that you might see in the Navigator view.
View
Within the workbench window, you can find various frames which are separated by movable splitter bars. In Eclipse, these frames are called views. Access to most views is through a tabbed interface. Examples of views are the Navigator view, the Outline view, the Console view, and the Tasks view.
Editor
Editors in Eclipse treated differently than views. Multiple editors can be open with a tabbed interface, and can be an unlimited number of types of editors within your Eclipse workbench. For example, there is an editor for plain text, there is an editor for Java, and there is an editor for C++.
Perspective
A perspective describes an arrangement and visibility of the various elements on the workbench. That includes the menus, button bars, views, and editors. There are default perspectives for Java, Java Debugging, for ProRefactor, and for many other plug-ins. Although perspectives all come with a default layout, those layouts can be modified. Eclipse remembers your layout changes.
Plug-in
Eclipse uses a plug-in architecture. This means that the Eclipse core is extremely small, and most of the elements that you see on the Eclipse workbench are actually contributed by plug-ins - even those elements of the workbench which are part of the basic workbench. There are also plug-ins which are not part of the basic workbench, and those include Java and C++ plug-ins which are part of the Eclipse.org project, as well as third-party plug-ins such as ProRefactor.

Menus

ProRefactor contributes menus to various parts of the Eclipse workbench. By default, these ProRefactor menus are only available when you are in the ProRefactor perspective. In the main menu bar, across the top of the workbench window, a ProRefactor menu is available.

The ProRefactor plug-in contributes a context sub-menu to the Navigator view. A context menu is a pop-up menu which is available by using right-mouse-click.

The ProRefactor plug-in also contributes a context menu to editors such as Eclipse's Text Editor.

Operating on Many Resources

Some features of ProRefactor are only available for a single compile unit at a time. However, some features are capable of operating on a selection of one or more resources, as selected in Eclipse's Navigator view. In the Navigator view, you can select one or more files, directories, projects, or any combination of these.

Once the desired resources have been selected, the Navigator context menu shows what operations may be performed on those. There is a ProRefactor sub-menu on the Navigator's context menu.

Expand and Contract Selection

Sometimes when viewing poorly formatted source code, it is difficult to find where a block begins and ends. ProRefactor provides a feature which can help.

ProRefactor contributes two buttons to the button bar which are called Expand Selection and Contract Selection. These actions are also available from the editor's context menu. This feature only works on source files which can be compiled. (i.e. It does not currently work within an include file if that file can not be compiled on its own.)

Simply place your cursor in the editor anywhere within the block of code that you are interested in, and choose Expand Selection. ProRefactor will parse the file, and then select some of the text in the editor around where your cursor was placed.

Expand and Contract Selection selects text by finding the begin and end points for the source code for a node in Proparse's syntax tree. By choosing Expand Selection repeatedly, you can climb the nodes in the syntax tree until you have selected the text for the code block that you are interested in.

Parse

The menu item Parse can be found on the main menu under ProRefactor, as well as on the Navigator view's ProRefactor context menu. This causes ProRefactor to parse the selected resource(s). For example, you can select your entire project (or multiple projects), and choose Parse in order to parse all of the compile units in those projects.

ProRefactor considers .p, .w, and .cls files to be compile units. This may be configurable in future versions.

ProRefactor stores information about your compile units, and it uses this information to determine if the compile unit needs to be re-parsed. It does this by checking the time stamps on the compile unit and all the include files that it references.

Output Directory

Most of the refactorings in ProRefactor make use of a wizard style of interface, with one or more pages to be reviewed before launching.

Some of the refactorings, especially those which process any number of resources, write their results out to a temporary directory.

The target directory that you select must exist, and it must be empty. Choosing the Browse button presents you with a dialog from which you can choose or create a directory.

Typically, once the refactoring has been completed, you will want to review the newly changed code before copying it into your project directory. See Links and Resources for some free tools which ease the process of reviewing many changes.


Catalog of Refactorings

This section describes each of the refactorings available from ProRefactor's user interface.


Names Refactoring

This refactoring is intended to be used as part of a legacy code clean-up project. It is unlikely to be used on a day-to-day basis.

This refactoring allows you to automatically:

  • set the case on table and field names to uppercase or lowercase
  • add the table name qualifier to unqualified field names
  • expand abbreviated table and field names

Usage

  1. In the Navigator view, select the files, directories, and/or projects to be refactored.
  2. From the Navigator view's context menu, choose ProRefactor > Names (table and field) Refactoring.
  3. The Names Refactoring wizard will be presented, where you are required to choose an empty output directory for the source code that gets changed.

This refactoring checks if an include file is changed multiple times for multiple compile units. If the changes are different for different compile units, then the include file is not written to the output directory, and warning messages are issued.

Example Output

Using "sports2000" schema, we automatically refactored this:

FIND FIRST CUST.
DISPLAY BAL.

into this:

FIND FIRST customer.
DISPLAY customer.balance.

NO-UNDO Refactoring

This refactoring is intended to be used as part of a legacy code modernization project. It is unlikely to be used on a day-to-day basis.

Although most Progress programmers recognize that NO-UNDO is important for application performance, it is not unusual to find legacy applications where NO-UNDO was not consistently added to DEFINE statements. This refactoring is intended to be run as a single pass through old code, in order to correct those statements.

Usage

  1. In the Navigator view, select the files, directories, and/or projects to be refactored.
  2. From the Navigator view's context menu, choose ProRefactor > NO-UNDO Refactoring.
  3. The NO-UNDO Refactoring wizard will be presented, where you are required to choose an empty output directory for the source code that gets changed.

Features

This NO-UNDO refactoring has a few features which are worthy of note.

  • It does not add NO-UNDO to DEFINE statements which are preceded by a "prolint nowarn" directive.
  • It does not add NO-UNDO to DEFINE statements which have a comment containing the string "undo".
  • It does not add NO-UNDO to DEFINE statements where that variable being defined is assigned or otherwise updated and then explicitly undone by an UNDO statement. See the code below for an example.

These features work nicely with Prolint. If ProRefactor's NO-UNDO Refactoring does not add NO-UNDO to a DEFINE statement, for whatever reason, Prolint will continue to warn about that DEFINE statement until you explicitly tell Prolint that it is intentional with a prolint-nowarn directive. (Or else of course you might review the code and decide to add NO-UNDO by hand.) Here is a snippet of code which is the output from the unit test used for the NO-UNDO refactoring. All of the NO-UNDO options on DEFINE statements were added by the NO-UNDO refactoring, but more interesting is to note where the NO-UNDO refactoring did not add NO-UNDO.

/* n o - u n d o . p
 * This file contains tests for our  N O - U N D O  refactoring.
 * IMPORTANT!! Comments containing "u n d o" (without the spaces)
 * have an impact on the refactoring's behaviour!
 */


{&_proparse_ prolint-nowarn(noundo)}
DEFINE VARIABLE myInt AS INTEGER.

PROCEDURE myProc1:
  DEFINE INPUT PARAMETER p1 AS LOGICAL NO-UNDO.
END.

PROCEDURE myProc2 EXTERNAL "whatever.dll":
  DEFINE INPUT PARAMETER p2 AS LONG.
END.


/* Test for  U N D O  statement. */
DEFINE VARIABLE myChar AS CHARACTER.
DEFINE VARIABLE myChar2 AS CHARACTER NO-UNDO.
DEFINE VARIABLE myChar3 AS CHARACTER NO-UNDO.
DO:
  myChar3 = "".
  DO:
    myChar = "".
    UNDO, LEAVE.
    myChar2 = "".
  END.
END.


/* U N D O  statement tests for named block and OUTPUT val */
DEFINE VARIABLE myChar10 AS CHARACTER.
DEFINE VARIABLE myChar11 AS CHARACTER.
my-block:
DO:
  RUN changeVal(OUTPUT myChar10).
  DO:
    RUN changeVal(OUTPUT myChar11).
    UNDO my-block, LEAVE.
  END.
END.

PROCEDURE changeVal:
  DEFINE OUTPUT PARAMETER changed AS CHARACTER NO-UNDO.
END.


/* This should remain UNDO */
DEFINE VARIABLE c1 AS CHARACTER.

/* This var should be UNDO           */
/* with this two line comment.       */
DEFINE VARIABLE c2 AS CHARACTER.

DEFINE /* UNDOable */ VARIABLE c3 AS CHARACTER.

DEFINE VARIABLE c4 AS CHARACTER. /* not no-undo */

/* This comment does not change UNDO for the next define,
   because of the blank line between the comment and the statement.
*/

DEFINE VARIABLE c5 AS CHARACTER NO-UNDO.

DEFINE VARIABLE c6 AS CHARACTER NO-UNDO.
/* Comment on line after does not change UNDO for previous statement. */

Qualify Fields Refactoring

Generally, using unqualified field names is not viewed as "best practice" for 4gl developers. There are a number of pitfalls in using unqualified field names, one of the most significant is that when moving code around you can accidentally change which buffers are being implicitly used for the unqualified fields. This can lead to bugs that are rather difficult to track down.

ProRefactor has the semantic and contextual knowledge necessary to determine which buffer name is being implicitly used for unqualified field names. This refactoring finds unqualified field names in the current file (must be a compilable file) and adds the appropriate table or buffer names.

Usage

  1. This refactoring operates on the currently selected file or the file that is currently open in an editor. The file must be compilable.
  2. From the main menu, choose ProRefactor > Qualify Field Names. The preview dialog appears, allowing you to accept or reject the changes.

Features

This refactoring recognizes FIELDS, EXCEPT, and USING phrases, and does not qualify field names within those.


Rename Schema Refactoring

This refactoring finds and changes all hard-coded references to database table and fields that you want to rename. You select the projects or directories that you want to change, input a list of old and new names, and the refactoring writes the modified files to an output directory which you can review before merging the changes into your source.

Usage

  1. In the Navigator view, select the files, directories, and/or projects to be refactored.
  2. From the Navigator view's context menu, choose ProRefactor > Rename Schema.
  3. The Rename Schema Refactoring wizard is presented. Input a list of old and new name pairs into the editor, or else enter the name of a file which contains old and new name pairs.
  4. This refactoring writes the modified files out to a temporary directory of your choice.

Features

  • Is not fooled by buffer names or by abbreviated table or field names.
  • Name qualifiers (db.table, table.field, db.table.field) are retained during the change.
  • A table can be renamed at the same time that fields in that table are renamed.

Caveats

  • Database name changes are not supported (logical, alias, or otherwise).
  • Buffer names are not changed.
  • Once this refactor is complete, and you have merged the results into your code, you should use a grep tool to find any remaining old name references in comments, strings, and preprocessor directives. Also consider any dynamic queries where you may be deriving the strings from external sources.

Example

Here is an example renaming list:

sports2000.customer client
sports2000.customer.name legalName
sports2000.invoice.amount totalAmount

Here is the code before refactoring:

find first customer.
display customer.name.
display cust.na.
display customer except name.
display customer.creditLimit.

find first invoice.
display invoice.amount.
display invoice except amount.

def buffer bCustomer for customer.
find first bCustomer.
display bCustomer.name.
display bCustomer except name.
display bCustomer.creditLimit.

def buffer bInvoice for invoice.
find first bInvoice.
display bInvoice.amount.
display bInvoice except amount.

Here is the code after Rename Schema refactoring has been run:

find first Client.
display Client.legalName.
display Client.legalName.
display Client except legalName.
display Client.creditLimit.

find first invoice.
display invoice.totalAmount.
display invoice except totalAmount.

def buffer bCustomer for Client.
find first bCustomer.
display bCustomer.legalName.
display bCustomer except legalName.
display bCustomer.creditLimit.

def buffer bInvoice for invoice.
find first bInvoice.
display bInvoice.totalAmount.
display bInvoice except totalAmount.

SUBSTITUTE Refactoring

This refactoring is intended to be used as part of a legacy code modernization project. It is unlikely to be used on a day-to-day basis.

This feature assists in the transformation from string concatenation to the use of a single string with SUBSTITUTE. This is done because the full sentence in a single string is easier to translate (for internationalization) than the sentence fragmented into two strings.

Usage

  1. In the Navigator view, select the files, directories, and/or projects to be refactored.
  2. From the Navigator view's context menu, choose ProRefactor > SUBSTITUTE Refactoring.
  3. ProRefactor will parse the selected resources, and find any string concatenations which might require refactoring. It will present the suspect code in the Review Changes dialog.
  4. Review, and then accept or reject the code change. You may make manual changes to the source code before accepting the changes.

In some cases, ProRefactor may be unable to automatically suggest changes to the source code. You will be informed of this by a message, and you will be able to make the code change by hand if desired.

In the Review Changes dialog, the Cancel button will reject the current changes and stop any further processing.


Extending ProRefactor

This section describes how to load ProRefactor source into an Eclipse workspace, so that you can use it and reference it from your own projects.

Experience programming in Java with Eclipse's IDE is not necessary, but since these instructions gloss over a few details, it would be helpful if you were familiar with Eclipse basics such as creating a Java project.

Prepare Your Eclipse Workspace

  1. In Window > Preferences > Java > Compiler, set the compliance level to 5.0. While in Java preferences, also go to Installed JREs, and edit your default JRE. Add -ea -Xss2M to the default VM args. This enables assertions, and sets the stack size to be large enough for recursively descending large syntax trees.
  2. Switch to the Java perspective.

You now have two choices for importing ProRefactor source. If you want to use the latest released version of ProRefactor as a library or framework, then follow the steps under Option One, below. If you want access to the version control repository for the very latest source code as it happens, then follow the steps under Option Two, below.

Option One: Import ProRefactor Plug-in Source

This option assumes that you have installed ProRefactor into Eclipse.

  1. Choose File > Import... > External Plug-ins
    and Fragments    .
  2. Choose The target platform,
    Select from all plug-ins, and
    Projects with source folders. Click
    Next.
  3. Select the org.prorefactor.* plug-ins, and
    click Finish.

Option Two: Checkout ProRefactor from Subversion

For this option, it is not necessary to have installed ProRefactor into Eclipse. See the ProRefactor Project page to gain access to the development project and repository. The project web page has instructions for anonymous access to the repository.

There are three top level directories in the ProRefactor repository: branches, tags, and trunk. You only want the trunk directory, not the entire project repository.

From the Eclipse menu, choose File > Import... > Existing Project into Workspace. Browse to select trunk as the root directory, and then Eclipse will allow you to import all of the projects at once.

Assigning java.library.path

In order for proparse.dll and proparse_jni.dll to be found, they must be in the Java library path. Those two DLLs are found in the org.prorefactor.core project directory. Select that project, and choose Properties > Info and copy the project location (full path). In Window > Preferences > Java > Installed JREs, edit your default JRE. Add: -Djava.library.path=/your/path to the default VM args, where /your/path is your org.prorefactor.core project directory.

Unit Tests and Run-time Workbench

Now you can run the unit tests. You can navigate to the following source file, and run it as a JUnit test:

To launch the Run-time Workbench, choose Run > Run As > Eclipse Application. From the Run-time Workbench, you should be able to select the ProRefactor perspective, create test projects, and launch the ProRefactor menu actions. This is where you would test any changes or additions that you make to ProRefactor or your own plug-ins.

Building the Tree Parsers

There are a few tree parsers which come within their own packages in ProRefactor, each serving its own purpose. You can use any of these as examples for your own tree parser, but a copy of the treeparserbase package should normally be your starting point. The treeparser01 package gives plenty of examples of how to add action code to the tree parser.

Antlr is a parser-generator which reads a .g grammar file in order to generate the Java source code for the tree parser.

We use Ant to do the build, but Ant has to be able to find Antlr. Choose Window > Preferences > Ant > Runtime > Classpath. Select Global Entries. Click Add Jars..., and select org.prorefactor.lib/antlr/antlr.jar file. Click Add Variable... and enter ${workspace_loc:/org.prorefactor.core/bin/}. This is required because there is additional post-processing which must be done after Antlr has generated the source for the tree parser. You can close this Preferences window now.

In the package which contains the tree parser, select the build.xml file. Right-click, and choose Run > Ant Build... to get the properties dialog.

Choose the Refresh tab, and enable the automatic refresh of the containing folder. Now when you choose Run > Ant Build, the refresh and re-compile will be done automatically.

Leave the JRE set to launch a separate JRE. Running from the same JRE as the workspace does not set up the class path correctly for our needs.

Your Own Plugin

This section describes how to create your own plugin, in order to add your own tools to the Eclipse workbench. It briefly describes in general terms how to contribute menus and actions to an Eclipse workbench, and it also gives a few initial pointers for creating menu actions which use Proparse and the ProRefactor class libraries.

This section assumes that you have followed the steps in the previous sections in this chapter.

  1. Choose File > New > Project > Plug-in Development > Plug-in Project.
  2. Enter your project name (ex: com.xyz.example), and if desired, specify the directory to create it in. Click Next. The defaults on the next page should be OK. Note that the Plug-in ID must be unique. Click Next.
  3. Choose Create a plug-in project using a code generation wizard, and choose Hello, World. Click Next.
  4. The defaults on the Content page should be OK, but you can change these settings if desired and then click Next.
  5. The defaults on the Action Set page should be OK, but you can change these settings if desired and then click Finish.
  6. Your new plugin should already be capable of being run and tested. Choose Run... > Run Time Workbench. Your Sample Menu that your new plugin contributes should appear in the Resource Perspective. However, there appears to be a bug when we run this test: The new Sample Menu only appears in the ProRefactor perspective. There is probably a bug in ProRefactor's plugin manifest. (3 Jan 2004) Your new plugin contributes the Sample Menu, as well as a new button on the button bar. Close your Runtime Workbench, and we'll add a new menu item.
  7. Open your plugin.xml, and click on the Dependencies page. Add the ProRefactor.org plug-ins as dependencies for your plug-in project. You also have to add those plugins to your project's build path. To do that, simply right-click on that Dependencies page, and choose the menu item Compute Build Path.
  8. In the Package Explorer, navigate to org.prorefactor.prorefactor.actions, then copy ParseAction.java, and paste it into your project's actions package.
  9. Double-click on your copy of ParseAction to open it into the Java editor. There are compile problems due to missing imports. Choose Source > Organize Imports to automatically add the necessary import statements. Save your changes, and press Ctrl-b to force a fresh build of ParseAction.
  10. Open your plugin.xml, and click on the Extensions page. Under Sample Action Set right-click on Sample Action and click Copy. Back on Sample Action Set right-click and click Paste. You should now have two identical Sample Action items.
  11. Double-click on the second Sample Action item, so that the Properties page appears. Change the class name to ParseAction. Change the id. Change the label to Simple &Parse. Delete the toolbarPath and the tooltip.
  12. Save your changes to your plugin.xml file, and then launch the Run Time Workbench again. You will have to load or create a small project with a valid .p program file in order to test your new menu item. All it does is parse the selected resource (or resources).

What Next?

Have a look at the ParseAction class that you took a copy of. It makes use of an instance of org.prorefactor.prorefactor.actions.ActionManager, which does all of the work of examining what resources are selected, load the parser, parse the selected resources, and show results in the Console. For each resource that it parses, it does a callback to processFile(), passing the integer node handle for the top node of the AST. In the case of ParseAction, no further action is taken for the given AST, but if you look at other class files such as org.prorefactor.prorefactor.actions.QualifyFieldsAction, you will see examples of doing some processing against the selected compile units. The same as we copied ParseAction, you can copy these other examples of launching parsing and refactoring actions.


Eclipse UI and Scripting

Scripting opens up an entire new world for working with ProRefactor and the syntax tree. As of version 1.7.2, the Groovy scripting engine has been added to ProRefactor's bundled libraries.

Using Scripts with ProRefactor in Eclipse

A menu item has been added to the context menu on the Navigator view: ProRefactor -> Tool Devel Utils -> Groovy Console. If you have a .p, .w, or .cls file selected when you launch the Groovy console, then within your console the File variable selectedFile is available for your script.

For example, you could select a .p, launch the Groovy Console, and then do something like this:

import org.prorefactor.core.JPNode
import org.prorefactor.core.TokenTypes
import org.prorefactor.treeparser.ParseUnit

indent = 0
def walker(node) {
	if (!node) return
	println '  '*indent + TokenTypes.getTokenName(node.getType()) + ' ' + node.getText()
	indent++
	walker(node.firstChild())
	indent--
	walker(node.nextSibling())
}

ParseUnit pu = new ParseUnit(selectedFile)
pu.loadOrBuildPUB()
walker(pu.topNode)

To see what variables are available in your Groovy Console, select Actions -> Inspect Variables from the console menu. The following little script shows something similar:

binding.variables.each {key, value ->
  if (key[0] == '_') return
  println "$key : $value"
  }

I found a way to get my own scripts directory into the classpath when using the console:

this.class.classLoader.addClasspath('/home/john/scripts')
println this.class.classLoader.classPath

ConsoleWaiter.groovy

Provides a wrapper for the console. (Download below)

The run() method launches the console and causes this thread to sleep until the console's window is closed. Allows easy interaction with the objects alive at a given point in an application's execution, similar to a debugger session.

Example 1:

new ConsoleWaiter().run()

Example 2:

def waiter = new ConsoleWaiter()
waiter.console.setVariable("node", node)
waiter.run()

NOTE: I've run across a documented bug. If launching a GUI Swing application (like the Groovy Console) from Eclipse running under Java 6, the JVM is likely to crash. This doesn't happen with Java 5.

I found that if I am editing .groovy scripts, and testing them from the console, that in some cases the .groovy class files get cached and don't get reloaded. I use the following to force the cache to be cleared. (Substitute MyScript with your own .groovy class name.) 

MyScript.classLoader.clearCache()

Links and Resources

www.joanju.com
Joanju provides commercial support and service for ProRefactor. Joanju wrote ProRefactor and released it to this open-source project in December, 2004. Joanju is also the home of Proparse, which is an integral part of ProRefactor.

www.peg.com
by Greg Higgins. The Progress Email Group, and host of the Proparse mailing list.

www.antlr.org
by Terence Parr. Antlr is our parser-generator of choice.

Directory Diff

Some of ProRefactor's refactorings write modified files (sometimes many files) out to a temporary directory, where you can review the modifications before merging those into your working source directory.

If you do not already have a utility for visually reviewing all file changes from one copy of a directory structure to another, then you will need one. Here are a couple of free alternatives. These free tools make it simple and pleasant to review large sets of modified files:


Prolint/Eclipse

I've started work on a Prolint plug-in for Eclipse, but just written in Java, rather than ABL. Because it relies a lot on ProRefactor, I've just included my work so far in the ProRefactor SVN repository.

The Prolint/Eclipse page in the Prolint book is where you can find a bit more information and maybe a couple of pretty pictures. :)


Troubleshooting

Eclipse shuts down when parsing

Summary

Add -vmargs -Xss2M to your shortcut for running Eclipse.

Details

Proparse requires a large stack in order to store all nodes of a large program in memory. Since it was Java that loaded Proparse, it is the Java Virtual Machine which needs to be configured for running with a large stack.

If the stack size is not large enough, the JVM will shut down with a stack overflow error, but you will not see any error details when launching Eclipse and javaw.exe as Windows processes. It doesn't take a huge program to cause this stack overflow. A program with one or two thousand lines of COMPILE statements is enough to do it. The stack size for javaw.exe by default is 1MB. The argument -vmargs tells eclipse.exe to pass any following arguments on to the JVM. The argument -Xss2M tells the JVM to set the stack size to 2MB.


License

ProRefactor.org Software User Agreement

29th December, 2004

Usage Of Content

PROREFACTOR.ORG MAKES AVAILABLE SOFTWARE, DOCUMENTATION, INFORMATION AND/OR
OTHER MATERIALS (COLLECTIVELY "CONTENT").
USE OF THE CONTENT IS GOVERNED BY THE TERMS AND CONDITIONS OF THIS AGREEMENT
AND/OR THE TERMS AND CONDITIONS OF LICENSE AGREEMENTS OR NOTICES INDICATED OR
REFERENCED BELOW.
BY USING THE CONTENT, YOU AGREE THAT YOUR USE OF THE CONTENT IS GOVERNED BY THIS
AGREEMENT AND/OR THE TERMS AND CONDITIONS OF ANY APPLICABLE LICENSE AGREEMENTS
OR NOTICES INDICATED OR REFERENCED BELOW.
IF YOU DO NOT AGREE TO THE TERMS AND CONDITIONS OF THIS AGREEMENT AND THE TERMS
AND CONDITIONS OF ANY APPLICABLE LICENSE AGREEMENTS OR NOTICES INDICATED OR
REFERENCED BELOW, THEN YOU MAY NOT USE THE CONTENT.

Applicable Licenses

Unless otherwise indicated, all Content made available by ProRefactor.org is
provided to you under the terms and conditions of the Eclipse Public License
Version 1.0 ("EPL").
A copy of the EPL is provided with this Content and is also available at

http://www.eclipse.org/legal/epl-v10.html

.
For purposes of the EPL, "Program" will mean the Content.

Content includes, but is not limited to, source code, object code, documentation
and other files maintained in the ProRefactor.org repository
and made available as downloadable archives.