The OpenEdge Hive

Here's some recent questions from Thomas and answers from me which I'd like to share here.

> There are references to classes which are clearly not there in the ABL,
> so I presume these are in the assemblies? What kind of documentation is
> there for those ... other than downloading the Java code and reading it?
>
> Do you have a recommendation for where I should look to understand how
> the various pieces work and what capabilities there are? Is ProLint a
> good source of examples ... or is it significantly incomplete relative
> to what you have built in to the tools?
>
> E.g., ParseUnit and JPNode seem like classes I would like to understand
> and know how to manipulate. The operations in ExamineNode are a tiny
> start, but I presume there is a lot more.
The Java code is indeed the only real source of documentation.

In terms of examples, AutoDox2 is the latest and possibly best example of using Proparse. You already have access to the source code on https://joanju.repositoryhosting.com/. AutoDox2 is relatively small and easy to understand, at least compared to Proparse. You would have to understand the XML 'builder' concept in Groovy to make sense of AutoDox2, but it's not terribly tough.

Prolint is not invalid, but it uses the clumsy old API from the C DLL days. Today you wouldn't write anything that way - you'd use direct access to the classes and objects in Proparse.

> Is there something like Javadoc for it?
Yes, the java code for Proparse has plenty of javadoc comments, and it generates a pretty large body of javadoc HTML documentation.

> node:toStringFulltext() gives me the whole line including the preceding
> comment. While I can see the point of this in some cases, there are
> others where I would like just the line itself. node:toString is just
> the keyword. node:getComments lets me get the comments on their own.
>
> How should I be getting the actual source line without the comment.
>
> Is there a basic technique for getting the sequence of tokens, e.g., to
> break down a run statement into its component parts?
Can you be more specific? ie Can you provide an example RUN statement you want to parse, and what you want to see for output?

> My first pass was just to see the text of the line. Given the ADM
> source, the comments are more a distraction than a help. I tried
>
> define variable chComments as character.
> define variable chFullText as character.
> chComments = node:getComments().
> chFullText = node:toStringFullText().
> if length(chComments)> 0
> then chFullText = substring(chFullText,length(chComments) + 1).
>
> but that tends to get me the end of the comments in many cases.
>
> More to the point, of course, is wanting to get the sequence of token so
> that I can tell what is a run of a program, what is a run of an IP, what
> is being referenced, etc.
Once you have a RUN node, then use the firstChild and nextSibling etc methods to examine the structure of the run statement.

Like with programming Prolint, extensive use of a "tokenlister" helps you see the structure of the syntax tree (with your sample statements) so you can program with it.

> After lots of experiments, this seems to work:
>
> obToken = ipobNode:firstChild().
> chReportLine =
> chReportLine
> + obToken:getText()
> + "("
> + NodeTypes:getTypeName(obToken:GetType())
> + ")|".
>
> repeat:
> obToken = obToken:NextSibling().
>
> if obToken = ? then leave.
> inTokenCount = inTokenCount + 1.
> chReportLine =
> chReportLine
> + obToken:toStringFullText()
> + "("
> + NodeTypes:getTypeName(obToken:GetType())
> + ")|".
> if obToken:getType() eq ProParserTokenTypes:period then leave.
> if inTokenCount> 9 then leave.
> end.
>
> The inTokenCount is just a safety net, so if there is a better way to
> ensure stopping at the end, advice is welcome.
>
> Still a bit confused about the various options for text ...
You should grab the source for Proparse via svn and have a look at the comments in JPNode. I think you've found the basic 'text' type methods, but there are lots more of interest in there.

You should also grab the source for Prolint and look at the 'shim' layer between old Prolint code and more recent Proparse.Net.

A statement like RUN is a branch in the tree - you are at the end when there are no more nodes. That's assuming you are using recursion to descend through its children and its children's siblings.

An alternative is using TreeUtils.flatListAsArray, which would get you an array of all the nodes in the branch.

> obToken:GetType()
> is giving me 269 for the firstChild of the RUN statement. Is that
> reasonable, even though many of them are IP names?
String typename = NodeTypes.getTypeName(269)

typename is FILENAME

yes, that's expected. The lexer assigns most of the node types, the parser only fine tunes a few. The parser's job is mostly to shape the tree. The parser takes a token stream from the lexer, and is in no position to figure out if it's an internal or external proc name.

> You don't happen to remember what the "shim" is called, do you?
There's a proparse-shim subdirectory

> As you saw from the code, I wasn't using recursion. What is the test
> that tells you there is no more?
nextSibling or firstChild is null

> A Java array or an ABL array?
yes, or .net array, depending on where you are programming from

> I am switching the code now to do something more useful than a report
> which seems to be going OK. The biggest question at the moment is how
> to tell if it is an IP or a program. I have something working based on
> .p and .w, which works for some code bases, but isn't workable for a
> Varnet base, for example.
Assuming there are no super procedures, then you need to build a list of PROCEDUREs defined in the compile unit to tell if a RUN is for an internal or external. If there are super procedures in the app then, as you know, it is extremely complicated.

> What is a "hidden" token?
The lexer identifies whitespace and comment lexemes as "hidden" tokens, in other words, they are not and cannot be nodes in the syntax tree. Traditionally a compiler would simply discard these.

The parser consumes the stream of lexemes to build the syntax tree. Antlr has a feature which allows these 'hidden' tokens to become linked object attributes of the nodes in the syntax tree. Proparse uses that feature.

> So "hidden" is antlr vocabulary?
>
> I see some possibly interesting method that return JPNode[]. Does one
> just then define a variable of type class with an extent to use for the
> return ... and guess at the size needed?
Yes, "hidden" is an antlr term.

I don't remember if I've used any of the JPNode[] methods from ABL. You might look through the shim in Prolint in case there are any examples in there. I'd hope that the ABL support for defining arrays with unknown size would do the trick here.

> Is there any way with the Proparse scripting to tell what include files
> a compile unit contains? I see the ProParserTokenTypes INCLUDEREFARG ,
> but that doesn't sound like the right thing. I also see that there are
> classes IncludeFile, IncludeRef, and IncludeExpansion
Have a look at ParseUnit.java: public String [] getFileIndex()

It calls JPNode: String[] getFilenames()
...which has more helpful comments.

> I earlier created a Proparse project in PDS and imported the code I
> downloaded into a Java project. Do you know why the src/org directory
> shows up, but has no apparent contents? It won't let me import the
> contents of that directory either saying that it is already in the
> hierarchy. I can see the ParseUnit.java file in the filesystem under
> the project, but can't see it in Eclipse.
I did an svn checkout and imported (File -> Import existing project) into Eclipse. 'org' shows up as an empty package, but everything else is there, like ParseUnit.java. Sorry, I don't know why it's not showing up correctly in your Eclipse workspace.

> In CustomProcessing I tried:
> define variable chFilename as character extent 100 no-undo.
> define variable inWhich as integer no-undo.
> define variable lgFoundOne as logical no-undo initial false.
>
> chFilename = ipobParseUnit:getFileIndex().
> do inWhich = 1 to 100:
> if chFilename[inWhich] = ? then leave.
> lgFoundOne = true.
> if inWhich = 1 then put stream stFilenames unformatted
> obTopNode:getFilename() ":".
> put stream stFilenames unformatted chFilename[inWhich] "|".
> end.
> if lgFoundOne then put stream stFilenames unformatted skip.
>
> This compiles and runs, but produces no output. Is the =? test the
> wrong thing? Or do I need a different handling of the target of the
> getFileIndex(). I haven't done any of this cross language stuff before.

Try: def var chFilename no-undo extent.
Number of extents unknown until it is assigned.
After assigning it, use EXTENT(chFilename) to get number of extents, like:

do inWhich = 1 to EXTENT(chFilename)

Note that JPNode internally works with a zero based array, so you'd want to use:
chFilename[JPNode:getFileIndex() + 1]
...this is the equivalent of JPNode:getFilename()