A parser accepts a stream of tokens, and then builds the tree. Proparse's token stream has some token types that get filtered and attached to nodes as hidden tokens.
com.joanju.proparse.DoParse.java:
filter.hide(NodeTypes.WS); filter.hide(NodeTypes.COMMENT); filter.hide(NodeTypes.AMPMESSAGE); filter.hide(NodeTypes.AMPANALYZESUSPEND); filter.hide(NodeTypes.AMPANALYZERESUME); filter.hide(NodeTypes.AMPGLOBALDEFINE); filter.hide(NodeTypes.AMPSCOPEDDEFINE); filter.hide(NodeTypes.AMPUNDEFINE);
Those are the only 'hidden' tokens that get into the stream. (WS is WhiteSpace)
The other tokens, like &IF, are dropped and don't get into the stream because they themselves change what gets into the token stream.
&IF FALSE &THEN blah blah blah &ELSE display 3. &ENDIF
The "blah blah blah" doesn't get into the stream at all, not even as hidden tokens. Neither does the &IF, &ELSE, &ENDIF. The text is just dropped.
There is, however, another set of data entirely, and that is the "macro source graph", which contains a lot of the data you would go looking at a COMPILE..LIST for, including all the &IF, {&whatever}, and {myfile.i} references. The Java objects in the Macro Source Graph come from this:
http://www.joanju.com/dist/docs/api.html#ListingFile
and those Java objects are in the blob. You can see them in the UML I posted. We haven't started working on the API for those yet.
this gives me some problems
this gives me some problems : for example, in 90% of my .w files created in the appbuilder, I have
&IF DEFINED(UIB_IS_RUNNING) &THEN
DEF VAR ip_foo AS CHAR NO-UNDO.
&ELSE
DEF INPUT PARAMETER foo AS CHAR NO-UNDO.
&ENDIF
this allows me to run the .w directly from the appbuilder.
What I think that you are saying is that depending on where UIB_IS_RUNNING is set, either the DEF VAR _or_ the DEF INPUT would be used, and the other dropped.
If I was trying to refactor some other part of the code then I would lose this when writing out the new source ?
We don't write out the new
We don't write out the new source directly from the syntax tree.
Our refactoring logic has two phases. The first phase analyzes the syntax tree, and makes a list of files, lines, columns that need changing.
In the second phase, we change the text of the source files, but only the portions of the lines of text that need to be changed. For example: