Parsing test
James and I are continuing to have NullPointerException problems. I wrote a program which just goes through the files in a directory tree and does a Proparse parse of them, with the hope that this would show us the files with parsing errors. *But*, we are getting no errors from that. But, in the first ABL2DB task where we use Proparse, we are getting the NullPointerExceptions. The only thing that occurs to me as an explanation is that Proparse successfully builds the parse tree, but there is something wrong with it so that when we start to navigate the tree it steps in a hole. Does that seem possible to you?
If so, what would you recommend as test to extend the existing scan which would help find the holes?
Now that I am finally back on this, we only have about a month before PCEMEA and I would really like to have James code as a part of the talk.
- Login to post comments
Thanks John. So very much
Thanks John. So very much appreciated.
So, where does this leave us
So, where does this leave us on open issues?
As far as I'm concerned it's
As far as I'm concerned it's just the issue we can't recreate in a simple way and is therefore not on John's list. I'm happy to say everything is therefore resolved. (assuming this is working - I haven't tested).
I have just completed a
I have just completed a TestScanner run and I still have a error on InputParameter.p, which is the one with a line like:
DEFINE INPUT PARAMETER ip-CtpApplicationMeaningType LIKE INPUT ISrel._USER._USERID NO-UNDO.
where it doesn't like the second INPUT. To me, this looks like something that should annoy the compiler, but it seems to be legal.
I've removed that particular
I've removed that particular nut from our code base as it works without the extra INPUT.
So, you are convinced it is
So, you are convinced it is non-meaningful and just fixed the code? And, didn't have too many instances?
Certainly in our case it was
Certainly in our case it was non meaningful. We only had a small number of instances.
So, fixing the INPUT in a
So, fixing the INPUT in a parameter definition is optional ... since we may never again encounter an example, and the only known issue is the one you can't reproduce in a simple example because of the complexity of includes and preprocessor statements.
Correct?
Have you done a run with the new release yet?
Yes correct. Not run a test
Yes correct.
Not run a test yet. Will hopefully do that later. Been a busy weekend with my day job.
Sunday night isn't usually
Sunday night isn't usually "day job".
True, although out of hours
True, although out of hours maintenance is my day job. Sort of!
Full run of build blocks -
Full run of build blocks - no new issues.
Great. Perhaps we should
Great. Perhaps we should start a new thread if anything comes up ... and do it on the Proparse forum! :)
Couple of comments. 1) All
Couple of comments.
1) All issues mentioned in the release notes do seem to have been fixed, including the freezing. Thank you.
2) Unfortunately there are some new issues introduced. I'm trying to recreate those now and will post up code shortly.
Another preprocessor
Another preprocessor problem:
ON ANY-PRINTABLE,BACKSPACE,DELETE-CHARACTER,CTRL-V OF {&BrowseField} IN FRAME {&SearchFrame}
I can't simply reproduce this either as it's heavily nested in icode.
[15/10/09@12:37:42.347+0100] P-003852 T-009144 1 4GL APPL ScanOneCU: Other error in C:\Users\jpalmer\Copy\ABL2DB\ICMAS\test\csiselwt.w: org.prorefactor.refactor.RefactorException: C:\Users\jpalmer\Copy\ABL2DB\ICMAS\test\lookup.i:311:70: unexpected token: IN
So, in this case it seems to
So, in this case it seems to have done something funny with the {&BrowseField} so that it doesn't recognize the IN.
How about the first one? Can you show us the line of code it failed on and point to the column of failure?
The code in question is:
The code in question is:
&IF DEFINED(EXCLUDE-CKL-FILTER-CHECK) EQ 0 &THEN
IF fn-CheckFiltered(BUFFER lb-CKL) THEN NEXT.
&ENDIF
The error was on the middle line and along the lines of getting '(' but expecting THEN
Got another example that's
Got another example that's harder to reproduce but it's also surrounding prepocessors etc so I'm expecting it's to do with the same thing. If that doesn't go away with the fix on the previous then I'll report it later.
First issue. Seems to be to
First issue. Seems to be to do with preprocessors. See attached.
[15/10/09@12:29:03.260+0100] P-006528 T-007384 1 4GL APPL Parsing C:\Users\jpalmer\Copy\ABL2DB\ICMAS\test\chsfax.p
[15/10/09@12:29:03.491+0100] P-006528 T-007384 1 4GL APPL ScanOneCU: Other error in C:\Users\jpalmer\Copy\ABL2DB\ICMAS\test\chsfax.p: org.prorefactor.refactor.RefactorException: C:\Users\jpalmer\Copy\ABL2DB\ICMAS\test\chsfax.p:20: Unknown table name: tt-Diary
[15/10/09@12:29:03.493+0100] P-006528 T-007384 1 4GL OtherERROR Other error (cont.): antlr.SemanticException: C:\Users\jpalmer\Copy\ABL2DB\ICMAS\test\chsfax.p:20: Unknown table name: tt-Diary
[15/10/09@12:29:03.494+0100] P-006528 T-007384 1 4GL CALLSTACK DoParse com.cintegrity.Proparse.ScanOneCU at line 80 (C:\Users\jpalmer\Copy\ABL2DB Workspace\ABL2DB\src\com\cintegrity\Proparse\ScanOneCU.r)
[15/10/09@12:29:03.496+0100] P-006528 T-007384 1 4GL CALLSTACK ScanOne com.cintegrity.Proparse.TestScanner at line 158 (C:\Users\jpalmer\Copy\ABL2DB Workspace\ABL2DB\src\com\cintegrity\Proparse\TestScanner.r)
[15/10/09@12:29:03.497+0100] P-006528 T-007384 1 4GL CALLSTACK ScanAll com.cintegrity.Proparse.TestScanner at line 79 (C:\Users\jpalmer\Copy\ABL2DB Workspace\ABL2DB\src\com\cintegrity\Proparse\TestScanner.r)
[15/10/09@12:29:03.499+0100] P-006528 T-007384 1 4GL CALLSTACK Process com.cintegrity.Proparse.TestScanner at line 72 (C:\Users\jpalmer\Copy\ABL2DB Workspace\ABL2DB\src\com\cintegrity\Proparse\TestScanner.r)
[15/10/09@12:29:03.500+0100] P-006528 T-007384 1 4GL CALLSTACK C:\Users\jpalmer\Copy\ABL2DB Workspace\ABL2DB\src\TestScanner.p at line 88 (C:\Users\jpalmer\Copy\ABL2DB Workspace\ABL2DB\src\TestScanner.r)
Bug introduced in previous build
Tsk tsk. Shame on me. The previous build introduced a new bug. I fixed it and posted a new build. Sorry about that.
Yes I can confirm this
Yes I can confirm this solves the preprocessor issues.
So, are you left with the
So, are you left with the same issues as I was?
Yes indeedy.
Yes indeedy.
OK, I have completed a
OK, I have completed a TestScanner run on my code base. It seems to have taken slightly longer than previously ... perhaps 24 minutes versus 20 ... but it did not throw any errors other than the InputParameter.p and MessageDigest.p ones from before. Of course, my code is notably lacking in preprocessor commands, so it will testing on James code which will tell us more about that aspect.
We'll take it for a spin ...
We'll take it for a spin ... you sounded a bit tenuous about the last one, so perhaps it is not a big surprise.
r-code
By the way, I found that r-code for a program referencing proparse.net.dll had a reference to the specific version of the DLL that the r-code was compiled against. Just a heads-up!
Isn't the only implication
Isn't the only implication that a new version of Proparse implies a need to recompile any programs which reference it? I would think that would be expected in any case.
New build
I posted a new build of Proparse. Give it a whirl and let me know where we're at!
I've done a TestScanner pass
I've done a TestScanner pass on the code I have here and it went through just fine. The only error was the InputParameter.p one. I don't have the long IF code though. James will take a shot at it tomorrow.
What is covered by the new
What is covered by the new build? James has to put stuff back that he has excluded, so if this doesn't address the stops, it would be a big advantage not to put those back yet.
Ah, I see that it does address that in the release notes.
if defined, long else if statements
I managed to come up with a fix for &IF DEFINED. I'll let you know when I post a new build.
Now I'm looking at the long IF THEN ELSE IF THEN statement. I see what you mean. It looks like the time to parse grows exponentially, and anything more than about 10 ELSE IF conditions becomes too much. I doubt this will be easy to fix. I'll let you know!
Interesting. So, it isn't
Interesting. So, it isn't so much a question of it getting too big and falling off the end as an exponential progression which becomes seemingly infinite ... or at least exceeding our patience to wait it out?
With the programs which
With the programs which cause a stop moved out of the way, we are now getting full passes where the only errors relate to the known reported issues. The one we have not succeeded in replicating in isolation is the one with multiple preprocessors and the function definition and there are only 4 instances of that ... and it is a report, so not critical to processing.
Another one-off:
Another one-off:
COPY-LOB ICMAS.BinaryObject.BloObject TO lv-Mem.
lv-Digest = MESSAGE-DIGEST("SHA-256",lv-Mem).
I think it's the MESSAGE-DIGEST keyword it doesn't like.
Here's a test program for
Here's a test program for this issue. It fails with
ScanOneCU: Other error in C:\Users\jpalmer\Copy\ABL2DB\ICMAS\test\blowr.p: org.prorefactor.refactor.RefactorException: C:\Users\jpalmer\Copy\ABL2DB\ICMAS\test\blowr.p:4:31: unexpected token: (
Other error (cont.): antlr.NoViableAltException: NoViableAlt
line 4 character 31 is at the end of MESSAGE-DIGEST.
New release supports MESSAGE-DIGEST
I just published a new release which I hope will deal with the syntax for MESSAGE-DIGEST.
Here's another one. This one
Here's another one. This one doesn't complete the parse. It just seems to hang itself up completely. No errors.
The issue seems to be the
The issue seems to be the assign var = if this then this else this else this else this type sort of thing. I've got LOADS of examples of it in my code.
This should be a priority to fix if at all possible as it's not throwing errors, just hanging and I have to start from the top again each time it happens.
var = if...
That looks interesting. I don't see anything in the sample code which should be a problem for Proparse. I'll look into it.
John, do you have some kind
John, do you have some kind of intermediate structure you use while building the tree for a single statement ... might this statement simply be too long or complex?
I should be able to look at
I should be able to look at it this weekend. Please make sure the sample is complete and will compile clean.
Sample Programs
OK, Here are some test programs. All compile and all fail in Proparse.
The first, IfDefined.p is the UIB like code above. It fails with
org.prorefactor.refactor.RefactorException: java.lang.RuntimeException: java.lang.IllegalArgumentException: c:\work\IS\Joe\src\IfDefined.p:1 Bad DEFINED function in &IF preprocessor condition
The second, InputParameter.p fails with:
org.prorefactor.refactor.RefactorException: Could not find input field ISrel._USER._USERID c:\work\IS\Joe\src\InputParameter.p:5 -> File: c:\work\IS\Joe\src\InputParameter.p Line: 5 Column: 8 Type: INPUT Text: INPUT
This clearly relates to the second INPUT which the compiler allows, but seems to politely ignore. See discussion here:
https://community.progress.com/community_groups/openedge_development/f/1...
The third, SystemDialog.p fails with:
org.prorefactor.refactor.RefactorException: c:\work\IS\Joe\src\SystemDialog.p:9:1: unexpected token: UPDATE
antlr.NoViableAltException: NoViableAlt
This clearly points to the UPDATE, which might be V11.x syntax.
We have one more, which is the series of &IF DEFINED shown above, which I have not succeeded in duplicating yet.
Enjoy!
Re: Sample Programs
Hi guys,
The UPDATE option is not V11 syntax. It is undocumented. But, it was easy enough to add.
The &IF DEFINED code is pretty ugly. I don't think it will be quick or easy to make Proparse accept it. I imagine if it were changed to &IF DEFINED("EXCLUDE-352729.2"), then it would work both in the compiler and in Proparse. I'm going to test that out.
I haven't looked at the extra INPUT token yet.
How common are these in the code - do they affect a lot of files? I can't spend a lot of time adding undocumented quirks to Proparse. Thomas, can you come up with a simple workaround for your project? Would it be reasonable to work with a tweaked copy of the code that feeds cleanly through Proparse? Usually the errors that come from Proparse are pretty clear to tell exactly what code isn't being accepted.
Who knows about the decimal
Who knows about the decimal procedure name ... it does look like a problem waiting to happen. Any idea where this came from, James.
I have asked James to do some scans to see how common some of these are. He has already excluded a bunch of code from the test because it is code rarely used or that he hopes to replace with another tool.
One big problem has been that we can't get a scan to complete, so we never get a full set of errors. I thought I might get around this by restructuring so that each compile unit got a fresh copy of the Proparse environment so there was no contamination, but it is still happening. It appears to start parsing a unit and then never returns and no error. This is the code sample below which James has attached. If we could get past whatever that is, we could at least get a complete run and find out how many of each error we had.
Some of these are messy in the ABL2DB context because the failure of the Proparse task means that a table has not been built which means that the ABL piece spits out a bunch of errors since all the lookups in the table throw an error. Now that the TestScanner is working properly, it is a lot cleaner. But, the hang up is still a problem since it never goes on to finish.
Logging errors
Good grief, no wonder you are having so much trouble. When I was doing Proparse projects in Java, I would just have my loop catch any exception from Proparse, log it, then move onto the next compile unit.
If you can't get OpenEdge to nicely catch and log the Proparse exceptions, you might want to write a Java "TestScanner" instead. At least that way we'd have an exception list to work from.
I wonder if the problem is between OpenEdge and .Net, or if the problem is with the way IKVM builds Proparse.net from Proparse.jar.
The TestScanner now does a
The TestScanner now does a good job of everything except the ones that freeze, possibly due to the very large assignments with IF. It reports the error, gives a stack trace, and all that and then goes on with the next one. The one place it has a problem is with the freeze ones and there is no error generated from that. It just freezes and doesn't do anything.
I have just heard from James that he has managed to isolate and remove all of the ones where it was freezing, so he now has a complete TestScanner run. There remain 8 programs with NullPointerException which he is going to investigate.
The decimal procedure names
The decimal procedure names aren't actually decimals. The first part is the customer key, the second part is the section of the code. So for 20 odd customers there are 20 odd sets of IPs with this naming convention. The calls are constructed dynamically. A mess. I can exclude this piece of code from the parse if necessary, but would be nice to have it fixed.
But we also have other non-standard IP names such as:
&IF DEFINED(EXCLUDE-"S&SPreparse") = 0 &THEN
&ANALYZE-SUSPEND _UIB-CODE-BLOCK _PROCEDURE "S&SPreparse" win-icmas-main-stub
PROCEDURE "S&SPreparse":
This also throws an error.
I wonder if it's actually the "" around the procedure name that's the issue rather than the contents therein?
The decimal procedure names
I'm starting to remember why I stopped peddling Proparse.
The decimal procedure names
Oh - I hadn't noticed that the procedure names have quotation marks in them, so I misunderstood part of what was going on. It's interesting, you can't compile &SCOPED-DEFINE EXCLUDE-"352729.2", because the preprocessor does not accept the quotation mark. So I don't think there's any way for &IF DEFINED(EXCLUDE-"352729.2") to ever be true. If I &SCOPED-DEFINE EXCLUDE-352729.2, that doesn't seem to do the trick.
I can only guess that for &IF DEFINED(...) the preprocessor is combining the text of all the tokens between the two parens. This compiles and runs:
...but if I have any mismatched quotes between those parens, it fails to compile.
Aren't the quotes there
Aren't the quotes there because otherwise the compiler would have trouble with the period in the procedure name? Admittedly, one would think the person would have used a different character rather than gone for the quotes, but it is at least understandable ... in a perverted way!