The OpenEdge Hive

If a call to ShellExecute succeeds, the returned hInstance parameter will be the instance handle of the executed program. But if the hInstance is less then 33, it indicates a failure.
Actually if the hInstance>=0 and hInstance<33 because a very large hInstance will be casted into a signed integer variable and may appear to be negative from Progress' perspective.
This page lists the descriptions for some of the possible failures.

It's actually a combination from different resources, both from WinExec and other procedures so some of the result may never occur when using ShellExecute. There are also some gaps in the list, and moreover: some results should have a different description in 16-bit mode. Where descriptions in 16-bit are different then 32-bit I've picked the 32-bit description

{windows.i}
 
DEFINE VARIABLE hInstance AS INTEGER.
 
RUN ShellExecute{&A} IN hpApi(0,
                              "open",
                              "xyz.doc",
                              "",
                              "",
                              1,
                              OUTPUT hInstance).
 
RUN TestInstance (hInstance).
IF RETURN-VALUE > "" THEN
   MESSAGE "ShellExecute failed, reason: " SKIP
           RETURN-VALUE
           VIEW-AS ALERT-BOX ERROR.
 
 
 
PROCEDURE TestInstance :
 
DEFINE INPUT PARAMETER hInstance AS INTEGER. /* =return value from ShellExecute */
DEFINE VARIABLE txt AS CHARACTER NO-UNDO.
 
IF hInstance<0 OR hInstance>32 THEN RETURN "". /* not failed */
 
CASE hInstance :
  WHEN  0 THEN txt = "The operating system is out of memory or resources.".
  WHEN  2 THEN txt = "The specified file was not found".
  WHEN  3 THEN txt = "The specified path was not found.".
  WHEN  5 THEN txt = "Windows 95 only: The operating system denied " 
                      + "access to the specified file".
  WHEN  8 THEN txt = "Windows 95 only: There was not enough memory to "
                      + "complete the operation.".
  WHEN 10 THEN txt = "Wrong Windows version".
  WHEN 11 THEN txt = "The .EXE file is invalid (non-Win32 .EXE or "
                      + "error in .EXE image).".
  WHEN 12 THEN txt = "Application was designed for a different operating system".
  WHEN 13 THEN txt = "Application was designed for MS-DOS 4.0".
  WHEN 15 THEN txt = "Attempt to load a real-mode program".
  WHEN 16 THEN txt = "Attempt to load a second instance of "
                      + "an application with non-readonly data segments".
  WHEN 19 THEN txt = "Attempt to load a compressed application file".
  WHEN 20 THEN txt = "Dynamic-link library (DLL) file failure".
  WHEN 26 THEN txt = "A sharing violation occurred.".
  WHEN 27 THEN txt = "The filename association is incomplete or invalid.".
  WHEN 28 THEN txt = "The DDE transaction could not be completed " 
                      + "because the request timed out.".
  WHEN 29 THEN txt = "The DDE transaction failed.".
  WHEN 30 THEN txt = "The DDE transaction could not be completed because "
                      + "other DDE transactions were being processed.".
  WHEN 31 THEN txt = "There is no application associated with "
                     + "the given filename extension.".
  WHEN 32 THEN txt = "Windows 95 only: The specified dynamic-link " 
                     + "library was not found.".
  OTHERWISE    txt = "undocumented".
END.
 
RETURN txt.
 
END PROCEDURE.