Replace results template with revised one without losing parameters

[Britton Ward 0]

Hi,

 

I have setup to parse a text file with a list of values by adding the file as a results template.  I have then manually selected the results variables and then linked these to parameters.

 

After doing this I realized some items were missing from the result file and so regenerated it with the new values appended.

 

I can't see a way to replace the results template with the updated one in the software connector and so I cannot add the new values as additional results parses and parameters.

 

It seems I have to delete the results template and re-add it and then manually redefine all the results parses I did previously.  It is some 50 values so it is a bit time consuming.

 

Is there a way to refresh the results template without losing my results definitions?

 

Separately, it would be really nice if there were functionality to read a list of name, value pairs and automatically derive the parameter names from the name list in one click?

 

Also, some of my software now output XML files directly. Do you have any plans to add some XML parsing functionality into the software connector template selection?  It would be nice to use XML query tags to pull items from a file .. something for a rainy day!

[Mr. Arne Bergmann 27]

Hi Britt,

 

indeed, updating a result template is currently not possible, as far as I can tell (unfortunately my colleague who knows most about the templates will not be in office until January). There is a workaround, though (I admit, it's not really a nice workaround, but it should be better than setting up those 50 result values again):

• Copy the content of the template
• Go to a website that allows you to convert strings to base64 encoding (e.g. https://www.base64encode.org/)
• Paste the content of the template and edit the desired changes
• Encode the string and copy the result

Now, you need to find out the object name of the template. It is derived from the filename of the file that was used as a template, it is just cleaned of characters that are not allowed in an object's name (e.g. '.' or '-'). The easiest way to determine the resulting name is to (in a new project) create any object (e.g. a Parameter) and try to rename it to the original filename. The resulting name should be the same as the generated name of your template. For the rest of my post I assume the template's object name is "data".

With that info, go back to your project and enter "data.setContent("[b64]<encoded base64 data>")" in the console, replacing "<encoded base64 data>" with the result you got from the encoding website. This should do the trick (at least it worked in my test right now - all result values are still there and my changes are there as well).

I will create a backlog entry in our change management system that there should be an easier way to do this.

 

The list of parameter names is a good idea, I will "take it on the list", as well.

 

Regarding XML: I like the idea of integrating XML in the template mechanism (it's been brought up internally a couple of times, but the requests for it were too few to actually do it).

We already have a XML result file type (xffl). I must admit that it is not very well documented, though, and that I don't have a lot of experience with it (a search in the Documentation Browser for "xffl" will give you some results, though). If you enter an XML file in the right XML dialect, CAESES will automatically extract the objects that are contained in it. It is mainly for basic types (e.g. doubles, vectors, strings). E.g. the XML file 

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE xffl>
<xffl name="caseDef1">
  <FDouble name="foobar">
    <value>7</value>
  </FDouble>
</xffl>

Would be imported as a single double value with the name "foobar". Creating a parameter by double-clicking in the result table will create a parameter with the name "eval_foobar".

I assume, your tools do not produce this kind of XML, though (of course). In that case, it is possible to use XSLT (http://en.wikipedia.org/wiki/XSLT) to transform any XML file in the correct format. But unfortunately, I just realized that XSLT is only available for the Custom integration type. This should not be the case and I will change that for 3.1.3. Once that is released (and if you are interested) I can explain more about how to use XSLT transformations in the integration.

 

Best regards,

Arne

[Britton Ward 0]

Arne,

 

Thanks for your thoughts. I'll give it a go ... it seems I am pretty good at finding these little issues.. you all must be getting tired of me!

 

Is there a concise summary on the preferred methods of how a project accessing external software should be using the file structure commands?

 

I seem to be stumbling a bit in making sure program B) gets the right file from program A)  .. and what the framework is actually doing with moving files from input directories etc.?

 

I gather that you need to use getInputDir() and getResultsDir()  in order to insure that the dependencies are correctly constructed?

 

I'm really close through trial and error but would be nice if there was a tutorial that walked you through the actual disk processes that occur ..

 

Britt

[Mr. Arne Bergmann 27]

Hi Britt,

 

Unfortunately, I am not aware of such a guide.

 

First of all: From your question, it seems like you are trying to "concatenate" multiple Software Connectors with each other? If that is the case, there is an inbuilt mechanism, that takes care of everything file related: In the SoftwareConnector for program B, push the little button that looks like the SoftwareConnector icon on the top right of the "Input Files" pane. You will get a list of all other SoftwareConnectors in the project with their result files. Select the file that you want and everything else will be taken care of automagically.

 

If you really need to do this manually (e.g. for exports/imports in Features), here are some pointers (which may also serve as a start for a guide like you requested):

• The getInputDir()/getResultsDir() commands are the right way to go
• The difference between those commands is only really relevant when trying to export files for a computation that is supposed to run remotely (in that case the "getInputDir()" command must be used (otherwise the exported file will not be transferred to the remote computer). If the computation runs
• If you use those commands without arguments, they will return the corresponding directory of the computation that is CURRENTLY EXECUTED. If no computation is running (or is currently being prepared for running, i.e. its configuration is updated) the returned path will be equal to "getDesignDir()" (or "getDesignDir() + "/input" in the case of the getInputDir() command).
• If you supply a computation to any of those two commands, the returned path will be the correct directory for the computation in the CURRENT DESIGN.
• If you want to make sure that an exporting Feature is executed prior to a computation run in the correct directory, it needs to 1. have one of those two commands set AS AN ARGUMENT and 2. the configuration needs to access a value that is calculated inside the Feature

So, what does CAESES do with files in the context of Computations?

• If the "Remove existing Result Files" option is set at a computation, the RESULTSDIR will be cleared immediately before the external process is started. This is to avoid to read in old result files that do not belong to the current run of the computation
• For Local:
  • Only files that are specified in the "Output files" setting under the "Local Execution Settings" category of the computation are kept (comma separated list). All other files will be deleted. Wildcards are allowed. The default setting "*" means, that CAESES should keep all files.
  • The semantics of the "Output files" setting can be reversed by checking the checkbox on that setting. In that case all files specified there will be deleted after the run
• For Ssh:
  • The "Requested Output Files" setting is pretty much the same thing as the "Output files" setting for local execution. The only difference is that the files do not need to be deleted, but are not transferred from the remote computer in the first case.
• When input files for computations are generated (e.g. exports, command files) they are put into the "input" sub-folder of the computation's results directory. The reason for this is, that CAESES needs to be able to determine which files need to be transferred to remote hosts in the case of Ssh execution.
• For local execution the files from the "input" directory are MOVED to the computation's results directory just before the external process is started. So, after a computation run, the "input" directory *should* be empty (if no errors occurred).
• For Ssh execution all files from the "input" directory are sent to the SshResourceManager. When fetching the result files from the SshResourceManager those files will be put into the computation's results directory (including the directory structure they had on the remote computer). By default, the input directory will still include the input files after a run (it can be cleared after a run, though, by setting the "Clear Input Directory after Run" option under "Ssh Execution Settings".

There are some special cases when multiple SoftwareConnectors are "sharing" files (i.e. result files of computation A are input files for computation B ). When setting up such a chain (using the button I mentioned in the beginning of this post), the entry that is added to the input files of computation B has some additional options:

• "Fail if not exists" - If this is set, computation B will not run, if computation A did not produce the referenced file
• "Fail if file is empty" - If this is set, computation B will not run, if the referenced file does exist, but has size 0.
• "Move instead of copy" - If this is set, the referenced file will not be copied, but moved to the input directory of computation B. This - of course - has some implications, if either computation A tries to access the file or if the "Fail if not exists" option is set and you try to run computation B again, without running computation A first (if computation A did not go out of date. If it did, it would run either way, due to the dependency from B to A)
• "Create symbolic link" - If this is set, the file will neither be moved nor copied, but a symbolic link would be produced instead. On Linux, this is the same as typing "ln -s <to> <from>" in the console, on Windows a shortcut file (.lnk) is created that points to the original file. Note that the external program needs to be able to follow links.

I think, that pretty much covers it all. If you have additional questions or something is still unclear, just ask.

 

Arne

[Karsten Wenzke 12]

Hey Britt,

just as an addition to arne's post:

 

...

 

First of all: From your question, it seems like you are trying to "concatenate" multiple Software Connectors with each other? If that is the case, there is an inbuilt mechanism, that takes care of everything file related: In the SoftwareConnector for program B, push the little button that looks like the SoftwareConnector icon on the top right of the "Input Files" pane. You will get a list of all other SoftwareConnectors in the project with their result files. Select the file that you want and everything else will be taken care of automagically.

.....

 

For the coupling (linking) of two software connectors. Take a look in this post in the pdf "CAESES+SU2" on page 9. There it is explained how it can be done.

 

Hope this helps.

 

Cheers,

Karsten

[Britton Ward 0]

Thanks for taking the time to write that summary. It confirmed most of what I thought I understood.

 

The issue I was having is I have a geometry file for an external computation in a custom format being written as a text file so it took a little wrangling to get it seen as an input.

 

I was beginning to exercise my model and I'm seeing something else that is causing issues that you may be able to help me with.

 

I'm using the Design Assembler to create a single variant.

 

When I execute the Design Engine it creates the folders:

 

DesignAssembler_01

and DesignAssembler_01_des000

 

For some reason the model executes the first external computation and puts the results in the main DesignAssembler_01 directory.  It then exports the geometry file to the DesignAssembler_01/input directory but does not execute the second external computation.

 

It then creates the des000 directory. It runs the first external computation again successfully but the required feature geometry export does not happen and the second external computation fails.

 

All file calls use getResultsDir() or getInputDir()

 

Why is that toplevel DesignAssembler_01 directory populated at all? I would have thought it should only execute for each des_**** ?

 

Any thoughts?

Britt

[Britton Ward 0]

I think I fixed it. 

Strangely though when running the baseline design I have to manually execute the second external computation but when I run the design assembler it now run all the way through.!

 

Britt

[Britton Ward 0]

Hi,

 

I'm back puzzling over this issue of concatenating multiple software connectors.  When I run a design study the process runs successfully through the first two software connectors but the third [and therefore 4th] are never triggered.

 

The first simply takes a result file from the previous steps, along with some template files, executes and returns a file that is used by the 4th connector.

 

I've tried to follow the instructions above.

 

The 2nd Step is called "RAGOO" a custom tool that produces the output result file I need as a *.vin file. 

 

[This tool could generate a lot of these vin files ... is there anyway to grab all of the *.vin files for input in the next directory?]

 

The 3rd Step is a performance prediction program WinDESIGN [customized and executed via python script] in the software connector I used the Input file selector to select the file from the RAGOO output. I then edited it because the filename will change based on the design name as follows:

 

Ragoo.getResults().getResultFileName(4_RagooPrep|43_RagooInputParam|ModelID + "_01.vin")

 

The WINDESIGN step produces a csv file named: 4_RagooPrep|43_RagooInputParam|ModelID + ".csv"

 

The 4th Step is an excel spreadsheet called by a python script called EXCELIRCRMP this needs the csv output file from WINDESIGN to execute.

I use the Software connector selection to pick the output then edit it to incorporate the modename as follows:

 

WinDesign.getResults().getResultFileName(4_RagooPrep|43_RagooInputParam|ModelID + ".csv")

 

For each of the third and fourth steps I don't have a specific feature that refers to output from the previous steps. I have tried to add a string parameter based on output from the previous step to try and force it but with no luck.

 

Do I have to create some dummy feature to force the path manager to act once the 2nd step is available?  Any ideas on how to make a simple one?

 

 

Looking at the dependency tree from an output parameter read from the 4th computation it appears that all the dependencies are in order as expected.

 

Anxious to flex this new license on a range of variants but I am stuck until I can get it working reliably!

 

Britt

[Ben Zeitz 1]

Hello Britton,

 

To me it looks like one or more files of your tools were not found which interrupts the process chain.

 

... in the software connector I used the Input file selector to select the file from the RAGOO output. I then edited it because the filename will change based on the design name as follows:

 

Ragoo.getResults().getResultFileName(4_RagooPrep|43_RagooInputParam|ModelID + "_01.vin")

You should rather make the name of the RAGOO output file dynamic than the name of the input file of the next Software Connector in line.

How about creating a string parameter and set it as name of the RAGOO ouput file. The string parameter can get an expression set as value, so both the name of the RAGOO output file and the name of the input file for WinDESIGN change dynamically to what the string parameter evaluates to.

 

I have attached a very small and simple project which hopefully makes my suggestion more clearly.

 

 

[This tool could generate a lot of these vin files ... is there anyway to grab all of the *.vin files for input in the next directory?]

No sorry, this is not possible yet. One file corresponds to one item (input or output) of the Software Connector. There is no item for multiple files.

Not a bad idea though. Perhaps we provide such functionality in the future.

 

Best regards

Ben

processChain.zip