Installation and Administration Getting Started Command Line Configuration Eclipse Plugin Reference Manual
Show on single page Show on multiple pages
Reference Manual > Data Providers > Creating your own Data Providers

Creating your own Data Providers

All Data Providers are utilities that run during an analysis. They usually take an input file to parse or parameters specified by the user to generate output files containing data other metrics to add to your project. Here is a non-exhaustive list of what some of them do:

This section describes two ways to add your own Data Providers to Squore:

  1. Using one of the generic, built-in Data Provider frameworks. Each solution uses a different approach, but the overall goal is to produce one or more CSV files that your Data Provider will send to Squore to associate metrics, findings, textual information or links to artefacts in your project.

  2. Writing your own utility to generate the XML files expected by Squore so they can be imported as part of the analysis result.

Tip

If you are interested in developping Data Providers that go beyond the scope of what is described in this manual, consult Squoring Technologies to learn more about the available training courses in writing Data Providers.

Choosing the Right Data Provider Framework

The following is a list of the available Data Provider frameworks:

Data Provider frameworks and their capabilities

  1. Csv

    The Csv framework is used to import metrics or textual information and attach them to artefacts of type Application or File. While parsing one or more input CSV files, if it finds the same metric for the same artefact several times, it will only use the last occurrence of the metric and ignore the previous ones. Note that the type of artefacts you can attach metrics to is limited to Application and File artefacts. If you are working with File artefacts, you can let the Data Provider create the artefacts by itself if they do not exist already. Refer to the full Csv Reference for more information.

  2. csv_findings

    The csv_findings framework is used to import findings in a project and attach them to artefacts of type Application, File or Function. It takes a single CSV file as input and is the only framework that allows you to import relaxed findings directly. Refer to the full csv_findings Reference for more information.

  3. CsvPerl

    The CsvPerl framework offers the same functionality as Csv, but instead of dealing with the raw input files directly, it allows you to run a perl script to modify them and produce a CSV file with the expected input format for the Csv framework. Refer to the full CsvPerl Reference for more information.

  4. FindingsPerl

    The FindingsPerl framework is used to import findings and attach them to existing artefacts. Optionally, if an artefact cannot be found in your project, the finding can be attached to the root node of the project instead. When launching a Data Provider based on the FindingsPerl framework, a perl script is run first. This perl script is used to generate a CSV file with the expected format which will then be parsed by the framework. Refer to the full FindingsPerl Reference for more information.

  5. Generic

    The Generic framework is the most flexible Data Provider framework, since it allows attaching metrics, findings, textual information and links to artefacts. If the artefacts do not exist in your project, they will be created automatically. It takes one or more CSV files as input (one per type of information you want to import) and works with any type of artefact. Refer to the full Generic Reference for more information.

  6. GenericPerl

    The GenericPerl framework is an extension of the Generic framework that starts by running a perl script in order to generate the metrics, findings, information and links files. It is useful if you have an input file whose format needs to be converted to match the one expected by the Generic framework, or if you need to retrieve and modify information exported from a web service on your network. Refer to the full GenericPerl Reference for more information.

  7. ExcelMetrics

    The ExcelMetrics framework is used to extract information from one or more Microsoft Excel files (.xls or .xslx). A detailed configuration file allows defining how the Excel document should be read and what information should be extracted. This framework allows importing metrics, findings and textual information to existing artefacts or artefacts that will be created by the Data Provider. Refer to the full ExcelMetrics Reference for more information.

Extending a Framework

After you choose the framework to extend, you should follow these steps to make your custom Data Provider known to Squore:

  1. Create a new configuration tools folder to save your work in your custom configuration folder: MyConfiguration/configuration/tools.

  2. Create a new folder for your data provider inside the new tools folder: CustomDP. This folder needs to contain the following files:

    • form.xml defines the input parameters for the Data Provider, and the base framework to use, as described in the section called “Data Provider Parameters”

    • form_en.properties contains the strings displayed in the web interface for this Data Provider, as described in the section called “Localising your Data Provider”

    • config.tcl contains the parameters for your custom Data Provider that are specific to the selected framework

    • CustomDP.pl is the perl script that is executed automatically if your custom Data Provider uses one of the *Perl frameworks.

  3. Edit Squore Server's configuration file to register your new configuration path, as described in the Installation and Administration Guide.

  4. Log into the web interface as a Squore administrator and reload the configuration.

Your new Data Provider is now known to Squore and can be triggered in analyses. Note that you may have to modify your Squore configuration to make your wizard aware of the new Data Provider and your model aware of the new metrics it provides. Refer to the relevant sections of the Configuration Guide for more information.

Creating a Freestyle Data Provider

Instead of using one of the Data Provider frameworks, you can directly produce your results in an XML format that Squore can read and import. The syntax of the XML file to generate is as follows:

<!-- input-data.xml: -->
<bundle version="2.0">
	<artifact [local-key=""] [local-parent=""|parent=""]>
        <artifact [id="<guid-stable-in-time-also-a-key>"] name="Component" type="REQ" [location=""]>
			<info name|n="DESCR" value="The description of the object"/>
			<key value="3452-e89b-ff82"/>
			<metric name="TEST_KO" value="2"/>
			<finding name="AR120" loc="xxx" p0="The message" />
			<link name="TEST" local-dst=""|dst="" />
				<artifact id="" name="SubComponent" type="REQ">
				...
				</artifact>
		</artifact>
	</artifact>

	<artifact id="" local-key="" name="" type="" local-parent=""|parent="" [location=""] />
	...

	<link name="" local-src=""|src="" local-dst=""|dst="" />
	...

	<info local-ref=""|ref="" name="" value=""/>
	...

	<metric local-ref=""|ref="" name="" value=""/>
	...

	<finding local-ref=""|ref="" [location=""] p0="" />
	<finding local-ref=""|ref="" [location=""] p0="">
		<location local-ref=""|ref="" [location=""] />
		...
		<relax status="RELAXED_DEROGATION|RELAXED_LEGACY|RELAXED_FALSE_POSITIVE"><![CDATA[My Comment]]></relax>
	</finding>
	...
</bundle>

Tip

You can find the XML schema for input-data.xml in input-data-2.xsd.

input-data.xml must be written in a specific location by an executable or a script declared in the Data Provider's form.xml in an exec-phase element, as described in the section called “Data Provider Parameters”.

Data Provider Parameters

A Data Provider's parameters are defined in a file called form.xml. The following is an example of form.xml for a Data Provider extending the GenericPerl framework:

CustomDP parameters
<?xml version="1.0" encoding="UTF-8"?>
<tags baseName="GenericPerl" needSources="true" image="CustomDP.png" projectStatusOnFailure="ERROR">
	<tag type="multipleChoice" displayType="checkbox" key="tests">
		<value key="ux" option="usability" />
		<value key="it" option="integration" />
		<value key="ut" option="unit" />
	</tag>
	<tag type="booleanChoice" key="ignore_missing_sources" defaultValue="false" />
	<tag type="text" key="input_file" defaultValue="myFile.xml" changeable="false" />
	<tag type="multipleChoice" key="old_results" style="margin-left:10px" displayType="radioButton" defaultValue="Exclude">
		<value key="Exclude" />
		<value key="Include" />
	</tag>
	<tag type="text" key="java_path" defaultValue="/usr/bin/java" hide="true" />
	<tag type="password" required="true" key="password" />
</tags>

The tags element accepts the following attributes:

  • baseName (mandatory) indicates which framework you are basing this Data Provider on

  • needSources (optional, default: false) allows specifying whether the Data Provider requires sources or not. When set to true, an error will be displayed if you try to select this Data Provider without adding any Repository Connector to your project.

  • image (optional, default: none) allows displaying a logo in the web UI for the Data Provider

  • projectStatusOnFailure (optional, default: ERROR) defines what status the project ends in when this Data Provider produces an error. The following values are allowed:

    • IGNORE

    • WARNING

    • ERROR

  • projectStatusOnWarning (optional, default: WARNING) defines what status the project ends in when this Data Provider produces a warning. The following values are allowed:

    • IGNORE

    • WARNING

    • ERROR

Each tag element is a Data Provider option and allows the following attributes:

  • key (mandatory) is the option's key that will be passed to the perl script, or can be used to specify the parameter's value from the command line

  • type (mandatory) defines the type of the parameter. The following values are accepted:

    • text for free text entry

    • password for password fields

    • booleanChoice for a boolean

    • multipleChoice for offering a selection of predefined values

      Note

      Predefined values are specified with a value element with a mandatory key attribute and an optional option attribute that allows modifying the value of the option from the UI.

  • displayType (optional) allows specifying how to display a multipleChoice parameter by using one of:

    • comboBox

    • radioButton

    • checkbox

  • defaultValue (optional, default: empty) is the value used for the parameter when not specified

  • hide (optional, default: false) allows hiding a parameter from the web UI, which is useful when combining it with a default value

  • changeable (optional, default: true) allows making a parameter configurable only when creating the project but read-only for following analyses when set to true

  • style (optional, default: empty) allows setting basic css for the attribute in the web UI

  • required (optional, default: false) allows showing a red asterisk next to the field in the web UI to make it visibly required. Note that this is only a visual aid at the moment and cannot be used to force users to enter a value for the parameter.

In order to create a freestyle Data Provider, your form.xml must contain an exec-phase element with a mandatory id="add-data" attribute:

<exec-phase id="add-data">
  <exec name="tcl|perl|java|javascript or nashorn" | executable="/path/to/bin" failOnError="true|false" failOnStdErr="true|false" warn="[WARN]" error="[ERROR|ERR]" fatal="[FATAL]"> 
    <arg value="${<function>(<args>)}"/>
    <arg value="-freeText" />
    <arg value="${<predefinedVars>}" />
    <arg value="versions" />
    <arg value="-myTag"/>
    <arg tag="myTag"/>
    <env key="MY_VAR" value="SOME_VALUE"/>
  </exec>
  <exec ... />
</exec-phase>

The exec-phase element accepts one or more launches of scripts or executables specified in an exec child element, that can receive arguments and environment variables specified via arg and env elements.

There are four built-in languages for executables:

  • tcl

  • perl

  • java

  • javascript or nashorn

The scripts are launched using the tcl, perl, or java runtimes defined in your Squore installation. This is also the case for javascript, which is handled by Java's Nashorn engine. Alternatively, you can call an executable directly by specifying its absolute path using the executable attribute.

The following attributes of the exec element allow you to control error handling:

  • failOnError (optional, default: true) marks the Data Provider execution as failed if the executable returns an error code

  • failOnStdErr (optional, default: true) marks the Data Provider execution as failed if the executable prints something to stdErr during the execution

  • warn , error and fatal (optional, default: see code block above) allow you to define patterns to look for in the executable's standard output to fine-tune the result of the execution.

Argument values can be:

  1. Free text, useful to specify a parameter for your script

  2. A tag key declared in form.xml to retrieve the input specified by the user

  3. One of the predefined functions

    • getToolConfigDir(<relative/path>) to get the absolute path of the Data Provider's configuration folder

    • getToolAddonsDir(<relative/path>) to get the absolute path of the Data Provider's addons folder

  4. One of the predefined variables

    • ${tmpDirectory} to get an absolute path to a temp folder to create files

    • ${sourcesList} to get a list of the aliases and locations containing the data extracted by the repository connectors used in the analysis

    • ${outputDirectory} to get the absolute path of folder where the Data Provider needs to write the final input-data.xml

Localising your Data Provider

In order to display your Data Provider parameters in different languages in the web UI, yout Data Provider's form.xml does not contain any hard-coded strings. Instead, Squore uses each parameter's key attribute to dynamically retrieve a translation from a form_xx.properties file located next to form.xml.

When you create a Data Provider, it is mandatory to include at least an English version of the strings in a file called form_en.properties. You are free to add other languages as needed. Here is a sample .properties for for the CustomDP you created in the previous section:

FORM.GENERAL.NAME  = CustomDP
FORM.DASHBOARD.NAME  = Test Status
FORM.GENERAL.DESCR = CustomDP imports test results for my project
FORM.GENERAL.URL   = http://example.com/CustomDP

TAG.tests.NAME  = Test Types
TAG.tests.DESCR = Check the boxes next to the types of test results contained in the results

TAG.ignore_missing_sources.NAME = Ignore Missing Sources

TAG.input_file.NAME = Test Results
TAG.input_file.DESCR = Specify the absolute path to the file containing the test results

TAG.old_results.NAME = Old Test Results
TAG.old_results.DESCR = If the previous analysis contained results that are not in this results file, what do you want to do with the old results?
OPT.Exclude.NAME = discard
OPT.Include.NAME = keep

TAG.password.NAME = File Password
TAG.password.DESCR = Specify the password to decrypt the test results file

The syntax for the .properties file is as follows:

  • FORM.GENERAL.NAME is the display name of the Data Provider in the project wizard

  • FORM.DASHBOARD.NAME is the display name of the Data Provider in the Explorer

  • FORM.GENERAL.DESCR is the description displayed in the Data Provider's tooltip in the web UI

  • FORM.GENERAL.URL is a reference URL for the Data Provider. Note that it is not displayed in ther web UI yet.

  • TAG.tag_name.NAME allows setting the display name of a parameter

  • TAG.tag_name.DESCR is a help text displayed in a tooltip next to the Data Provider option in the web UI

  • OPT.option_name.NAME allows setting the display name of an option

Using the form_en.properties above for CustomDP results in the following being displayed in the web UI when launching an analysis:

CustomDP pulling translations from a .properties file
Prev Up Next
Providing a catalog file to a Data Provider for Offline XSL Transformations Home | ToC Chapter 5. Cloning Detection
Squore 17.1.11
© 2019 Squoring Technologies All rights reserved. This material may not be reproduced, displayed, modified or distributed without the express prior written permission of the copyright holder. Squore is protected by an Interdeposit Certification registered with Agence pour la Protection des Programmes under the Inter Deposit Digital Number IDDN.FR.001.390035.000.S.P.2013.000.10600.