Skip to content

How to use pipeline config files when using shrike.pipeline

Structure of config files

In this page, we perform a detailed review of an example standard config file. This should give you a good idea of how to use config files properly based on your scenarios.

For a pipeline, we set up 4 config files under the config directory, which includes 4 sub-folders: experiments, modules, aml, and compute. The demograph_eyeson.yaml file linked above lives in the experiments folder; it is the main config file which specifies the overall pipeline configuration. This main config file refers to three other config files under the config directory:

  1. a config file under the aml folder which lets you point at your Azure ML workspace by specifying subscription_id, resource_group, workspace_name, tenant and auth;
  2. a config file under the compute folder which specifies configurations such as the compliant data store name, compute targets names, data I/O methods, etc;
  3. a config file under the modules folder, which lists all the available components with their properties (key, name, default version, and location of the component specification file).

Now we will go through the config file linked above and explain each section.

1. Brief summary section

At the beginning of the config file, it is suggested to provide a brief comment explaining which pipeline this config file is used for, and also provide an example command to run the pipeline with this config file. See below for an example:

# This YAML file configures the accelerator tutorial pipeline for eyes-on

# command for running the pipeline:
# python pipelines/experiments/demograph_eyeson.py --config-dir pipelines/config --config-name experiments/demograph_eyeson run.submit=True

2. defaults section

The defaults section contains references of the aml resources, pointing to two other config files under the aml and compute folders. It also points to the file listing all available components, which is located in the modules folder. This section looks like below.

defaults:
  - aml: eyeson # default aml references
  - compute: eyeson # default compute target names
  - modules: module_defaults # list of modules + versions

See below for the contents of the aml config file. You will need to update the info based on your own aml resources. To find your workspace name, subscription Id, and resource group, go to your Azure ML workspace, then click the "change subscription" icon in the top right (between the settings and question mark), then "Download config file". You will find the 3 values in this file. The Torus TenantId for eyes-off workspaces is cdc5aeea-15c5-4db6-b079-fcadd2505dc2, whereas the 72f988bf-86f1-41af-91ab-2d7cd011db47 used below is the Microsoft TenantId that you will use for personal workspaces.

# @package _group_
subscription_id: 48bbc269-ce89-4f6f-9a12-c6f91fcb772d
resource_group: aml1p-rg
workspace_name: aml1p-ml-wus2
tenant: 72f988bf-86f1-41af-91ab-2d7cd011db47
auth: "interactive"

See below for the contents of the compute config file (update the info based on your own aml resources).

# @package _group_
# name of default target
default_compute_target: "cpu-cluster"
# where intermediary output is written
compliant_datastore: "workspaceblobstore"

# Linux targets
linux_cpu_dc_target: "cpu-cluster"
linux_cpu_prod_target: "cpu-cluster"
linux_gpu_dc_target: "gpu-cluster"
linux_gpu_prod_target: "gpu-cluster"

# data I/O for linux modules
linux_input_mode: "download"
linux_output_mode: "upload"

# Windows targets
windows_cpu_prod_target: "cpu-cluster"

# data I/O for windows modules
windows_input_mode: "download"
windows_output_mode: "upload"

# hdi cluster
hdi_prod_target: "hdi-cluster"

# data transfer cluster
datatransfer_target: "data-factory"

3. run section

In this section, you configure the parameters controlling how to run your experiment. Update the info based on your own pipeline. Parameter names should be self-explanatory.

# run parameters are command line arguments for running your experiment
run: # params for running pipeline
  experiment_name: "demo_graph_eyeson" # IMPORTANT
  experiment_description: "Demo for eyeson graph"
  regenerate_outputs: false
  continue_on_failure: false
  verbose: false
  submit: false
  resume: false
  canary: false
  silent: false
  wait: false

4. module_loader section

This section includes 5 arguments: use_local, force_default_module_version, force_all_module_version, local_steps_folder, and use_remote_when_component_not_in_manifest.

  • The use_local parameter specifies which components of the pipeline you would like to build from your local code (rather than consuming the remote registered component). Use a comma-separated string to specify the list of components from your local code. If you use "*", all components are loaded from local code. For more information, please check out Use component key to run this component locally.
  • The force_default_module_version argument enables you to change the default version of the component in your branch (the default version is the latest version, but this argument allows you to pin it to a given release version if you prefer).
  • The force_all_module_version argument enables you to force all components to consume a fixed version, even if the version is specified otherwise in the pipeline code.
  • The argument local_steps_folder should be clear and self-explanatory: this is the directory where all the component folders are located.
  • The use_remote_when_component_not_in_manifest argument is a boolean that controls shrike's behavior when a component cannot be found in the manifest. If false or unspecified, shrike will throw an exception when attempting to build the pipeline. If true, shrike will attempt to load the component from the workspace. 
# module_loader
module_loader: # module loading params
  # IMPORTANT: if you want to modify a given module, add its key here
  # see the code for identifying the module key
  # use comma separation in this string to use multiple local modules
  use_local: "DemoComponent"

  # fix the version of modules in all subgraphs (if left unspecified)
  # NOTE: use the latest release version to "fix" your branch to a given release
  # see https://eemo.visualstudio.com/TEE/_release?_a=releases&view=mine&definitionId=76
  force_default_module_version: null

  # forces ALL module versions to this unique value (even if specified otherwise in code)
  force_all_module_version: null

  # path to the steps folder, don't modify this one
  # NOTE: we're working on deprecating this one
  local_steps_folder: "../../../components" # NOTE: run scripts from accelerator-repo

  # flag to control the behavior when shrike cannot find a component in the manifest
  # 'true' for loading the component from the workspace, 'false' or unspecified for throwing an exception
  use_remote_when_component_not_in_manifest: true

5. Other sections

The sections above only defined overall pipeline parameters, not component parameters. We recommend gathering the component parameters into distinct sections, one per component. The example for the eyes-on demo experiment is shown below.

# DemoComponent config
democomponent:
  input_data: irisdata # the data we'll be working on
  input_data_version: "latest" # use this to pin a specific version
  message: "Hello, World!"
  value: 1000 # the size of the sample to analyze