On-prem Task Runner (Server)

On-prem Task Runner (Server)

On-premises task runners consist of two parts:

  • A task runner registered as an On-premise Server through the Task Runner page in the integrate.ai workspace.

  • An on-premise service that is installed in a compatible environment, such as a virtual machine (VM) or bare metal machine.

You install the on-prem task runner agent through the integrate.ai command line tool (IAI CLI). In addition to the agent, a dedicated on-prem ECS cluster is created in the integrate.ai infrastructure to maintain the status of the on-prem agents, tasks, and logs.

PreRequisites for On-Prem task runner:

  • Any VM or compatible environment that allows root user access

  • Root user access - required to install the agent

Register an on-prem task runner

# Example configuration for notebooks:
container_path = "/data"
train_path1 = f"{container_path}/train_silo0.parquet"
test_path1 = f"{container_path}/test.parquet"

train_path2_aws = f"{container_path}/train_silo1.parquet"
test_path2_aws = f"{container_path}/test.parquet"

active_train_path = f"{container_path}/active_train.parquet"
active_test_path = f"{container_path}/active_test.parquet"
passive_train_path_aws = f"{container_path}/passive_train.parquet"
passive_test_path_aws = f"{container_path}/passive_test.parquet"

aws_storage_path = f'{container_path}}/model'

vfl_predict_active_storage_path = f'{container_path}/vfl_predict/active_predictions.csv'
vfl_predict_passive_storage_path_aws = f'{container_path}/vfl_predict/passive_predictions.csv'

# storage path for models
vfl_predict_passive_storage_path_aws = f'{container_path}/vfl_predict/passive_predictions.csv'

Task runners simplify the process of running training sessions on your data.

Step 1: To register an on-prem task runner:

  1. Log in to your integrate.ai workspace.
  2. In the left navigation bar, Click *Settings*.
  3. Under Workspace, click *Task Runners*.
  4. Click *Register* to start registering a new task runner.
  5. Select *Server* under the On-premises section.
  6. Follow the instructions provided.

    • Task runner name - must be unique

    • Storage path - enter the default storage path location on the VM. For example: /data. Note that this must be a path location and not simply a folder name.

  7. Click *Register*. Wait for the installation to complete.

Step 2: Install the task runner agent:

  1. Create a Python virtual env (venv) on the VM. Agent installation must be done as root user. Execute the command `sudo su` before installation. `python3 -m venv /home/{installation dir}`
  2. Install the IAI CLI tool: `pip install integrate-ai`

Step 3: Register the VM instance as an agent for the on-prem task runner created in Step 1:

  1. Register the on-prem node with the task runner using the following command: `iai onprem_node install`
  2. When prompted, provide your IAI_TOKEN.
  3. When prompted, provide the name of the task runner you created in Step 1.
  4. Wait for registration to complete.

Step 4: Create and run a session using the on-prem task runner:

  • Modify any dataset paths to match the storage location path from Step 1.

Removing an on-prem task runner

To remove a VM task runner agent instance, run the following command: iai onprem_node uninstall

Contents