Running Pipelines in Grace

Overview

Pipelines in Grace allow users to automate processes such as model training or test runs. Pipelines are Kubernetes-native, meaning they run different containers in a defined sequence. Each container executes a script (e.g., a Python script), and steps can be run sequentially or in parallel with dependencies.

This guide explains how to create, configure, and run pipelines in Grace.

User Guide

Step 1: Access Pipelines

1. Navigate to the MLOps section in the left-hand side ribbon.
2. Select Pipelines.

Step 2: Create a New Pipeline

1. Click Add New.
2. A wizard will open to guide you through setup.
3. Initial Setup
   • Application Folder: Select the folder containing your script(s).
   • Pipeline Name: Provide a name (e.g., test_123).
   • Scheduling Options:
     • Manual Run Only – Execute pipeline on demand.
     • Scheduled Run – Define a regular date/time for automatic execution.
   • Token (Optional): Add a token for user identification if required.

Step 3: Define Pipeline Steps

1. Choose which Python scripts to execute and in what order.
2. Configure each step:
   • Container Name: e.g., test_run.
   • Script: e.g., test_run.py.
   • Dependencies: Define requirements.txt or other dependencies for installation.
   • Arguments: Add any additional parameters if needed.
   • Resources:
     • CPU vs GPU execution
     • Number of CPUs
     • Memory allocation
3. Click Create Step to add the first container.
4. (Optional) Add additional steps:
   • Each step can run sequentially or in parallel.
   • Dependencies can be configured between steps (e.g., Step 2 depends on Step 1).
   • This allows for building complex pipelines with multiple stages.

Step 4: Save and Run

1. Save the pipeline once all steps are defined.
2. Depending on setup:
   • Manual Run: Click Run to start execution immediately.
   • Scheduled Run: Pipeline will run at defined intervals.

Step 5: Monitor Execution

1. During execution, you can monitor progress in real time:
2. Pipeline Status: View status such as Running, Succeeded, or Failed.
3. Timestamps: Track execution times.
4. Step Logs:
   • See installation of libraries, function execution, and outputs.
   • Debug errors directly in the UI.
5. When all steps complete, the pipeline status changes to Succeeded.

Step 6: Review History

1. All executions of a pipeline are logged in the history view.
2. Running the pipeline multiple times over days/weeks generates a complete execution history for tracking and auditing.

Summary

Step	Description
Access Pipelines	Go to MLOps → Pipelines
Create New Pipeline	Select application folder, name, scheduling options, and token
Define Steps	Assign scripts, dependencies, resources, and execution order
Run Pipeline	Start manually or schedule for automatic execution
Monitor Execution	Track status, logs, and errors in real time
Review History	View all past pipeline runs for auditing and traceability

By using Pipelines in Grace, you can automate script execution, scale workloads across containers, and maintain a clear execution history for your AI and data science processes.