Logging Model Metrics
Overview
Grace provides the ability to ingest, track, and logg model metrics directly under a model registry item. Metrics are tied to a registered model and support traceability across performance, risk, and monitoring dimensions.
User Guide
1. Accessing Model Metrics
- Go to the Registry in Grace.
- Open an existing Model (created via UI or API).
- At the top of the model page, you will see:
- Details
- Assessments
- Metrics (model-specific)
2. Available Metric Types
The available metrics depend on the model type selected during registration and can be custom-configured.
Typical categories include:
- Performance metrics (e.g., F1 score, ROC AUC)
- Explainability
- Data drift
- Bias
Grace also shows:
- Statistics about the latest ingested metrics
- Historical metric values over time
3. Ingesting Metrics via the API
Metric ingestion is performed programmatically using the Grace API.
Required Information
To submit metrics, you need:
- Project name
- Authentication bearer token (from the project)
- Model ID
- Metric variable name
- Metric value
- Timestamp
4. Finding the Model ID
- Use the Models API endpoint.
- List available models.
- Retrieve the ID of the target model.
5. Metric Submission Script
Metrics can be sent from:
- A script running inside Grace
- Any external environment
Example Logic:
- Metrics are created as structured data objects.
- Values can be:
- Hardcoded (for testing)
- Generated dynamically from training or evaluation pipelines
Example metric attributes:
- Variable name – must match the metric definition in Grace
- Value – the actual metric result
- Date – must follow the required timestamp format
The list of valid metric variable names can be retrieved:
- Via the UI
- Via the Metrics API
6. Sending Metrics
Once configured with:
- Model ID
- API endpoint
- API key
The script submits metric values to Grace. After submission:
- Refreshing the model’s Metrics tab shows the newly ingested values
- Metrics are displayed chronologically
7. Example Outcome
- Existing metrics may already exist for earlier months
- Sending a new metric (e.g., F1 score for May) immediately updates the timeline
- The newly ingested value appears in the model’s metrics view
8. Summary
- Centralized tracking of model performance
- Full historical traceability of metric evolution
- Programmatic integration with training and monitoring pipelines
- Support for governance, auditing, and compliance requirements