3.8 KiB
| stage | group | info |
|---|---|---|
| Create | Incubation | Machine Learning Experiment Tracking is a GitLab Incubation Engineering program. No technical writer assigned to this group. |
Machine Learning Experiment Tracking (FREE)
DISCLAIMER: Machine Learning Experiment Tracking is an experimental feature being developed by the Incubation Engineering Department, and will receive significant changes over time. This feature is being release with the aim of getting user feedback, but is not stable and can lead to performance degradation. See below on how to disable this feature.
When creating machine learning models, data scientists often experiment with different parameters, configurations, feature engineering, and so on, to improve the performance of the model. Keeping track of all this metadata and the associated artifacts so that the data scientist can later replicate the experiment is not trivial. Machine learning experiment tracking enables them to log parameters, metrics, and artifacts directly into GitLab, giving easy access later on.
What is an experiment?
An experiment is a collection of comparable model candidates. Experiments can be long lived (for example, when they represent a use case), or short lived (results from hyperparameter tuning triggered by a merge request), but usually hold model candidates that have a similar set of parameters and metrics.
Model candidate
A model candidate is a variation of the training of a machine learning model, that can be eventually promoted to a version of the model. The goal of a data scientist is to find the model candidate whose parameter values lead to the best model performance, as indicated by the given metrics.
Example parameters:
- Algorithm (linear regression, decision tree, and so on).
- Hyperparameters for the algorithm (learning rate, tree depth, number of epochs).
- Features included.
Usage
User access management
An experiment is always associated to a project. Only users with access to the project an experiment is associated with can view that experiment data.
Tracking new experiments and trials
Experiment and trials can only be tracked through the MLFlow client integration. More information on how to use GitLab as a backend for MLFlow Client can be found at the documentation page.
Exploring model candidates
To list the current active experiments, navigate to https/-/ml/experiments. To display all trials
that have been logged, along with their metrics and parameters, selecting an experiment.
Logging artifacts
Trial artifacts are saved as generic packages, and follow all their
conventions. After an artifact is logged for a candidate, all artifacts logged for the candidate are listed in the
package registry. The package name for a candidate is ml_candidate_<candidate_id>, with version -.
Limitations and future
- Searching experiments, searching trials, visual comparison of trials, and creating, deleting and updating experiments and trials through GitLab UI is under development.
- No support for experiment and trial metadata that do not classify as parameters or metrics.
Disabling or enabling the Feature
On self-managed GitLab, ML Experiment Tracking is disabled by default. To enable the feature, ask an administrator to disable the feature flag named ml_experiment_tracking.
On GitLab.com, this feature is currently on private testing.
Feedback, roadmap and reports
For updates on the development, feedback and bug reports, refer to the development epic.