Machine learning

A registry that sits beside the data

A model registry answers one question well: which version is this, and what is it. DataLens records the model, each of its versions, and the machine-readable description of how to call it - in the same database as the datasets those versions were trained on.

What is recorded about a model

Task

What the model is for: classification, regression, clustering, forecasting, anomaly detection, recommendation, sentiment, named entity recognition, summarisation or embedding. A model can carry more than one.

Framework

scikit-learn, PyTorch, TensorFlow, XGBoost, LightGBM, ONNX, Spark ML, or proprietary. Recorded rather than guessed from a filename.

Licence

Held as a value on the model, because the licence question arrives late and at the worst moment if it was never asked at registration.

Source

Where the model came from - built here, brought in, or referenced from a public catalogue - so provenance is part of the record rather than folklore.

Versions

Each version is its own row. Approval, evaluation and every run attach to a version, never to the model as a whole.

Tags and metadata

Free-form labelling in its own table, so a tag is a row that can be searched rather than a delimited string in a column.

How a version is called

An integration binds one model version to one connection, with the contract that describes the call.

Connection

Credentials and endpoint for the provider, held by the platform like any other connection rather than pasted into a flow definition.

Execution provider

REST endpoint, Azure ML, SageMaker, Vertex AI, Databricks, Fabric, Snowflake, MLflow, a Hugging Face endpoint, a batch endpoint, an ONNX reference or a custom target.

Input contract

The fields the model expects, each with its own row, so a mapping can be checked before a call is made rather than after it fails.

Output contract

The fields the model returns, described the same way, so what comes back can be landed as columns instead of parsed hopefully.

Input and output mapping

Which dataset column feeds which model field, and which returned field lands where. Junction tables in both directions.

Invocation mode

Synchronous or asynchronous. An asynchronous run that has left the platform carries the external job reference it was given.

Why the contracts are rows and not a blob

The obvious shortcut is to store the input shape as a JSON document on the integration. It is quicker to write and it is unqueryable: you cannot then ask which integrations send a column that has just been reclassified as personal data, which is the question that actually gets asked.

Each contract field being a row means the egress rules can attach to fields, the mapping can be validated field by field, and a sensitivity change anywhere upstream can be traced to every call that would carry it.

Questions about the registry

Does the registry store the model weights?

No. It records what the model is, which version is which, and how to reach the version where it is served. The artefact stays with the provider that serves it.

Can two versions of the same model be live at once?

Yes, and that is the normal case: approval is per version per environment, so a newer version can be approved in staging while an older one carries production.

What happens to a version nobody approved?

It stays in the registry and cannot be run. Registration and permission to run are deliberately two different acts.

Where this is today

The machine learning layer is in build. What is described on this page is the data and governance model that is implemented in the platform: the tables that hold it, the rules that enforce it, and the refusals that happen when a rule is not satisfied.

The Data Science tab is in the application. It reads: the model registry and its versions, the approval held for each version in each environment, the integrations and whether each one may currently be called, the record of every run including the refused ones, the projects, and the egress rules. It does not invoke a model - running belongs to the execution provider, and the record of a run is audit evidence that a browser must not be able to write.

Talk to us about the ML layer

DataLens is in private beta. If model governance is the reason you are here, say so when you request access - the ML layer is being built against real use rather than a roadmap.

Request beta access