Failing to validate valid signature #61
Description
Description
I have the following model and signature: https://huggingface.co/tarilabs/model-joblib/tree/main
The signature is Valid, example:
% model_signing verify sigstore . --signature model.sig --identity "matteo.mortari@gmail.com" --identity_provider "https://accounts.google.com"
Key a687e5bf4fab82b0ee58d46e05c9535145a2c9afb458f43d42b45ca0fdce2a70 failed to verify targets
Verification succeeded
% docker run -it --rm -v $(pwd)/:/mnt/models:z -w /mnt/models ghcr.io/sigstore/model-transparency-cli:v1.1.1 verify sigstore --signature="/mnt/models/model.sig" /mnt/models --identity "matteo.mortari@gmail.com" --identity_provider "https://accounts.google.com"
Key a687e5bf4fab82b0ee58d46e05c9535145a2c9afb458f43d42b45ca0fdce2a70 failed to verify targets
Verification succeededUsing both Python and Docker from https://github.com/sigstore/model-transparency.
When I deploy the controller to a K8s cluster on KinD, it fails.
I am using the following KServe Isvc:
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: my-inference-service
labels:
validation.ml.sigstore.dev/ml: "demo"
spec:
predictor:
model:
modelFormat:
name: sklearn
storageUri: hf://tarilabs/model-jobliband Controller ModelValidation policy following blog post:
apiVersion: ml.sigstore.dev/v1alpha1
kind: ModelValidation
metadata:
name: demo
spec:
config:
sigstoreConfig:
certificateIdentity: "matteo.mortari@gmail.com"
certificateOidcIssuer: "https://accounts.google.com"
model:
path: /mnt/models
signaturePath: /mnt/models/model.sigWith the error:
Key 6f260089d5923daf20166ca657c543af618346ab971884a99962b01988bbe0c3 failed to verify root
Key 22f4caec6d8e6f9555af66b3d4c3cb06a3bb23fdc7e39c916c61f462e6f52b06 failed to verify root
Key 6f260089d5923daf20166ca657c543af618346ab971884a99962b01988bbe0c3 failed to verify root
Key a687e5bf4fab82b0ee58d46e05c9535145a2c9afb458f43d42b45ca0fdce2a70 failed to verify targets
Verification failed with error: not enough timestamps validated to meet the validation threshold (0/1)
For completeness, the KServe Isvc boils down to the following Pod definition:
apiVersion: v1
kind: Pod
metadata:
annotations:
internal.serving.kserve.io/storage-initializer-sourceuri: hf://tarilabs/model-joblib
prometheus.kserve.io/path: /metrics
prometheus.kserve.io/port: '8080'
serving.kserve.io/deploymentMode: RawDeployment
serving.kserve.io/enable-metric-aggregation: 'false'
serving.kserve.io/enable-prometheus-scraping: 'false'
creationTimestamp: '2025-10-23T14:58:31Z'
generateName: my-inference-service-predictor-786bdfdd47-
generation: 1
labels:
app: isvc.my-inference-service-predictor
component: predictor
pod-template-hash: 786bdfdd47
serving.kserve.io/inferenceservice: my-inference-service
validation.ml.sigstore.dev/ml: demo
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:internal.serving.kserve.io/storage-initializer-sourceuri: {}
f:prometheus.kserve.io/path: {}
f:prometheus.kserve.io/port: {}
f:serving.kserve.io/deploymentMode: {}
f:generateName: {}
f:labels:
.: {}
f:app: {}
f:component: {}
f:pod-template-hash: {}
f:serving.kserve.io/inferenceservice: {}
f:validation.ml.sigstore.dev/ml: {}
f:ownerReferences:
.: {}
k:{"uid":"da7dbfeb-5ec2-4369-b041-5d6ca5629c76"}: {}
f:spec:
f:containers:
k:{"name":"kserve-container"}:
.: {}
f:args: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:readinessProbe:
.: {}
f:failureThreshold: {}
f:periodSeconds: {}
f:successThreshold: {}
f:tcpSocket:
.: {}
f:port: {}
f:timeoutSeconds: {}
f:resources:
.: {}
f:limits:
.: {}
f:cpu: {}
f:memory: {}
f:requests:
.: {}
f:cpu: {}
f:memory: {}
f:securityContext:
.: {}
f:allowPrivilegeEscalation: {}
f:capabilities:
.: {}
f:drop: {}
f:privileged: {}
f:runAsNonRoot: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:dnsPolicy: {}
f:enableServiceLinks: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext: {}
f:terminationGracePeriodSeconds: {}
manager: kube-controller-manager
operation: Update
time: '2025-10-23T14:58:31Z'
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:conditions:
k:{"type":"ContainersReady"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"Initialized"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"PodReadyToStartContainers"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:status: {}
f:type: {}
k:{"type":"Ready"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
f:containerStatuses: {}
f:hostIP: {}
f:hostIPs: {}
f:initContainerStatuses: {}
f:podIP: {}
f:podIPs:
.: {}
k:{"ip":"10.244.0.32"}:
.: {}
f:ip: {}
f:startTime: {}
manager: kubelet
operation: Update
subresource: status
time: '2025-10-23T15:21:15Z'
name: my-inference-service-predictor-786bdfdd47-vtw44
namespace: default
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: my-inference-service-predictor-786bdfdd47
uid: da7dbfeb-5ec2-4369-b041-5d6ca5629c76
resourceVersion: '51502'
uid: ec9a096e-06c2-458b-8367-cf11d81aa961
spec:
containers:
- args:
- '--model_name=my-inference-service'
- '--model_dir=/mnt/models'
- '--http_port=8080'
image: kserve/sklearnserver:v0.15.0
imagePullPolicy: IfNotPresent
name: kserve-container
readinessProbe:
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 8080
timeoutSeconds: 1
resources:
limits:
cpu: '1'
memory: 2Gi
requests:
cpu: '1'
memory: 2Gi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
runAsNonRoot: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-49p89
readOnly: true
- mountPath: /mnt/models
name: kserve-provision-location
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
imagePullSecrets:
- name: ghcr-secret
initContainers:
- args:
- hf://tarilabs/model-joblib
- /mnt/models
image: kserve/storage-initializer:v0.15.0
imagePullPolicy: IfNotPresent
name: storage-initializer
resources:
limits:
cpu: '1'
memory: 1Gi
requests:
cpu: 100m
memory: 100Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
runAsNonRoot: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- mountPath: /mnt/models
name: kserve-provision-location
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-49p89
readOnly: true
- args:
- verify
- sigstore
- '--signature=/mnt/models/model.sig'
- '--identity'
- matteo.mortari@gmail.com
- '--identity_provider'
- https://accounts.google.com
- /mnt/models
command:
- /usr/local/bin/model_signing
image: ghcr.io/sigstore/model-transparency-cli:v1.0.1
imagePullPolicy: Always
name: model-validation
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-49p89
readOnly: true
- mountPath: /mnt/models
name: kserve-provision-location
readOnly: true
nodeName: kind-control-plane
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: kube-api-access-49p89
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
- emptyDir: {}
name: kserve-provision-location
status:
conditions:
- lastProbeTime: null
lastTransitionTime: '2025-10-23T14:58:32Z'
status: 'True'
type: PodReadyToStartContainers
- lastProbeTime: null
lastTransitionTime: '2025-10-23T14:58:31Z'
message: 'containers with incomplete status: [model-validation]'
reason: ContainersNotInitialized
status: 'False'
type: Initialized
- lastProbeTime: null
lastTransitionTime: '2025-10-23T14:58:31Z'
message: 'containers with unready status: [kserve-container]'
reason: ContainersNotReady
status: 'False'
type: Ready
- lastProbeTime: null
lastTransitionTime: '2025-10-23T14:58:31Z'
message: 'containers with unready status: [kserve-container]'
reason: ContainersNotReady
status: 'False'
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: '2025-10-23T14:58:31Z'
status: 'True'
type: PodScheduled
containerStatuses:
- image: kserve/sklearnserver:v0.15.0
imageID: ''
lastState: {}
name: kserve-container
ready: false
restartCount: 0
started: false
state:
waiting:
reason: PodInitializing
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-49p89
readOnly: true
recursiveReadOnly: Disabled
- mountPath: /mnt/models
name: kserve-provision-location
readOnly: true
recursiveReadOnly: Disabled
hostIP: 10.89.0.9
hostIPs:
- ip: 10.89.0.9
initContainerStatuses:
- allocatedResources:
cpu: 100m
memory: 100Mi
containerID: >-
containerd://a6161ac1056d9ad71bfc39fc6ff2796b22517b2a49318a60b34dd54dca37e405
image: docker.io/kserve/storage-initializer:v0.15.0
imageID: >-
docker.io/kserve/storage-initializer@sha256:72be1c414b11f45788106d6e002c18bdb4ca851048c4ae0621c9d57a17ccc501
lastState: {}
name: storage-initializer
ready: true
resources:
limits:
cpu: '1'
memory: 1Gi
requests:
cpu: 100m
memory: 100Mi
restartCount: 0
started: false
state:
terminated:
containerID: >-
containerd://a6161ac1056d9ad71bfc39fc6ff2796b22517b2a49318a60b34dd54dca37e405
exitCode: 0
finishedAt: '2025-10-23T14:58:57Z'
reason: Completed
startedAt: '2025-10-23T14:58:32Z'
user:
linux:
gid: 1000
supplementalGroups:
- 1000
uid: 1000
volumeMounts:
- mountPath: /mnt/models
name: kserve-provision-location
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-49p89
readOnly: true
recursiveReadOnly: Disabled
- containerID: >-
containerd://3112fa6573cc24fa8cf2ea596ce5978116360737b07d152bdb12bdd3c9374d7f
image: ghcr.io/sigstore/model-transparency-cli:v1.0.1
imageID: >-
ghcr.io/sigstore/model-transparency-cli@sha256:0cbc546ecd26bef359bdbd357a3277452dc186a322068a3ab7b3b33536e39f15
lastState:
terminated:
containerID: >-
containerd://3112fa6573cc24fa8cf2ea596ce5978116360737b07d152bdb12bdd3c9374d7f
exitCode: 1
finishedAt: '2025-10-23T15:21:02Z'
reason: Error
startedAt: '2025-10-23T15:20:58Z'
name: model-validation
ready: false
resources: {}
restartCount: 9
started: false
state:
waiting:
message: >-
back-off 5m0s restarting failed container=model-validation
pod=my-inference-service-predictor-786bdfdd47-vtw44_default(ec9a096e-06c2-458b-8367-cf11d81aa961)
reason: CrashLoopBackOff
user:
linux:
gid: 0
supplementalGroups:
- 0
uid: 0
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-49p89
readOnly: true
recursiveReadOnly: Disabled
- mountPath: /mnt/models
name: kserve-provision-location
readOnly: true
recursiveReadOnly: Disabled
phase: Pending
podIP: 10.244.0.32
podIPs:
- ip: 10.244.0.32
qosClass: Burstable
startTime: '2025-10-23T14:58:31Z'Given I can Verify the signature using the https://github.com/sigstore/model-transparency as reported above, this looks like a bug of sort in this Controller.
I'm happy to provide additional info if guided in order to best assess and identify the issue.
Version
- https://pypi.org/project/model-signing/1.1.1/
- ghcr.io/sigstore/model-transparency-cli:v1.1.1
- e016866 built docker image and pushed to the K8s KinD cluster with:
diff --git a/Makefile b/Makefile
index 46f203d..2fd8437 100644
--- a/Makefile
+++ b/Makefile
@@ -53,7 +53,7 @@ endif
# This is useful for CI or a project to utilize a specific version of the operator-sdk toolkit.
OPERATOR_SDK_VERSION ?= v1.41.1
# Image URL to use all building/pushing image targets
-IMG ?= controller:latest
+IMG ?= ghcr.io/sigstore/model-validation-operator:v0.0.1
# Get the currently used golang install path (in GOPATH/bin, unless GOBIN is set)
ifeq (,$(shell go env GOBIN))
@@ -182,7 +182,7 @@ generate-local-certs: ## Generate TLS certificates for local development
# More info: https://docs.docker.com/develop/develop-images/build_enhancements/
.PHONY: docker-build
docker-build: test ## Build docker image with the manager.
- $(CONTAINER_TOOL) build -t ${IMG} -f ${CONTAINER_FILE} .
+ $(CONTAINER_TOOL) build --load -t ${IMG} -f ${CONTAINER_FILE} .
.PHONY: docker-push
docker-push: ## Push docker image with the manager.make docker-build
kind load docker-image ghcr.io/sigstore/model-validation-operator:v0.0.1So that applying
kubectl apply -k config/overlays/productionsucceeded.