fix: pin embedding inference to one torch thread
torch's multi-threaded CPU forward pass intermittently raised SIGILL under CPU contention on the deployment host; the collector swallowed it as non-fatal, so that cycle silently computed no new embeddings. Pinning inference to a single thread avoids the racy parallel kernel and removes multi-threaded reduction nondeterminism, so recomputed vectors are bit-reproducible. Bump 0.8.2 -> 0.8.3: a vector recomputed after this change can differ from a pre-0.8.3 one at ~1e-6 and move a borderline article across the cluster threshold, which may shift a rating.
This commit is contained in:
@@ -8,6 +8,21 @@ requires a version bump and, if it changes methodology, a decision record in
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.8.3] - 2026-07-19
|
||||
|
||||
### Fixed
|
||||
|
||||
- Embedding no longer crashes under CPU contention. torch's multi-threaded
|
||||
forward-pass kernel intermittently raised `Illegal instruction` (SIGILL) on
|
||||
the deployment host when the collector's 6h cycle overlapped other load,
|
||||
which the loop swallowed as non-fatal — so that cycle silently computed no
|
||||
new embeddings. Inference is now pinned to one thread
|
||||
(`torch.set_num_threads(1)`), which also removes multi-threaded reduction
|
||||
nondeterminism, so newly computed vectors are bit-reproducible (D1/D10).
|
||||
Cached vectors are untouched; a vector recomputed after this change may
|
||||
differ from a pre-0.8.3 one at ~1e-6, which can move a borderline article
|
||||
across the cluster threshold and thus shift a rating — hence the version bump.
|
||||
|
||||
## [0.8.2] - 2026-07-10
|
||||
|
||||
Audit pass 3 (zero-confirmation over the pass-2 diff): 2 findings — one a
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "tiltmeter"
|
||||
version = "0.8.2"
|
||||
version = "0.8.3"
|
||||
description = "Auditable, reproducible political-lean ratings for news outlets"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.13"
|
||||
|
||||
@@ -40,8 +40,14 @@ def passage(title: str, text: str | None, summary: str | None) -> str:
|
||||
def _load_model():
|
||||
global _model
|
||||
if _model is None:
|
||||
import torch
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
# Pin CPU inference to one thread. torch's multi-threaded forward-pass
|
||||
# kernel intermittently raised SIGILL under CPU contention on the
|
||||
# deployment host, and multi-threaded reduction order is nondeterministic
|
||||
# regardless; one thread is both crash-free and bit-reproducible (D1/D10).
|
||||
torch.set_num_threads(1)
|
||||
_model = SentenceTransformer(MODEL_NAME, revision=MODEL_REVISION, device="cpu")
|
||||
return _model
|
||||
|
||||
|
||||
Reference in New Issue
Block a user