Publikation
United We Pretrain, Divided We Fail! Representation Learning for Time Series by Pretraining on 75 Datasets at Once
Maurice Kraus; Felix Divo; David Steinmann; Devendra Singh Dhami; Kristian Kersting
In: Computing Research Repository eprint Journal (CoRR), Vol. abs/2402.15404, Pages 1-14, arXiv, 2024.
Zusammenfassung
In natural language processing and vision, pre-
training is utilized to learn effective representa-
tions. Unfortunately, the success of pretraining
does not easily carry over to time series due to
potential mismatch between sources and target.
Actually, common belief is that multi-dataset pre-
training does not work for time series! Au con-
traire, we introduce a new self-supervised con-
trastive pretraining approach to learn one encod-
ing from many unlabeled and diverse time se-
ries datasets, so that the single learned represen-
tation can then be reused in several target do-
mains for, say, classification. Specifically, we pro-
pose the XD-MixUp interpolation method and the
Soft Interpolation Contextual Contrasting (SICC)
loss. Empirically, this outperforms both super-
vised training and other self-supervised pretrain-
ing methods when finetuning on low-data regimes.
This disproves the common belief: We can actu-
ally learn from multiple time series datasets, even
from 75 at once.