Contrastive Learning Approaches for Deep Learning

Abstract Deep learning technology has made significant progress in fault diagnosis. However, real-world industrial settings, most existing methods require substantial labeled data for training, while harsh operating conditions and collection constraints often result scarce samples. This limitation significantly impairs their diagnostic performance practical applications. To address this challenge, we propose a few-shot diagnosis approach based on time-frequency contrastive (TF-CL) framework. The TF-CL framework adopts pre-training downstream task pipeline, enabling the model to automatically learn extract multi-perspective features from unlabeled self-supervised conditions. During pre-training, dedicated encoders separately time-domain frequency-domain feature representations abundant extracted are then projected into shared space using projector. ensure that can be data, paper introduces consistency loss function, constructed novel positive negative sample pairs. In task, is combined with multilayer perceptron classifier optimized fine-tuned end-to-end limited data. Gradient updates... Read More

Source

Abstract: Self-supervised learning (SSL) offers a transformative path for addressing domain adaptation in medical imaging, where annotated datasets are often limited and expensive to acquire. This paper explores how various SSL approachescontrastive (SimCLR), masked image modeling (MAE), transformer-based (DINO)improve performance segmentation classification across heterogeneous imaging domains (MRI, X-ray, CT). Using such as BraTS, CheXpert, NIH ChestXray14, we evaluate pretraining followed by fine-tuning with minimal supervision. We demonstrate statistically significant improvements (615%) Dice scores AUC. Regression analysis shows strong correlation between representation similarity (CKA) downstream task performance. Explainability tools SHAP LIME used validate model reliability transparency. Keywords: Self-Supervised Learning, Domain Adaptation, Medical Imaging, Contrastive SimCLR, DINO, Swin UNet, SHAP, LIME, Transfer Learning

Source

Specific emitter identification (SEI), as an emerging physical-layer security authentication method, is crucial for maintaining information in the Internet of Things. However, existing deep learning-based SEI methods require extensive labeled data training, which are often unavailable untrusted scenarios. Furthermore, due to subtle nature radio-frequency fingerprints, unsupervised struggles achieve high accuracy without guidance labels. In this paper, we propose method based on group label-driven contrastive learning (GLD-CL). We a novel constructing dataset: all input samples derived from same received signal segment grouped together and assigned unique identifier, termed label. Based this, improve loss function self-supervised learning. With assistance labels, feature vectors class space become more closely clustered, enhancing SEI. Extensive experimental results real-world datasets demonstrate that normalized mutual GLD-CL achieves 96.4% accuracy, representing improvement 5.68% or compared baseline algorithms. exhibits robust performance, achieving good across various signal-to-no... Read More

Source

Training neural networks to generate high quality feature representations by optimizing a spectral inference objective using a bilevel optimization technique. This involves maintaining moving averages of covariance measures and the Jacobian of the covariance during training. It also involves computing kernel-weighted mini-batch covariance estimates and using them to generate gradient estimates for updating the network parameters.

Source

Training a medical image reconstruction network using prior knowledge from real images to improve stability and convergence compared to generative adversarial networks. The training involves extracting feature vectors from real images, reconstructing images from those vectors, reconstructing from original hidden vectors, and discriminating between real and reconstructed images. The reconstruction network is optimized based on the discriminations. This introduces prior knowledge guidance to stabilize training and converge easier than pure generative adversarial networks.

Source

Traffic sign classification (TSC) based on deep neural networks (DNNs) plays a crucial role in the perception subsystem of autonomous driving systems (ADSs). However, studies reveal that TSC system can make dangerous and potentially fatal errors under adversarial attacks. Existing defense strategies, such as training (AT), have demonstrated effectiveness but struggle to generalize across diverse attack scenarios. Recent advancements self-supervised learning (SSL), particularly contrastive (ACL) methods, strong potential enhancing robustness generalization compared AT. conventional ACL methods lack mechanisms ensure effective transferability different stages. To address this, we propose robust pre-trainingfine-tuning algorithm for meta-adversarial (RPF-MAD), designed enhance sustainability throughout pipeline. Dual-track pre-training (Dual-MAP) integrates meta-learning with which improves ability upstream model conditions. Meanwhile, adaptive variance anchoring fine-tuning (AVA-RFT) utilizes prototype regularization stabilize feature representations reinforce generalizable capabili... Read More

Source

System for generating and organizing driving scenarios for autonomous vehicles to improve safety, efficiency, and reliability. The system uses neural networks to encode and decode multimodal sensor data like video, audio, and text prompts. It aligns sequences of sensor data with prompts using contrastive learning. This allows finding specific sensor sequences that match a given prompt. The system then generates a hierarchical structure representing the matching sensor sequence. By encoding, embedding, and aligning multimodal data, it creates a shared analytical space to discover cross-modal correlations and analyze latent dependencies. This improves understanding of navigation scenarios by capturing context and nuances like temporal evolution. The system also organizes scenarios for searchability and retrieval.

Source

Refining anomalous driving detection to better distinguish between safe driving variabilities and actual unsafe driving behaviors by learning and filtering driving variabilities based on environmental context. The method involves identifying repetitive and contrasting driving behaviors in sets of environmental conditions, then comparing and filtering out variabilities between sets to refine anomalous detection results. This allows removing false positives for variabilities that are normal in certain contexts.

Source

This research introduces a novel method that integrates both unsupervised and supervised learning, leveraging SimCLR (Simple Framework for Contrastive Learning of Visual Representations) self-supervised learning along with different pre-trained models to improve microscopic image classification Babesia parasite in canines. We focused on three popular CNN architectures, namely ResNet, EfficientNet, DenseNet, evaluated the impact pre-training their performance. A detailed comparison variants Densenet terms accuracy training efficiency is presented. Base such as DenseNet were utilized within framework. Firstly, unlabeled images, followed by classifiers labeled datasets. approach significantly improved robustness accuracy, demonstrating potential benefits combining contrastive conventional techniques. The highest 97.07% was achieved Efficientnet_b2. Thus, detection or other hemoparasites blood smear images could be automated high without using labelled dataset.

Source

Cross-modal retrieval tasks, encompassing the of image-text, video-audio, and more, are progressively gaining significance in response to exponential growth information on Internet. However, there has always been a cloud hanging over multimodal tasks due inherent challenges aligning different modalities with distinct physical meanings. Most previous works simply rely single encoder or novel similarity calculation for fusion, which often result unsatisfactory performance. To tackle this challenge, we introduce Momentum Hypergraph Convolutional Network (MoHGCN) representation learning, strengthens alignment both visual textual data before process. Specifically, MoHGCN utilizes contrastive learning select most challenging negative positive samples form hyperedges, completes modality through two rounds fusion. Subsequently, fully integrated node features global fused using fusion obtain final vector image-text retrieval. Extensive experiments conducted widely-used datasets, namely Flickr30K MSCOCO, demonstrate superiority proposed approach achieving state-of-the-art performances.

Source

<title>Abstract</title> Background It is possible to control the quality of capsule endoscopic images using artificial intelligence (AI), but it requires a great deal time for labeling. Methods SimCLR (a simple framework contrastive learning visual representations), capable acquiring inherent image representation with minimal annotation, feasibility not studied. 62850 were collected train models. In internal cross-validation (more training data and less testing data) reversed (less more data). Random forest Xgboost (eXtreme Gradient Boosting) used finish controlling after extracting features from images. Results reported that mean AUROC (Area Under Receiver Operating Characteristic) curve exceeded 0.98 0.97. Moreover, surpassed supervised CNN (Convolutional Neural Network). Extra 18636 pictures gathered 0.93 (95% CI 0.92710.9548), which close Network) (0.9645) in cross validation. surpass 0.96, better than (0.8374) Conclusions Through SimCLR, task can be completed performance similar or fewer annotations.

Source

Objectives Oral cavity-derived cancer pathological images (OPI) are crucial for diagnosing oral squamous cell carcinoma (OSCC), but existing deep learning methods OPI segmentation rely heavily on large, accurately labeled datasets, which labor- and resource-intensive to obtain. This paper presents a semi-supervised method mitigate the limitations of scarce data by leveraging both unlabeled data. Materials We use Hematoxylin Eosin (H&amp;amp;E)-stained dataset (OCDC), consists 451 with tumor regions annotated verified pathologists. Our combines transformation uncertainty multi-scale contrastive learning. The estimation evaluates models confidence transformed via different methods, reducing discrepancies between teacher student models. Multi-scale enhances class similarity separability while teacher-student model similarity, encouraging diverse feature representations. Additionally, boundary-aware enhanced U-Net is proposed capture boundary information improve accuracy. Results Experimental results OCDC demonstrate that our outperforms fully supervised approaches, achieving superior... Read More

Source

Training large language models to generate personalized and context-specific responses using contrastive in-context learning. The technique involves feeding both positive and negative examples to the model during training. The positive examples are desired responses based on user preferences, while the negative examples are undesired responses. The model learns to generate preferred responses and avoid the non-preferred ones. After training, the model can generate customized answers for new questions based on the learned user preferences.

Source

Training a contrastive learning model for query classification in a hierarchical category tree. The method involves optimizing the loss function using semantic relationships between categories. This is based on the relative positions of categories in the tree. The optimized loss function is used to train the contrastive learning model. It allows the model to predict categories accurately by leveraging the hierarchical category tree structure. By optimizing the loss function based on semantic relationships, the model learns to distinguish differences between categories based on their positions in the tree. This improves query classification accuracy compared to traditional flat category methods.

Source

Convergence in learning-to-optimize is hardly studied, because conventional convergence guarantees in optimization are based on geometric arguments, which cannot be applied easily to learned algorithms. Thus, we develop a probabilistic framework that resembles deterministic optimization and allows for transferring geometric arguments into learning-to-optimize. Our main theorem is a generalization result for parametric classes of potentially non-smooth, non-convex loss functions and establishes the convergence of learned optimization algorithms to stationary points with high probability. This can be seen as a statistical counterpart to the use of geometric safeguards to ensure convergence. To the best of our knowledge, we are the first to prove convergence of optimization algorithms in such a probabilistic framework.

Source

We use the PAC-Bayesian theory for the setting of learning-to-optimize. To the best of our knowledge, we present the first framework to learn optimization algorithms with provable generalization guarantees (PAC-Bayesian bounds) and explicit trade-off between convergence guarantees and convergence speed, which contrasts with the typical worst-case analysis. Our learned optimization algorithms provably outperform related ones derived from a (deterministic) worst-case analysis. The results rely on PAC-Bayesian bounds for general, possibly unbounded loss-functions based on exponential families. Then, we reformulate the learning procedure into a one-dimensional minimization problem and study the possibility to find a global minimum. Furthermore, we provide a concrete algorithmic realization of the framework and new methodologies for learning-to-optimize, and we conduct four practically relevant experiments to support our theory. With this, we showcase that the provided learning framework yields optimization algorithms that provably outperform the state-of-the-art by orders of magnitude.

Source

PAC-Bayes learning is a comprehensive setting for (i) studying the generalisation ability of learning algorithms and (ii) deriving new learning algorithms by optimising a generalisation bound. However, optimising generalisation bounds might not always be viable for tractable or computational reasons, or both. For example, iteratively querying the empirical risk might prove computationally expensive. In response, we introduce a novel principled strategy for building an iterative learning algorithm via the optimisation of a sequence of surrogate training objectives, inherited from PAC-Bayes generalisation bounds. The key argument is to replace the empirical risk (seen as a function of hypotheses) in the generalisation bound by its projection onto a constructible low dimensional functional space: these projections can be queried much more efficiently than the initial risk. On top of providing that generic recipe for learning via surrogate PAC-Bayes bounds, we (i) contribute theoretical results establishing that iteratively optimising our surrogates implies the optimisation of the origin... Read More

Source

The topic of learning to solve optimization problems has received interest from both the operations research and machine learning communities. In this paper, we combine ideas from both fields to address the problem of learning to generate decisions to instances of optimization problems with potentially nonlinear or nonconvex constraints where the feasible set varies with contextual features. We propose a novel framework for training a generative model to produce provably optimal decisions by combining interior point methods and adversarial learning, which we further embed within an iterative data generation algorithm. To this end, we first train a classifier to learn feasibility and then train the generative model to produce optimal decisions to an optimization problem using the classifier as a regularizer. We prove that decisions generated by our model satisfy in-sample and out-of-sample optimality guarantees. Furthermore, the learning models are embedded in an active learning loop in which synthetic instances are iteratively added to the training data; this allows us to progressive... Read More

Source

We introduce a data-driven approach to analyze the performance of continuous optimization algorithms using generalization guarantees from statistical learning theory. We study classical and learned optimizers to solve families of parametric optimization problems. We build generalization guarantees for classical optimizers, using a sample convergence bound, and for learned optimizers, using the Probably Approximately Correct (PAC)-Bayes framework. To train learned optimizers, we use a gradient-based algorithm to directly minimize the PAC-Bayes upper bound. Numerical experiments in signal processing, control, and meta-learning showcase the ability of our framework to provide strong generalization guarantees for both classical and learned optimizers given a fixed budget of iterations. For classical optimizers, our bounds are much tighter than those that worst-case guarantees provide. For learned optimizers, our bounds outperform the empirical outcomes observed in their non-learned counterparts.

Source

This paper investigates quantity-insensitive stress learning using the MaxEnt learner of Pater and Prickett (2022) and compares the performance of the learner equipped with three different constraint sets: a foot-based constraint set and two grid-based constraint sets, one drawn directly from Gordon (2002), and one that changes the formulation of the main stress constraint to match the foot-based learner. The learner equipped with the foot-based constraint set succeeds at learning all the languages from the Gordon (2002) typology that it can represent; the structural ambiguity of the foot-based representations is not a problem in this regard. The foot-based learner also learns the languages as quickly in terms of number of epochs as the faster of the grid-based learners, which is the one with the revised main stress constraint. We conclude that the foot-based learner and the grid-based learner fare similarly well in this initial comparison on a typologically grounded set of learning problems.

Source