Encoder-Only vs. Decoder-Only Models: A Specialist's Guide for AI-Driven Drug Discovery

Abstract

This article provides a comprehensive guide for researchers and professionals in drug discovery on the strategic selection and application of encoder-only and decoder-only large language models. It covers foundational architectural principles, details specific methodological applications in biomedical research—from target identification to clinical data processing—and offers practical optimization strategies. A rigorous comparative analysis equips readers to validate and choose the right model architecture, balancing efficiency, accuracy, and computational cost to accelerate and improve outcomes in pharmaceutical development.

Understanding the Core Architectures: From Transformers to Specialized LLMs

The transformer architecture, since its inception, has fundamentally reshaped the landscape of artificial intelligence and natural language processing. Its evolution has bifurcated into two predominant paradigms: encoder-only and decoder-only architectures, each with distinct computational characteristics and application domains. Encoder-only models, such as BERT and RoBERTa, utilize bidirectional attention mechanisms to develop deep contextual understanding of input text, making them exceptionally suited for interpretation tasks like sentiment analysis and named entity recognition [1]. Conversely, decoder-only models like the GPT series employ masked self-attention mechanisms that prevent the model from attending to future tokens, making them inherently autoregressive and optimized for text generation tasks [2] [1]. This architectural divergence represents more than mere implementation differences—it embodies fundamentally opposed approaches to language modeling that continue to drive innovation across research domains, including pharmaceutical development where both understanding and generation capabilities find critical applications [3].

The ongoing debate surrounding these architectures has gained renewed momentum with recent research challenging the prevailing dominance of decoder-only models. Studies demonstrate that encoder-decoder models, when enhanced with modern training methodologies, can achieve comparable performance to decoder-only counterparts while offering superior inference efficiency in certain contexts [4]. This resurgence of interest in encoder-decoder architectures coincides with growing concerns about computational efficiency and specialized domain applications, particularly in scientific fields like drug discovery where both comprehensive understanding and controlled generation are essential [3]. As we deconstruct these architectural blueprints, it becomes evident that the optimal choice depends heavily on specific task requirements, computational constraints, and desired outcome metrics.

Architectural Fundamentals: Encoder vs. Decoder

Encoder-Decoder Architecture

The original transformer architecture, as proposed in "Attention Is All You Need," integrated both encoder and decoder components working in tandem for sequence-to-sequence tasks like machine translation [1]. In this framework, the encoder processes the input sequence bidirectionally, meaning it can attend to all tokens in the input simultaneously—both preceding and following tokens—to create a rich, contextual representation of the entire input [5] [1]. This comprehensive understanding is then passed to the decoder, which generates the output sequence autoregressively, one token at a time, while attending to both the encoder's output and its previously generated tokens [1].

The encoder's bidirectional processing capability enables it to develop a holistic understanding of linguistic context, capturing nuanced relationships between words regardless of their positional relationships [1]. This characteristic makes encoder-focused models particularly valuable for tasks requiring deep comprehension, such as extracting meaningful patterns from scientific literature or identifying complex biomolecular relationships in pharmaceutical research [3]. The encoder's output represents the input sequence in a dense, contextualized embedding space that can be leveraged for various downstream predictive tasks.

Decoder-Only Architecture

Decoder-only architectures emerged as a simplification of the full encoder-decoder model, eliminating the encoder component entirely and relying exclusively on the decoder stack with masked self-attention [2] [1]. This architectural variant processes input unidirectionally, with each token only able to attend to previous tokens in the sequence, not subsequent ones [5] [2]. This causal masking mechanism ensures the model cannot "look ahead" at future tokens during training, making it inherently predictive and ideally suited for generative tasks [2].

The dominance of decoder-only architectures in contemporary LLMs stems from their remarkable generative capabilities and emergent properties [1]. Through pretraining on vast text corpora using simple next-token prediction objectives, these models develop sophisticated language understanding alongside generation abilities, enabling few-shot learning and in-context adaptation without parameter updates [1]. This combination of architectural simplicity and functional power has established decoder-only models as the default choice for general-purpose language modeling, though recent research suggests this dominance may not be universally justified across all application domains [4].

Table 1: Fundamental Differences Between Encoder and Decoder Architectures

Architectural Aspect	Encoder Models	Decoder Models	Encoder-Decoder Models
Attention Mechanism	Bidirectional (attends to all tokens)	Causal/masked (attends only to previous tokens)	Encoder: Bidirectional; Decoder: Causal
Primary Training Objective	Masked language modeling, next sentence prediction	Next token prediction	Sequence-to-sequence reconstruction
Information Flow	Comprehensive context understanding	Autoregressive generation	Understanding → Generation
Typical Applications	Text classification, sentiment analysis, information extraction	Text generation, conversational AI, code generation	Machine translation, text summarization, question answering
Example Models	BERT, RoBERTa	GPT series, Llama, Gemma	T5, BART, T5Gemma

Visualizing Architectural Differences

The following diagram illustrates the fundamental differences in information flow between encoder-only, decoder-only, and encoder-decoder architectures:

Architecture Comparison: Information flow differences between transformer variants.

Empirical Analysis: Performance Comparison

Scaling Properties and Efficiency Metrics

Recent comparative studies have systematically evaluated the scaling properties of encoder-decoder versus decoder-only architectures across model sizes ranging from ~150M to ~8B parameters [4]. These investigations reveal nuanced trade-offs that challenge the prevailing preference for decoder-only models. When pretrained on the RedPajama V1 dataset (1.6T tokens) and instruction-tuned using FLAN, encoder-decoder models demonstrate compelling scaling properties and surprisingly strong performance despite receiving less research attention in recent years [4].

While decoder-only architectures generally maintain an advantage in compute optimality during pretraining, encoder-decoder models exhibit comparable scaling capabilities and context length extrapolation [4]. More significantly, after instruction tuning, encoder-decoder architectures achieve competitive and occasionally superior results on various downstream tasks while offering substantially better inference efficiency [4]. This efficiency advantage stems from the architectural separation of understanding and generation capabilities, allowing for computational optimization that might be particularly valuable in resource-constrained environments like research institutions or for deploying models at scale in production systems.

Table 2: Performance Comparison of Architectural Paradigms (150M to 8B Scale)

Evaluation Metric	Decoder-Only Models	Encoder-Decoder Models	Performance Differential
Pretraining Compute Optimality	High	Moderate	Decoder-only more compute-efficient during pretraining
Inference Efficiency	Moderate	High	Encoder-decoder substantially more efficient after instruction tuning
Context Length Extrapolation	Strong	Comparable	Similar capabilities demonstrated
Instruction Tuning Response	Strong	Strong	Both architectures respond well to instruction tuning
Downstream Task Performance	Varies by task	Comparable/Superior on some tasks	Encoder-decoder competitive and occasionally better
Training Data Requirements	Typically high (100B+ tokens)	Potentially lower (e.g., 100B tokens)	Encoder-decoder may require less data for similar performance

Specialized Architectural Innovations

Beyond the fundamental encoder-decoder dichotomy, numerous specialized architectural innovations have emerged to address specific limitations of standard transformer architectures. DeepSeek's Multi-head Latent Attention (MLA) represents a significant advancement for long-context inference by reducing the size of the KV cache without compromising model quality [6]. Traditional approaches like grouped-query attention and KV cache quantization inevitably involve trade-offs between cache size and model performance, whereas MLA employs low-rank compression of key and value vectors while maintaining essential information through clever recomputation techniques [6].

Mixture-of-Experts (MoE) models constitute another transformative architectural evolution, decoupling model knowledge from activation costs by dividing feedforward blocks into multiple experts with context-dependent routing mechanisms [6]. This approach enables dramatic parameter count increases without proportional computational cost growth, though it introduces challenges like routing collapse where models persistently activate the same subset of experts [6]. DeepSeek v3 addresses this through auxiliary-loss-free load balancing and shared expert mechanisms that maintain training stability while leveraging MoE benefits [6].

The following diagram illustrates the key innovations in modern efficient transformer architectures:

Efficient Architecture Innovations: Key advancements improving transformer scalability.

Experimental Protocols and Methodologies

Comparative Scaling Studies

Rigorous experimental protocols are essential for meaningful architectural comparisons. Recent encoder-decoder versus decoder-only studies employ standardized training and evaluation pipelines that isolate architectural effects from other variables [4]. The pretraining phase utilizes the RedPajama V1 dataset comprising 1.6T tokens, with consistent preprocessing and tokenization across experimental conditions [4]. Models across different scales (from ~150M to ~8B parameters) undergo training with carefully controlled compute budgets, enabling direct comparison of scaling properties and training efficiency.

During instruction tuning, researchers employ the FLAN collection with identical procedures applied to all architectural variants [4]. Evaluation encompasses diverse downstream tasks including reasoning, knowledge retrieval, and specialized domain applications, with metrics normalized to account for parameter count differences [4]. This methodological rigor ensures observed performance differences genuinely reflect architectural characteristics rather than training or evaluation inconsistencies.

Text-to-Image Generation Conditioning Analysis

Beyond traditional language tasks, specialized experimental protocols have been developed to evaluate architectural components in multimodal contexts. Studies investigating decoder-only LLMs as text encoders for text-to-image generation employ standardized training and evaluation pipelines that isolate the impact of different text embeddings [7]. Researchers train 27 text-to-image models with 12 different text encoders while controlling for all other variables, enabling precise attribution of performance differences to architectural features [7].

These experiments systematically analyze critical aspects including embedding extraction methodologies (last-layer vs. layer-normalized averaging across all layers), LLM variants, and model sizes [7]. The findings demonstrate that conventional last-layer embedding approaches underperform compared to more sophisticated layer-normalized averaging techniques, which significantly improve alignment with complex prompts and enhance performance in advanced visio-linguistic reasoning tasks [7]. This methodological approach exemplifies how controlled experimentation can reveal optimal configuration patterns for specific application domains.

Research Reagent Solutions

The following table details key computational "research reagents" – essential components and methodologies used in modern transformer architecture research:

Table 3: Essential Research Reagents for Transformer Architecture Experiments

Research Reagent	Function	Example Implementations
Causal Self-Attention	Enables autoregressive generation by masking future tokens	PyTorch module with masked attention matrix [2]
Rotary Positional Embeddings (RoPE)	Encodes positional information without increasing parameters	Standard implementation in models like Supernova [8]
Grouped Query Attention (GQA)	Reduces KV cache size by grouping query heads	3:1 compression ratio used in Supernova [8]
Multi-Head Latent Attention (MLA)	Advanced KV cache compression without quality loss	DeepSeek's latent dimension approach [6]
Mixture of Experts (MoE)	Increases parameter count without proportional compute increase	DeepSeek v3's auxiliary-loss-free load balancing [6]
RMSNorm	Computational efficiency improvement over LayerNorm	Used in efficient architectures like Supernova [8]
SwiGLU Activation	Enhanced activation function for feedforward networks	Modern alternative to ReLU/GELU [8]
Layer-Normalized Averaging	Extracts embeddings across all layers for better conditioning	Superior to last-layer embeddings in text-to-image [7]

Domain-Specific Applications: Drug Discovery Case Study

The architectural dichotomy between encoder and decoder models takes on particular significance in specialized domains like pharmaceutical research, where both comprehension and generation capabilities are essential. Foundation models have demonstrated remarkable growth in drug discovery applications, with over 200 specialized models published since 2022 supporting diverse applications including target discovery, molecular optimization, and preclinical research [3].

Encoder-style architectures excel in analyzing existing biomedical literature, extracting relationships between chemical structures and biological activity, and predicting molecular properties—tasks requiring deep understanding of complex domain-specific contexts [3] [1]. Their bidirectional attention mechanisms enable comprehensive analysis of molecular structures and biomedical relationships, making them invaluable for target identification and validation phases. Decoder architectures, conversely, demonstrate exceptional capability in generative tasks like molecular design, compound optimization, and synthesizing novel chemical entities with desired properties [3] [5].

The emerging hybrid approach leverages both architectural paradigms in coordinated workflows, with encoder-style models identifying promising therapeutic targets through literature analysis and biological pathway understanding, while decoder-style models generate novel molecular structures targeting these pathways [3]. This synergistic application represents the cutting edge of AI-driven pharmaceutical research, demonstrating how architectural differences can be transformed from theoretical distinctions into complementary tools addressing complex real-world challenges.

The deconstruction of transformer architectures reveals a dynamic landscape where encoder-decoder and decoder-only paradigms each offer distinct advantages depending on application requirements and computational constraints. Recent research challenging decoder-only dominance suggests the AI community may have prematurely abandoned encoder-decoder architectures, which demonstrate compelling performance and efficiency characteristics when enhanced with modern training methodologies [4].

Future architectural evolution will likely focus on hybrid approaches that combine the strengths of both paradigms while integrating specialized innovations like Multi-head Latent Attention for efficient long-context processing [6] and Mixture-of-Experts models for scalable parameter increases [6]. For scientific applications like drug discovery, domain-adapted architectures that incorporate specialized embeddings, structured knowledge mechanisms, and multi-modal capabilities will increasingly bridge the gap between general language modeling and specialized research needs [3].

The optimal architectural blueprint remains context-dependent, with decoder-only models maintaining advantages in general-purpose generation, while encoder-decoder architectures offer compelling efficiency for specific understanding-to-generation workflows [4] [5]. As transformer architectures continue evolving, this nuanced understanding of complementary strengths rather than absolute superiority will guide more effective application across research domains, from pharmaceutical development to specialized scientific discovery.

In the landscape of transformer architectures, encoder-only models represent a distinct paradigm specifically engineered for deep language understanding rather than text generation. Models like BERT, RoBERTa, and the recently introduced ModernBERT utilize a bidirectional attention mechanism, allowing them to process all tokens in an input sequence simultaneously while accessing both left and right context for each token [9] [10] [11]. This fundamental architectural characteristic makes them exceptionally powerful for comprehension tasks where holistic understanding of the input is paramount.

Unlike decoder-only models that process text unidirectionally (left-to-right) and excel at text generation, encoder-only models are trained using objectives like Masked Language Modeling (MLM), where randomly masked tokens must be predicted using surrounding context from both directions [9] [12]. This training approach produces highly contextualized embeddings that capture nuanced semantic relationships, making these models particularly suitable for scientific and industrial applications requiring precision, efficiency, and robust language understanding without the computational overhead of generative models [13] [12].

Architectural Framework and Core Mechanisms

Fundamental Components

The encoder-only architecture consists of stacked transformer encoder layers, each containing two primary sub-components [10]:

• Self-Attention Layer: Computes attention weights between all token pairs in the input sequence, enabling each token to contextualize itself against all others bidirectionally.
• Feed-Forward Neural Network: Further processes the contextualized representations through position-wise fully connected layers.

A critical differentiator from decoder architectures is the absence of autoregressive masking in the attention mechanism. Without masking constraints, the self-attention layers can establish direct relationships between any tokens in the sequence, regardless of position [10] [11].

Visualizing Bidirectional Attention

The following diagram illustrates the bidirectional attention mechanism that enables each token to contextualize itself against all other tokens in the input sequence:

Diagram 1: Bidirectional attention in encoder-only models. Each output embedding derives context from all input tokens.

Training Methodology: Masked Language Modeling

The core training objective for most encoder-only models is Masked Language Modeling (MLM), where:

• 15% of input tokens are randomly replaced with a [MASK] token
• The model must predict the original tokens using bidirectional context
• Unlike autoregressive training, predictions can utilize both preceding and subsequent tokens [9]

Additional pretraining objectives like Next Sentence Prediction (NSP) help the model understand relationships between sentence pairs, further enhancing representation quality for tasks requiring cross-sentence reasoning [9].

Performance Comparison: Encoder vs. Decoder Architectures

Task-Specific Capabilities

Different transformer architectures demonstrate distinct strengths based on their structural designs:

Table 1: Architectural suitability for NLP tasks

Task Category	Suggested Architecture	Examples	Key Rationale
Text Classification	Encoder-only	BERT, RoBERTa, ModernBERT	Bidirectional context enables holistic understanding [11]
Named Entity Recognition	Encoder-only	BERT, RoBERTa	Full context needed for entity boundary detection [11]
Text Generation	Decoder-only	GPT, LLaMA	Autoregressive design matches sequential generation [5] [11]
Machine Translation	Encoder-Decoder	T5, BART	Combines understanding (encoder) with generation (decoder) [11]
Summarization	Encoder-Decoder	BART, T5	Requires comprehension then abstraction [11]
Question Answering (Extractive)	Encoder-only	BERT, RoBERTa	Context matching against full passage [11]

Quantitative Performance Benchmarks

Recent empirical studies directly compare architectural performance across various natural language understanding tasks:

Table 2: Performance comparison on classification tasks (accuracy %)

Model Architecture	Model Name	Sentiment Analysis	Intent Classification	Enhancement Report Approval	Params
Encoder-only	BERT-base	92.5	94.2	73.1	~110M [14] [13]
Encoder-only	RoBERTa-base	93.1	94.8	74.3	~125M [14] [13]
Encoder-only	ModernBERT-base	95.7	96.2	N/A	~149M [12]
Decoder-only	LLaMA 3.1 8B	89.3	90.1	79.0*	~8B [14] [13]
Decoder-only	GPT-3.5-turbo	90.8	91.5	75.2	~20B [14]
Traditional	LSTM+GloVe	88.7	89.3	68.5	~50M [14]

Note: LLaMA 3.1 8B achieved 79% accuracy on Enhancement Report Approval Prediction only after LoRA fine-tuning and incorporation of creator profile metadata [14]

Efficiency and Inference Metrics

Beyond raw accuracy, encoder models demonstrate significant advantages in computational efficiency:

Table 3: Computational efficiency comparison

Metric	Encoder-only (ModernBERT-base)	Decoder-only (LLaMA 3.1 8B)	Advantage Ratio
Inference Speed (tokens/sec)	~2,400	~380	6.3× [12]
Memory Footprint	~0.6GB	~16GB	~26× [12]
Context Length	8,192 tokens	8,000 tokens	Comparable [12]
Monthly Downloads (HF)	~1 billion	~397 million	~2.5× [12]

The efficiency advantage is particularly pronounced in filtering applications. Processing 15 trillion tokens with a fine-tuned BERT model required 6,000 H100 hours (~$60,000), while the same task using decoder-only APIs would exceed $1 million [12].

Experimental Protocols and Methodologies

Standard Evaluation Workflow

Research comparing architectural performance typically follows rigorous experimental protocols:

Diagram 2: Standard experimental workflow for architectural comparison studies.

Key Experimental Design Elements

• Dataset Curation: Studies utilize established benchmarks (GLUE, SuperGLUE) and domain-specific corpora [14] [12]
• Fine-tuning Protocols: Encoder models typically use task-specific heads with cross-entropy loss; decoder models employ instruction tuning or LoRA [14]
• Evaluation Metrics: Standard classification metrics (accuracy, F1, precision, recall) with statistical significance testing [14] [13]
• Computational Controls: Experiments control for parameter count, training data size, and computational budget [4] [13]

For example, the Enhancement Report Approval Prediction study evaluated 18 LLM variants using strict chronological data splitting to prevent temporal bias, with comprehensive hyperparameter optimization for each architecture [14].

Table 4: Key research reagents and resources for encoder model experimentation

Resource Category	Specific Examples	Function/Purpose	Access Method
Pretrained Models	BERT, RoBERTa, DeBERTa, ModernBERT	Foundation models for transfer learning	Hugging Face Hub [12]
Datasets	GLUE, SuperGLUE, domain-specific corpora	Benchmark performance evaluation	Academic repositories [14] [12]
Fine-tuning Frameworks	Transformers, Adapters, LoRA	Task-specific model adaptation	Open-source libraries [14]
Evaluation Suites	Scikit-learn, Hugging Face Evaluate	Standardized performance metrics	Python packages [14]
Computational Resources	GPU clusters, cloud computing	Model training and inference	Institutional/cloud providers

Encoder-only models provide a computationally efficient yet highly effective architecture for natural language understanding tasks predominant in scientific research and industrial applications. Their bidirectional contextualization capabilities deliver state-of-the-art performance on classification, information extraction, and similarity analysis tasks while requiring substantially fewer computational resources than decoder-only alternatives [13] [12].

The recent introduction of ModernBERT demonstrates that ongoing architectural innovations continue to enhance encoder capabilities, including extended context lengths (8K tokens) and improved training methodologies [12]. For research institutions and development teams operating under computational constraints, encoder-only models represent a Pareto-optimal solution, balancing performance with practical deployability.

As the field evolves, the strategic combination of encoder-only models for comprehension tasks and decoder models for generation scenarios enables the development of sophisticated NLP pipelines that maximize both capability and efficiency—a consideration particularly relevant for resource-constrained research environments.

The field of natural language processing has witnessed a significant architectural evolution, transitioning from encoder-dominated paradigms to the current era dominated by decoder-only models. This shift represents more than a mere architectural preference; it reflects a fundamental rethinking of how machines learn, understand, and generate human language. Decoder-only models, characterized by their autoregressive design, predict the next token in a sequence based on all previous tokens, enabling powerful text generation capabilities that underpin modern systems like GPT-4, LLaMA, and Claude [15] [1].

Within the broader research context comparing encoder-only versus decoder-only architectures, this guide objectively examines the performance, experimental protocols, and practical applications of decoder-only models. While encoder-only models like BERT excel in understanding tasks through bidirectional context, and encoder-decoder hybrids like T5 handle sequence-to-sequence tasks, decoder-only architectures have demonstrated remarkable versatility and scaling properties, often achieving state-of-the-art results in both generative and discriminative tasks when sufficiently scaled [16] [17]. This analysis provides researchers and drug development professionals with a comprehensive comparison grounded in experimental data and methodological details.

Architectural Comparison and Performance Analysis

Key Architectural Differences

The fundamental distinction between architectural paradigms lies in their attention mechanisms and training objectives. Encoder-only models utilize bidirectional self-attention, meaning each token in the input sequence can attend to all other tokens, creating a rich, contextual understanding ideal for classification and extraction tasks [1] [16]. In contrast, decoder-only models employ masked self-attention, where each token can only attend to previous tokens in the sequence, making them inherently autoregressive and optimized for text generation [1]. Encoder-decoder models combine both, using bidirectional attention for encoding and masked attention for decoding, suited for tasks like translation where output heavily depends on input structure [1] [16].

A critical theoretical advantage for decoder-only models is their tendency to maintain higher-rank attention weight matrices compared to the low-rank bottleneck observed in bidirectional attention mechanisms [16]. This suggests decoder-only architectures may have greater expressive power, as each token can retain more unique information rather than being homogenized through excessive contextual averaging [16].

Experimental Performance Comparison

Table 1: Comparative Performance Across Model Architectures

Model Architecture	Representative Models	Primary Training Objective	Strengths	Limitations
Encoder-Only	BERT, RoBERTa	Masked Language Modeling (MLM)	Excellent for classification, semantic understanding, produces high-quality embeddings [1] [17]	Poor at coherent long-form text generation [16] [17]
Decoder-Only	GPT series, LLaMA	Autoregressive Language Modeling	State-of-the-art text generation, strong zero-shot generalization, emergent abilities [1] [16]	Can struggle with tasks requiring full bidirectional context [17]
Encoder-Decoder	T5, BART	Varied (often span corruption or denoising)	Powerful for sequence-to-sequence tasks (translation, summarization) [1] [17]	Computationally more expensive, less parallelizable than decoder-only [16]

Table 2: Inference Efficiency and Scaling Properties

Architecture	Inference Efficiency	Scaling Trajectory	Context Window Extrapolation
Encoder-Only	Highly parallelizable during encoding	Performance plateaus at smaller scales [16]	Naturally handles full sequence
Decoder-Only	Sequential generation, but innovations like pipelined decoders improve speed [18]	Strong scaling to hundreds of billions of parameters [16]	Demonstrated strong extrapolation capabilities [4]
Encoder-Decoder	Moderate, due to dual components	Competitive scaling shown in recent studies [4]	Depends on implementation

Recent experimental evidence from direct architectural comparisons reveals nuanced performance differences. In a comprehensive study comparing architectures using 50B parameter models pretrained on 170B tokens, decoder-only models with generative pretraining demonstrated superior zero-shot generalization for generative tasks, while encoder-decoder models with masked language modeling performed best for zero-shot MLM tasks but struggled with answering open questions [16].

For inference efficiency, decoder-only models provide compelling advantages. The RADAr model, a transformer-based autoregressive decoder for hierarchical text classification, demonstrated comparable performance to state-of-the-art methods while providing a 2x speed-up at inference time [19]. Further innovations like pipelined decoders show potential for significantly improving generation speed without substantial quality loss or additional memory consumption [18].

Experimental Protocols and Methodologies

Pretraining Methodology for Decoder-Only Models

The training process for decoder-only models follows a self-supervised approach on large-scale text corpora. The fundamental protocol involves:

• Data Preparation: Large text collections are processed into sequences of tokens. Each sequence is split into overlapping samples where the input is all tokens up to position i, and the target is the token at position i+1 [15]. For example:

  • Input: ["This"] → Target: "is"
  • Input: ["This", "is"] → Target: "a"
  • Input: ["This", "is", "a"] → Target: "sample"

• Autoregressive Objective: The model is trained to predict the next token in a sequence given all previous tokens, formally maximizing the likelihood: P(tokeni | token1, token2, ..., token{i-1}) [15] [1].

• Architecture Configuration: A stack of identical decoder layers, each containing:

  • Masked multi-head self-attention mechanism
  • Feed-forward network (typically a multilayer perceptron)
  • Layer normalization and skip connections [15]

Recent advancements have explored unified decoder-only architectures for multimodal tasks. OneCAT, a decoder-only auto-regressive model for unified understanding and generation, demonstrates how a pure decoder-only architecture can integrate understanding, generation, and editing within a single framework, eliminating the need for external vision components during inference [20].

Benchmarking and Evaluation Protocols

Rigorous evaluation of decoder-only models involves multiple benchmarks across different task categories:

• Generative Tasks: HumanEval, MBPP, and CodeContests for code generation; narrative generation tasks for creative writing [21]
• Reasoning Tasks: MMLU (Massive Multitask Language Understanding) for general knowledge and problem-solving [21]
• Conversational Quality: MT-Bench for multi-turn dialogue capabilities [21]
• Long-Context Understanding: LongBench for evaluating performance on extended contexts [21]

For specialized domains like STEM question answering, experimental protocols involve generating challenging multiple-choice questions using LLMs themselves, then evaluating model performance with and without context, creating a self-evaluation framework [22].

Efficiency Optimization Techniques

Several innovative methods have been developed to address the sequential decoding limitation of autoregressive models:

• Pipelined Decoder Architecture: Initiates generation of multiple subsequences simultaneously, generating a new token for each subsequence at each time step to realize parallelism while maintaining autoregressive properties within subsequences [18].

• Modality-Specific Mixture-of-Experts (MoE): Employs expert networks where different parameters are activated for different inputs or modalities, providing scalability without proportional compute cost increases [20] [17].

Architectural Diagrams and Workflows

Decoder-Only Transformer Architecture

Decoder-Only Model Data Flow

Decoder Block Detailed Components

Single Decoder Block Structure

Research Reagent Solutions

Table 3: Essential Research Components for Decoder-Model Development

Research Component	Function	Example Implementations
Base Architecture	Core transformer decoder blocks with autoregressive attention	GPT architecture, LLaMA, RADAr [19] [1]
Pretraining Corpora	Large-scale text data for self-supervised learning	RedPajama V1 (1.6T tokens), Common Crawl, domain-specific collections [4] [21]
Tokenization Tools	Convert text to model-readable tokens and back	Byte-Pair Encoding (BPE), SentencePiece, WordPiece [15]
Positional Encoding	Inject sequence position information into embeddings	Learned positional embeddings, rotary position encoding (RoPE) [15]
Optimization Frameworks	Efficient training and fine-tuning	AdamW optimizer, learning rate schedulers, distributed training backends [1]
Instruction Tuning Datasets	Align model behavior with human instructions	FLAN collection, custom instruction datasets [4]
Evaluation Benchmarks	Standardized performance assessment	MMLU, HumanEval, MT-Bench, LongBench [21]
Efficiency Libraries	Optimize inference speed and memory usage	vLLM, Llama.cpp, TensorRT-LLM [21]

The architectural landscape of large language models presents researchers with distinct trade-offs between understanding, generation, and efficiency. Decoder-only models have established dominance in generative applications and shown remarkable scaling properties, while encoder-only models maintain advantages in classification and semantic understanding tasks requiring bidirectional context [16] [17]. Encoder-decoder architectures offer compelling performance for sequence-to-sequence tasks but face efficiency challenges compared to single-stack alternatives [16].

For research and development professionals, selection criteria should extend beyond benchmark performance to include data privacy requirements, computational constraints, customization needs, and integration capabilities with existing scientific workflows [21]. The future of architectural development appears to be leaning toward specialized mixtures-of-experts and unified decoder-only frameworks that can efficiently handle multiple modalities and tasks within a single autoregressive paradigm [20] [17]. As the field progresses, the most impactful applications will likely come from strategically matching architectural strengths to specific research problems rather than pursuing one-size-fits-all solutions.

The rapid evolution of Large Language Models (LLMs) has been characterized by a fundamental architectural schism: the division between encoder-only models designed for comprehension and decoder-only models engineered for generation [1]. This architectural dichotomy is not merely a technical implementation detail but rather a core determinant of functional capability, performance characteristics, and ultimately, suitability for specific scientific applications [23]. In domains such as drug development and materials research, where tasks range from molecular property prediction (comprehension) to novel compound design (generation), understanding this architectural imperative becomes crucial for leveraging artificial intelligence effectively [24].

The original Transformer architecture, introduced in the landmark "Attention Is All You Need" paper, contained both encoder and decoder components working in tandem for sequence-to-sequence tasks like machine translation [1] [25]. However, subsequent research and development has seen these components diverge into specialized architectures, each with distinct strengths, training methodologies, and operational characteristics [16]. This article provides a comprehensive comparison of these architectures, grounded in experimental data and tailored to the needs of researchers and scientists navigating the complex landscape of AI tools for scientific discovery.

Architectural Fundamentals: How Encoders and Decoders Work

Core Components and Mechanisms

At their core, both encoder and decoder architectures are built upon the same fundamental building block: the self-attention mechanism [2]. However, they implement this mechanism in critically different ways that dictate their functional capabilities:

• Encoder Architecture: Encoder-only models like BERT and RoBERTa utilize bidirectional self-attention, meaning each token in the input sequence can attend to all other tokens in both directions [1] [26]. This allows the encoder to develop a comprehensive, contextual understanding of the entire input sequence simultaneously. The training objective typically involves Masked Language Modeling (MLM), where random tokens in the input are masked and the model must predict them based on surrounding context [1] [16].

• Decoder Architecture: Decoder-only models such as GPT, LLaMA, and PaLM employ causal (masked) self-attention, which restricts each token from attending to future tokens in the sequence [2] [25]. This unidirectional attention mechanism preserves the autoregressive property essential for text generation, where outputs are produced one token at a time, with each new token conditioned on all previous tokens [1] [2].

Visualizing the Architectural Differences

The following diagram illustrates the fundamental differences in how encoder and decoder architectures process information:

Performance Comparison: Experimental Evidence

Quantitative Analysis Across Task Types

Multiple studies have systematically compared the performance of encoder-only, decoder-only, and encoder-decoder architectures across various tasks. The following table summarizes key findings from recent research:

Table 1: Performance comparison of model architectures across different task types

Architecture	Representative Models	Classification Accuracy	Generation Quality	Inference Speed	Training Efficiency	Key Strengths
Encoder-Only	BERT, RoBERTa, ModernBERT	High [22] [16]	Low [16]	Fast [26]	Moderate [26]	Bidirectional context understanding, efficiency [26]
Decoder-Only	GPT-4, LLaMA, PaLM	Moderate (requires scaling) [16]	High [27] [16]	Slow (autoregressive) [23]	High (parallel pre-training) [27]	Text generation, few-shot learning [1]
Encoder-Decoder	T5, BART, SMI-TED289M	High [24]	High [24]	Moderate [27]	Low (requires paired data) [27]	Sequence-to-sequence tasks [1]

Specialized Performance in Scientific Domains

In scientific domains such as chemistry and drug discovery, the performance characteristics of these architectures manifest in specialized ways. A 2025 study introduced SMI-TED289M, an encoder-decoder model specifically designed for molecular analysis [24]. The model was evaluated across multiple benchmark datasets from MoleculeNet, demonstrating the nuanced performance patterns of different architectures in scientific contexts:

Table 2: Performance of SMI-TED289M encoder-decoder model on molecular tasks [24]

Task Type	Dataset	Metric	SMI-TED289M Performance	Competitive SOTA	Outcome
Classification	BBBP	ROC-AUC	0.921	0.897	Superior
Classification	Tox21	ROC-AUC	0.854	0.851	Comparable
Classification	SIDER	ROC-AUC	0.645	0.635	Superior
Regression	QM9	MAE	0.071	0.089	Superior
Regression	ESOL	RMSE	0.576	0.580	Superior
Reconstruction	MOSES	Valid/Unique	0.941/0.999	0.927/0.998	Superior

The Scaling Perspective: How Model Size Affects Performance

The relationship between architecture and performance is further complicated by scaling effects. Research has demonstrated that encoder-only models typically achieve strong performance quickly with smaller model sizes but tend to plateau, while decoder-only models require substantial scale to unlock their full potential but ultimately achieve superior generalization at large scales [16].

A comprehensive study comparing architectures at the 50-billion parameter scale found that decoder-only models with generative pretraining excelled at zero-shot generalization for creative tasks, while encoder-decoder models with masked language modeling pretraining performed best for zero-shot MLM tasks but struggled with open-ended question answering [16]. This highlights how the optimal architecture depends not only on task type but also on the available computational resources and target model size.

Experimental Protocols: Methodologies for Comparison

Benchmarking Encoder vs. Decoder Performance

To ensure valid comparisons between architectural approaches, researchers have developed standardized evaluation methodologies:

Multilingual Machine Translation Protocol [27]:

• Models Compared: mT5 (encoder-decoder) vs. Llama 2 (decoder-only)
• Languages: Focus on Indian regional languages (Telugu, Tamil, Malayalam)
• Evaluation Metrics: BLEU scores for translation quality, computational efficiency metrics
• Key Findings: Encoder-decoder models generally outperformed in translation quality and contextual understanding, while decoder-only models demonstrated advantages in computational efficiency and fluency

STEM MCQ Evaluation Protocol [22]:

• Dataset: LLM-generated STEM Multiple-Choice Questions from Wikipedia topics
• Models: DeBERTa v3 Large (encoder), Mistral-7B (decoder), Llama 2-7B (decoder)
• Methodology: Fine-tuning with and without context, comparison against closed-source models
• Key Findings: Encoder models (DeBERTa) and smaller decoder models (Mistral-7B) outperformed larger decoder models (Llama 2-7B) on reasoning tasks when appropriate context was provided

Molecular Property Prediction Workflow

For scientific applications, specialized evaluation protocols have been developed. The following diagram illustrates a typical workflow for evaluating model performance on molecular property prediction:

The Scientist's Toolkit: Research Reagent Solutions

Selecting the appropriate model architecture represents a critical strategic decision in AI-driven scientific research. The following table catalogues essential "research reagents" in the AI architecture landscape, with specific guidance for scientific applications:

Table 3: Research reagent solutions for AI-driven scientific discovery

Tool Category	Specific Examples	Function	Considerations for Scientific Use
Encoder Models	BERT, ModernBERT	Text classification, named entity recognition, relation extraction	Ideal for literature mining, patent analysis, and knowledge base construction [26]
Decoder Models	GPT-4, LLaMA, PaLM	Hypothesis generation, research summarization, experimental design	Suitable for generating novel research hypotheses and explaining complex scientific concepts [25]
Encoder-Decoder Models	T5, SMI-TED289M	Molecular property prediction, reaction outcome prediction	Optimal for quantitative structure-activity relationship (QSAR) modeling and reaction prediction [24]
Specialized Scientific Models	SMI-TED289M, MoE-OSMI	Molecular representation learning, property prediction	Domain-specific models pretrained on scientific corpora often outperform general-purpose models [24]
Efficiency Optimization	Alternating Attention, Unpadding	Handling long sequences, reducing computational overhead	Critical for processing large molecular databases or lengthy scientific documents [26]
O-304	O-304 Powder|AMPK Activator	O-304 is a pan-AMPK activator for research into diabetes, metabolism, and cardiovascular function. This product is for research use only and not for human or veterinary use.	Bench Chemicals
Phenprocoumon	Phenprocoumon for Research|VKOR Inhibitor		Bench Chemicals

The architectural dichotomy between encoder and decoder models fundamentally dictates their functional capabilities, with encoder-focused architectures excelling at comprehension tasks and decoder-focused architectures dominating generation tasks [16]. For researchers and drug development professionals, this distinction has practical implications:

When to prefer encoder-style architectures:

• Molecular property prediction and classification [24]
• Scientific document analysis and information extraction [26]
• High-throughput screening and content moderation [26]
• Applications requiring bidirectional context understanding [23]

When to prefer decoder-style architectures:

• Hypothesis generation and research question formulation [25]
• Scientific writing assistance and summarization [1]
• Exploratory molecular design and optimization [24]
• Tasks requiring creative reasoning and analogical thinking [16]

When encoder-decoder models are optimal:

• Molecular representation learning and reconstruction [24]
• Reaction outcome prediction [24]
• Machine translation of scientific literature [27]
• Tasks requiring both deep understanding and sequential output [1]

The emerging trend toward hybridization and architecture-aware model selection promises to further enhance AI-driven scientific discovery, with models like ModernBERT demonstrating that encoder architectures continue to evolve with significant performance improvements [26]. As the AI landscape continues to mature, researchers who strategically match architectural strengths to specific scientific tasks will gain a significant advantage in accelerating discovery and innovation.

The evolution of Large Language Models (LLMs) has been largely defined by the competition and specialization between three core architectural paradigms: encoder-only, decoder-only, and encoder-decoder models. While the transformer architecture introduced both encoder and decoder components for sequence-to-sequence tasks like translation [1], recent years have witnessed a significant architectural shift. The research community has rapidly transitioned toward decoder-only modeling, dominated by models like GPT, LLaMA, and Mistral [4] [28]. However, this transition has occurred without rigorous comparative analysis from a scaling perspective, raising concerns that the potential of encoder-decoder models may have been overlooked [4] [28]. Furthermore, encoder-only models like DeBERTaV3 continue to demonstrate remarkable performance in specific tasks [29], maintaining their relevance in the modern NLP landscape. This guide provides an objective comparison of these architectural families, focusing on their performance characteristics, scaling properties, and optimal application domains for research professionals.

Architectural Fundamentals and Historical Context

Core Architectural Differences

The fundamental differences between architectural families stem from their distinct approaches to processing input sequences and generating outputs:

• Encoder-Only Models (e.g., BERT, RoBERTa, DeBERTa): These models utilize bidirectional self-attention to process entire input sequences simultaneously, capturing rich contextual relationships between all tokens [1]. They are pre-trained using objectives like masked language modeling, where random tokens in the input are masked and the model must predict the original tokens based on their surrounding context [1]. This architecture excels at understanding tasks but does not generate text autoregressively.

• Decoder-Only Models (e.g., GPT series, LLaMA, Mistral): These models employ masked self-attention with causal masking, preventing each token from attending to future positions [30] [1]. This autoregressive property enables them to generate coherent sequences token-by-token while maintaining the constraint that predictions for position i can only depend on known outputs at positions less than i [1]. Pre-trained using causal language modeling, they simply predict the next token in a sequence [28].

• Encoder-Decoder Models (e.g., T5, BART): These hybrid architectures maintain separate encoder and decoder stacks [28]. The encoder processes the input with bidirectional attention, while the decoder generates outputs using causal attention with cross-attention to the encoder's representations [1]. This decomposition often improves sample and inference efficiency for sequence-to-sequence tasks [28].

Evolution of Major Model Families

Table 1: Historical Evolution of Major Model Families

Architecture	Representative Models	Key Innovations	Primary Use Cases
Encoder-Only	BERT, RoBERTa, DeBERTaV3	Bidirectional attention, Masked LM, Next-sentence prediction	Text classification, Named entity recognition, Sentiment analysis
Decoder-Only	GPT-3/4, LLaMA 2/3, Mistral, Gemma	Causal autoregressive generation, Emergent in-context learning	Text generation, Question answering, Code generation
Encoder-Decoder	T5, BART, Flan-T5	Sequence-to-sequence learning, Transfer learning across tasks	Translation, Summarization, Text simplification

The rapid ascent of decoder-only models has been particularly notable, with architectures like LLaMA 3 (8B and 70B parameters) and Mistral's Mixture-of-Experts models dominating recent open-source developments [31]. However, concurrent research has revisited encoder-decoder architectures (RedLLM) with enhancements from modern decoder-only LLMs, demonstrating their continued competitiveness, especially after instruction tuning [4] [28].

Experimental Comparison: Performance and Scaling

Methodology for Architectural Comparison

Recent rigorous comparisons between architectural families have employed standardized experimental protocols to enable fair evaluation. The RedLLM study implemented a controlled methodology with these key components [28]:

• Training Data: All models were pretrained on RedPajama V1 for approximately 1.6 trillion tokens to ensure consistent comparison of scaling properties without data quality confounders.
• Model Scales: Comprehensive comparison across multiple model sizes ranging from ~150 million to ~8 billion parameters to analyze scaling laws.
• Architectural Alignment: RedLLM (encoder-decoder) was enhanced with recent recipes from DecLLM (decoder-only), including rotary positional embedding with continuous positions, SwiGLU FFN activation, and RMSNorm for pre-normalization.
• Training Objectives: Decoder-only models used causal language modeling, while encoder-decoder models employed prefix language modeling for pretraining.
• Evaluation Framework: Models were evaluated on both in-domain (RedPajama samples) and out-of-domain (Paloma samples) data, with zero-shot and few-shot capabilities assessed across 13 diverse downstream tasks after instruction tuning on FLAN.

Quantitative Performance Comparison

Table 2: Performance Comparison Across Model Architectures on STEM MCQs

Model Architecture	Specific Model	STEM MCQ Accuracy	Key Strengths	Computational Efficiency
Encoder-Only	DeBERTa V3 Large	High (Outperforms Llama 2-7B)	Superior on understanding tasks with provided context	Efficient inference
Decoder-Only	Mistral-7B Instruct	High (Outperforms Llama 2-7B)	Strong few-shot capability, text generation	Moderate inference cost
Decoder-Only	Llama 2-7B	Lower baseline	General language understanding	Moderate inference cost
Encoder-Decoder	RedLLM (Post-instruction tuning)	Comparable to Decoder-Only	Strong performance after fine-tuning, efficient inference	High inference efficiency

In a specialized evaluation on challenging LLM-generated STEM multiple-choice questions, encoder-only models like DeBERTa V3 Large demonstrated remarkable performance when provided with appropriate context through fine-tuning, even outperforming some decoder-only models like Llama 2-7B [22]. This highlights that architectural advantages are often task-dependent and context-reliant.

Scaling Properties and Efficiency Analysis

Table 3: Scaling Properties and Efficiency Comparison

Architecture	Scaling Exponent	Compute Optimality	Inference Efficiency	Context Length Extrapolation
Decoder-Only (DecLLM)	Similar scaling	Dominates compute-optimal frontier	Moderate efficiency	Strong capabilities
Encoder-Decoder (RedLLM)	Similar scaling	Less compute-optimal	Substantially better	Promising capabilities

The comprehensive scaling analysis reveals that while both RedLLM and DecLLM show similar scaling exponents, decoder-only models almost dominate the compute-optimal frontier during pretraining [28]. However, after instruction tuning, encoder-decoder models achieve comparable zero-shot and few-shot performance to decoder-only models across scales while enjoying significantly better inference efficiency [28]. This presents a crucial quality-efficiency trade-off for research applications.

Figure 1: Workflow of Architectural Performance Across Training Stages

Specialized Applications in Scientific Domains

Performance on Scientific and Technical Tasks

Different architectural paradigms demonstrate distinct advantages for scientific and research applications:

• Encoder-Only Models maintain strong performance on classification-based scientific tasks, with DeBERTaV3 remaining a top performer among encoder-only models even when newer architectures like ModernBERT are trained on identical data [29]. This suggests their performance edge comes from architectural and training objective optimizations rather than differences in data.

• Decoder-Only Models exhibit emergent capabilities in complex reasoning tasks, with specialized versions like DeepSeek-R1 demonstrating strong performance in mathematical problem-solving, logical inference, and complex reasoning through self-verification and chain-of-thought reasoning [32] [33].

• Encoder-Decoder Models show particular strength in tasks requiring sustained source awareness and complex mapping between input and output sequences, such as literature summarization, protocol translation, and data transformation tasks common in scientific workflows [30] [1].

Attention Mechanisms and Information Flow

A critical differentiator between architectures lies in their attention mechanisms and information flow:

Figure 2: Information Flow in Different Transformer Architectures

Decoder-only models face challenges with "attention degeneration," where decoder-side attention focus on source tokens degrades as generation proceeds, potentially leading to hallucinated or prematurely truncated outputs [30]. This is quantified through sensitivity analysis showing that as the generation index grows, sensitivity to the source diminishes in decoder-only structures [30]. Innovative approaches like Partial Attention LLM (PALM) have been developed to maintain source sensitivity for long generations [30].

Benchmarking Datasets and Evaluation Tools

Table 4: Essential Research Resources for Model Evaluation

Resource Name	Type	Primary Function	Architectural Relevance
RedPajama V1	Pretraining Corpus	Large-scale text corpus for model pretraining	Universal across architectures
FLAN	Instruction Dataset	Collection of instruction-following tasks	Critical for instruction tuning
Paloma	Evaluation Benchmark	Out-of-domain evaluation dataset	Scaling law analysis
STEM MCQ Dataset	Specialized Benchmark	Challenging LLM-generated science questions	Evaluating reasoning with context
HumanEvalX	Code Benchmark	Evaluation of code generation capabilities	Decoder-only specialization

These research reagents form the foundation for rigorous architectural comparisons. The STEM MCQ dataset, specifically created by employing various LLMs to generate challenging questions on STEM topics curated from Wikipedia, addresses the absence of benchmark STEM datasets on MCQs created by LLMs [22]. This enables more meaningful evaluation of model capabilities on scientifically relevant tasks.

The modern landscape of large language models reveals a nuanced architectural ecosystem where encoder-only, decoder-only, and encoder-decoder models each occupy distinct optimal application domains. Encoder-only models like DeBERTaV3 continue to excel in understanding tasks and maintain competitive performance through architectural refinements [29]. Decoder-only models dominate generative applications and demonstrate superior compute optimality during pretraining [28]. Encoder-decoder architectures, often overlooked in recent trends, offer compelling performance after instruction tuning with substantially better inference efficiency [4] [28].

For research professionals, the architectural choice involves careful consideration of task requirements, computational constraints, and performance priorities. While the field has witnessed a pronounced shift toward decoder-only models, evidence suggests that encoder-decoder architectures warrant renewed attention, particularly for applications requiring both comprehensive input understanding and efficient output generation. Future architectural developments will likely continue to blend insights from all three paradigms, creating increasingly specialized and efficient models for scientific applications.

Translating Architecture to Action: LLM Applications in Drug Development Pipelines

Within the rapidly evolving landscape of artificial intelligence for scientific discovery, a architectural dichotomy has emerged: encoder-only versus decoder-only transformer models. While decoder-only models have recently dominated headlines for their generative capabilities, encoder-only models maintain critical importance in scientific domains requiring deep understanding and analysis of complex data patterns, particularly in druggable target identification. Encoder-only architectures, characterized by their bidirectional processing capabilities, excel at extracting meaningful representations from input sequences by examining both left and right context of each token simultaneously [5] [1]. This architectural advantage makes them exceptionally well-suited for classification and extraction tasks where comprehensive context understanding outweighs the need for text generation.

In pharmaceutical research, the identification and classification of druggable targets represents a foundational challenge with profound implications for therapeutic development. Traditional approaches struggle with the complexity of biological systems, data heterogeneity, and the high costs associated with experimental validation [34]. Encoder-only models offer a transformative approach by leveraging large-scale biomedical data to identify patterns and relationships that elude conventional computational methods. As the field progresses, understanding the specific advantages, implementation requirements, and performance characteristics of encoder-only architectures becomes essential for researchers aiming to harness AI for accelerated drug discovery.

Architectural Advantages of Encoder-Only Models for Biomedical Data

Encoder-only models possess distinct architectural characteristics that make them particularly effective for handling the complexities of biomedical data. Unlike decoder-only models that use masked self-attention to prevent access to future tokens, encoder-only models employ bidirectional attention mechanisms that process entire input sequences simultaneously [5] [1]. This capability is crucial for biological context understanding, where the meaning of a protein sequence element or chemical compound often depends on surrounding contextual information.

The pretraining objectives commonly used for encoder-only models further enhance their suitability for biomedical classification tasks. Through masked language modeling (MLM), these models learn to predict randomly masked tokens based on their surrounding context, forcing them to develop robust representations of biological language structure [1]. For example, when processing protein sequences, this approach enables the model to learn the relationships between amino acid residues and their structural implications. Additional pretraining strategies like next sentence prediction help models understand relationships between biological entities, such as drug-target interactions or pathway components [1].

Another significant advantage lies in the computational efficiency of encoder-only architectures for classification tasks. Unlike autoregressive decoding that requires sequential token generation, encoder models can process entire sequences in parallel during inference, resulting in substantially faster throughput for extractive and discriminative tasks [35]. This efficiency becomes particularly valuable when screening large compound libraries or analyzing extensive genomic datasets where rapid iteration is essential.

Table 1: Architectural Comparison for Biomedical Applications

Feature	Encoder-Only Models	Decoder-Only Models	Relevance to Drug Target ID
Attention Mechanism	Bidirectional	Causal (Masked)	Full context understanding for protein classification
Training Objective	Masked Language Modeling	Next Token Prediction	Better representation learning for sequences
Inference Pattern	Parallel processing	Sequential generation	Faster screening of compound libraries
Output Type	Class labels, embeddings	Generated sequences	Ideal for classification tasks
Context Utilization	Full sequence context	Left context only	Comprehensive biomolecular pattern recognition

Implementation Framework: optSAE-HSAPSO for Target Identification

Experimental Protocol and Model Architecture

A groundbreaking demonstration of encoder-only capabilities in drug discovery comes from the optSAE-HSAPSO framework, which integrates a Stacked Autoencoder (SAE) for feature extraction with a Hierarchically Self-Adaptive Particle Swarm Optimization (HSAPSO) algorithm for parameter optimization [36]. This approach specifically addresses key limitations in conventional drug classification methods, including overfitting, computational inefficiency, and limited scalability to large pharmaceutical datasets.

The experimental protocol begins with comprehensive data preprocessing of drug-related information from curated sources including DrugBank and Swiss-Prot. The input features encompass molecular descriptors, structural properties, and known interaction profiles that collectively characterize each compound's potential as a drug candidate. The processed data then feeds into the Stacked Autoencoder component, which performs hierarchical feature learning through multiple encoding layers, progressively capturing higher-level abstractions of the input data [36]. This deep representation learning enables the model to identify complex, non-linear patterns that correlate with druggability.

The HSAPSO optimization phase dynamically adjusts hyperparameters throughout training, balancing exploration and exploitation to navigate the complex parameter space efficiently [36]. Unlike static optimization methods, this adaptive approach continuously refines model parameters based on performance feedback, preventing premature convergence to suboptimal solutions. The integration of swarm intelligence principles enables robust optimization without relying on gradient information, making it particularly effective for the non-convex optimization landscapes common in deep learning architectures.

Performance Metrics and Comparative Analysis

When evaluated on standard benchmarks, the optSAE-HSAPSO framework achieved remarkable performance metrics, including a 95.52% classification accuracy in identifying druggable targets [36]. This accuracy substantially outperformed traditional machine learning approaches like support vector machines and XGBoost, which typically struggle with the high dimensionality and complex relationships within pharmaceutical data. The model also demonstrated exceptional computational efficiency, processing samples in approximately 0.010 seconds each with remarkable stability (±0.003) across iterations [36].

The robustness of the approach was further validated through receiver operating characteristic (ROC) and convergence analyses, which confirmed consistent performance across both validation and unseen test datasets [36]. This generalization capability is particularly valuable in drug discovery contexts where model applicability to novel compound classes is essential. The framework maintained high performance across diverse drug categories and target classes, demonstrating its versatility for real-world pharmaceutical applications.

Table 2: Performance Comparison of Drug Classification Methods

Method	Accuracy	Computational Time (per sample)	Stability	Key Advantages
optSAE-HSAPSO	95.52%	0.010s	±0.003	High accuracy, optimized feature extraction
XGBoost	94.86%	Not Reported	Lower	Good performance, limited scalability
SVM-based	93.78%	Not Reported	Moderate	Handles high-dimension data, slower with large datasets
Traditional ML	89.98%	Not Reported	Lower	Interpretable, struggles with complex patterns

Encoder Specialization in Biomedical Domains: BioClinical ModernBERT

Domain Adaptation and Architecture Optimization

The development of BioClinical ModernBERT represents a specialized implementation of encoder-only architectures specifically designed for biomedical natural language processing tasks [37]. This model builds upon the ModernBERT architecture but incorporates significant domain adaptations through continued pretraining on the largest biomedical and clinical corpus to date, encompassing over 53.5 billion tokens from diverse institutions, domains, and geographic regions [37]. This extensive domain adaptation addresses a critical limitation of general-purpose language models when applied to specialized scientific contexts.

A key architectural enhancement in BioClinical ModernBERT is the extension of the context window to 8,192 tokens, enabled through rotary positional embeddings (RoPE) [37]. This expanded context capacity allows the model to process entire clinical notes and research documents without fragmentation, preserving critical long-range dependencies that are essential for accurate biomedical understanding. The model also features an expanded vocabulary of 50,368 terms (compared to BERT's 30,000), specifically tuned to capture the diversity and complexity of clinical and biomedical terminology [37].

The training methodology employed a two-stage continued pretraining approach, beginning with the base ModernBERT architecture and progressively adapting it to biomedical and clinical language patterns [37]. This strategy leverages transfer learning to preserve the general linguistic capabilities developed during initial pretraining while specializing the model's knowledge toward domain-specific terminology, relationships, and conceptual frameworks.

Performance Benchmarks and Applications

In comprehensive evaluations across four downstream biomedical NLP tasks, BioClinical ModernBERT established new state-of-the-art performance levels for encoder-based architectures [37]. The model demonstrated particular strength in named entity recognition, relation extraction, and document classification tasks essential for drug target identification. By processing longer context sequences, the model achieved superior performance in identifying relationships between biological entities dispersed throughout scientific literature and clinical documentation.

The practical utility of BioClinical ModernBERT in drug discovery pipelines includes its ability to extract structured information from unstructured biomedical text, such as identifying potential drug targets from research publications or clinical trial reports [37]. The model's bidirectional encoding capabilities enable it to capture complex relationships between genetic variants, protein functions, and disease mechanisms that would be challenging to discern with unidirectional architectures. Furthermore, the model's efficiency advantages make it suitable for large-scale literature mining applications, where thousands of documents must be processed to identify promising therapeutic targets.

Table 3: BioClinical ModernBERT Model Specifications

Parameter	Base Model	Large Model	Significance for Target ID
Parameters	150M	396M	Scalable capacity for complex tasks
Context Window	8,192 tokens	8,192 tokens	Processes full documents without fragmentation
Vocabulary	50,368 terms	50,368 terms	Comprehensive biomedical terminology
Training Data	53.5B tokens	53.5B tokens	Extensive domain adaptation
Positional Encoding	RoPE	RoPE	Supports long-context understanding

Comparative Analysis: Encoder vs. Decoder Architectures in Materials Research

The debate between encoder-only and decoder-only architectures extends beyond general NLP tasks to specialized applications in materials science and drug discovery. Recent research has systematically compared these architectural paradigms from a scaling perspective, evaluating performance across model sizes ranging from ~150M to ~8B parameters [4]. These investigations reveal that while decoder-only models generally demonstrate superior compute-optimal performance during pretraining, encoder-decoder and specialized encoder-only architectures can achieve comparable scaling properties and context length extrapolation capabilities [4].

For classification-focused tasks in drug discovery, encoder-only models exhibit distinct advantages in inference efficiency. After instruction tuning, encoder-based architectures achieve comparable and sometimes superior performance on various downstream tasks while requiring substantially fewer computational resources during inference [4]. This efficiency advantage becomes increasingly significant when deploying models at scale for high-throughput screening applications.

However, the architectural choice depends heavily on the specific requirements of the research task. Decoder-only models maintain advantages in generative applications, such as designing novel molecular structures or generating hypothetical compound profiles [38]. The emergent capabilities of large decoder models, including in-context learning and chain-of-thought reasoning, provide flexible problem-solving approaches that complement the specialized strengths of encoder architectures [1]. This suggests that integrated frameworks leveraging both architectural paradigms may offer the most powerful solution for comprehensive drug discovery pipelines.

Successful implementation of encoder-only models for drug target identification requires access to specialized data resources, computational frameworks, and evaluation tools. The following table summarizes key components of the research toolkit for encoder-based drug discovery pipelines:

Table 4: Research Reagent Solutions for Encoder-Based Target Identification

Resource Category	Specific Examples	Function	Access Considerations
Biomedical Databases	DrugBank, Swiss-Prot, ChEMBL	Provides structured drug and target information	Publicly available with registration
Chemical Databases	PubChem, ZINC	Source molecular structures and properties	Open access
Domain-Adapted Models	BioClinical ModernBERT, BioBERT	Pretrained encoders for biomedical text	Some publicly available, others require request
Optimization Frameworks	HSAPSO, LoRA	Efficient parameter tuning and adaptation	Open source implementations available
Evaluation Benchmarks	MedNLI, BioASQ	Standardized performance assessment	Publicly available
Specialized Libraries	Transformers, ChemBERTa	Implementation of model architectures	Open source

Encoder-only models represent a powerful and efficient architectural paradigm for drug target identification and classification tasks. Their bidirectional processing capabilities, computational efficiency, and specialized domain adaptations make them particularly well-suited for the complex challenges of pharmaceutical research. The demonstrated success of frameworks like optSAE-HSAPSO in achieving high-precision classification and BioClinical ModernBERT in extracting meaningful insights from biomedical literature underscores the transformative potential of these approaches.

As the field advances, several emerging trends are likely to shape the evolution of encoder architectures for drug discovery. The development of increasingly specialized encoders pretrained on domain-specific corpora will enhance performance on specialized tasks like binding site prediction and polypharmacology profiling. The integration of multimodal capabilities will enable encoders to process diverse data types, including molecular structures, omics profiles, and scientific literature within unified architectures [34]. Additionally, the emergence of hybrid architectures that strategically combine encoder and decoder components will provide balanced solutions that leverage the strengths of both approaches.

For researchers and drug development professionals, encoder-only models offer a validated pathway for enhancing the efficiency and accuracy of target identification workflows. By leveraging these architectures within comprehensive drug discovery pipelines, the pharmaceutical industry can accelerate the translation of biological insights into therapeutic interventions, ultimately reducing development timelines and improving success rates. The continued refinement of encoder architectures and their integration with experimental validation frameworks will further solidify their role as indispensable tools in modern drug discovery.

Leveraging Encoders for High-Throughput Data Extraction and Entity Recognition

Within materials science and drug development, the ability to automatically and accurately extract specific entities from vast volumes of unstructured text—such as research papers, lab reports, and clinical documents—is paramount for accelerating discovery. This task of Named Entity Recognition (NER) has become a key benchmark for natural language processing (NLP) models. The current landscape is dominated by two transformer-based architectural paradigms: the encoder-only models, exemplified by BERT and its variants, and the decoder-only models, which include large language models (LLMs) like GPT. While decoder-only models have captured significant attention for their generative capabilities, a growing body of evidence indicates that encoder-only architectures offer superior performance and efficiency for structured information extraction tasks. This guide provides a objective comparison of these architectures, underpinned by recent experimental data, to inform researchers selecting the optimal tools for high-throughput data extraction.

Fundamentally, both encoder and decoder architectures are built on the transformer's self-attention mechanism, but they are designed for different primary objectives [1].

• Encoder-Only Models (e.g., BERT, RoBERTa): These models are designed to create rich, bidirectional representations of input text. During pre-training, they use objectives like Masked Language Modeling (MLM), where random tokens in the input sequence are masked, and the model must predict them using the surrounding context from both the left and the right. This forces the model to develop a deep, contextual understanding of each word in a sentence, making the resulting embeddings exceptionally well-suited for discriminative tasks like classification and, crucially, Named Entity Recognition [1] [16].

• Decoder-Only Models (e.g., GPT series): These models are designed for autoregressive text generation. They use a causal language modeling objective, predicting the next token in a sequence based solely on the preceding tokens. This unidirectional, left-to-right context is ideal for generating coherent text but provides a less complete contextual understanding for each token compared to a bidirectional encoder [1] [16].

• Encoder-Decoder Models (e.g., T5, T5Gemma): These hybrid models, the architecture used in the original transformer for translation, are designed for sequence-to-sequence tasks. They encode the input text and then autoregressively decode it into an output sequence. Recent research suggests that with modern training recipes, they can achieve compelling performance and high inference efficiency [4].

The core architectural difference is summarized in the diagram below, which illustrates the flow of information and the primary pre-training objectives for each model type.

Performance Comparison in Data Extraction Tasks

Experimental results across multiple domains, particularly in technical and scientific fields, consistently demonstrate the advantage of encoder-only models for entity recognition. The following table summarizes key findings from recent comparative studies.

Table 1: Comparative Performance of Encoder and Decoder Models on Entity Recognition

Model Architecture	Task / Domain	Key Metric	Performance	Inference Efficiency
Encoder-Only (Flat NER) [39] [40]	NER on Clinical Reports (Pathology)	F1-Score	0.87 - 0.88	High
Encoder-Only (Flat NER) [39] [40]	NER on Clinical Reports (Radiology)	F1-Score	Up to 0.78	High
Decoder-Only (LLMs, Instruction-based) [39] [40]	NER on Clinical Reports	F1-Score	0.18 - 0.30	Lower
Encoder-Only (DeBERTa v3 Large) [22]	STEM MCQ Answering (with context)	Performance vs. Decoders	Outperformed 7B Decoders	High
Decoder-Only (Mistral-7B Instruct) [22]	STEM MCQ Answering (with context)	Performance vs. Decoders	Competitive	Medium
Decoder-Only (Llama 2-7B) [22]	STEM MCQ Answering (with context)	Performance vs. Decoders	Lower	Medium

The data reveals a clear trend: encoder-only models achieve significantly higher F1-scores in NER tasks. The primary weakness of decoder-only LLMs in these extraction tasks is their low recall, meaning they often fail to identify all relevant entities in a text, despite having high precision on the entities they do extract [39] [40]. This "overly conservative" output generation limits their comprehensiveness.

Experimental Protocols and Methodologies

To ensure the reproducibility of the results presented in the previous section, this section details the experimental methodologies employed in the cited studies.

Named Entity Recognition in Clinical Reports

A seminal study directly compared encoder and decoder models for extracting clinical entities from unstructured pathology and radiology reports [39] [40].

• Datasets: A curated dataset of 2,013 pathology reports and 413 radiology reports was annotated by medical students to serve as ground truth for training and testing.
• Compared Methods:
  • Flat NER: Utilized transformer-based encoder models (e.g., BERT-style) fine-tuned to predict entity spans.
  • Nested NER: Used a multi-task learning setup with encoder models to handle entities within other entities.
  • Instruction-based NER: Leveraged decoder-only LLMs, which were given natural language instructions to extract the required entities.
• Evaluation Protocol: Standard NER evaluation metrics were used, primarily the F1-score, which is the harmonic mean of precision and recall. The performance was evaluated separately on the pathology and radiology test sets.

The workflow for this comparative experiment is illustrated below.

Answering STEM Multiple-Choice Questions

Another study highlighted the importance of architectural choice and context for complex reasoning tasks in science and technology [22].

• Dataset Construction: Due to the absence of a benchmark, LLMs (including Vicuna-13B, Bard, and GPT-3.5) were used to generate challenging Multiple-Choice Questions (MCQs) on STEM topics curated from Wikipedia.
• Model Evaluation: The open-source encoder model DeBERTa v3 Large and decoder LLMs like Mistral-7B Instruct and Llama 2-7B were evaluated.
• Experimental Conditions: Models were tested under two conditions: inference with context (where the relevant text was provided alongside the question) and fine-tuning with and without context.
• Key Finding: The encoder-only model (DeBERTa) and the smaller, optimized decoder (Mistral-7B) outperformed the larger Llama 2-7B model, demonstrating that a model's parameter count is less critical than its architecture and training for such discriminative tasks when appropriate context is provided [22].

Case Studies in Drug Discovery and Healthcare

The theoretical advantages of encoders translate into tangible benefits in real-world research applications, from predicting drug-target interactions to analyzing electronic health records.

• Drug-Target Affinity (DTA) Prediction: The TEFDTA model exemplifies the power of encoder architectures in bioinformatics. It uses a transformer encoder to process the sequences of proteins and drugs (represented as SMILES strings) to predict binding affinity. This approach achieved a significant improvement—an average of 7.6% on non-covalent binding affinity prediction and a remarkable 62.9% on covalent binding affinity prediction over certain existing methods [41]. This demonstrates the encoder's capability to handle complex, sequential scientific data representations.

• Clinical Outcome Prediction: TransformEHR is a transformer-based encoder-decoder model designed for electronic health records (EHR). It is pre-trained on 6.5 million patient records with a novel objective: predicting all diseases and outcomes of a patient's future visit based on previous visits. This generative pre-training allows it to learn rich, contextual representations of medical codes. When fine-tuned, it set a new state-of-the-art in predicting specific, challenging outcomes like pancreatic cancer onset and intentional self-harm among patients with PTSD, showcasing the power of tailored encoder-decoder frameworks for complex predictive tasks in healthcare [42].

The Scientist's Toolkit: Research Reagents & Solutions

For researchers aiming to implement encoder-based models for data extraction, the following table catalogues essential "research reagents" – the key datasets, software, and model architectures.

Table 2: Essential Tools for Encoder-Based Data Extraction Research

Tool Name / Type	Function	Relevance to Encoder Models
Annotated Clinical Reports [39] [40]	Gold-standard data for training and evaluation	Provides the labeled data required to fine-tune encoder models for medical NER.
Biomedical Pre-trained Models (e.g., BioBERT)	Domain-specific language model	Encoder pre-trained on scientific/medical text, offering a superior starting point over general-purpose models.
Transformer Libraries (e.g., Hugging Face Transformers)	Software framework	Provides open-source implementations of major encoder architectures (BERT, RoBERTa) for easy fine-tuning and deployment.
SMILES Strings [41]	Representation of drug molecular structure	A sequential, text-based representation that encoder models can process to predict drug-target interactions.
Protein FASTA Sequences [41]	Representation of protein amino acid sequences	The standard sequential data format for proteins that encoders can use as input for binding affinity prediction.
Electronic Health Records (EHR) Datasets [42]	Longitudinal patient data for pre-training	Large-scale datasets used for pre-training encoder models on medical concepts, enabling transfer learning for specific tasks.
Vut-MK142	VUT-MK142|Cardiomyogenic Small Molecule|For Research Use	VUT-MK142 promotes stem cell differentiation into cardiomyocytes for cardiac repair research. This product is for Research Use Only, not for human or veterinary use.
Yunnandaphninine G	Yunnandaphninine G, MF:C30H47NO3, MW:469.7 g/mol	Chemical Reagent

The empirical evidence is clear: for the critical task of high-throughput data extraction and entity recognition in scientific and medical research, encoder-only models currently provide a superior combination of performance and efficiency compared to decoder-only large language models. Their bidirectional nature, born from pre-training objectives like Masked Language Modeling, equips them with a deeper understanding of textual context, which directly translates to higher accuracy and recall in extracting entities from complex documents. While decoder-only LLMs excel in generative tasks and can be prompted for extraction, their tendency towards low recall makes them less reliable for comprehensive information extraction. As the field evolves, hybrid encoder-decoder models are showing renewed promise. However, for researchers and drug development professionals building tools today where precision and recall are non-negotiable, encoder-based architectures remain the definitive and most robust choice.

The field of molecular AI has witnessed a significant architectural evolution, transitioning from encoder-only models to decoder-only frameworks for generative tasks. Encoder-only models, such as BERT-like architectures, excel at understanding molecular representations and property prediction through their bidirectional attention mechanisms. In contrast, decoder-only models have emerged as powerful tools for de novo molecular design and optimization through their autoregressive generation capabilities [43]. This architectural shift mirrors developments in natural language processing but presents unique challenges and opportunities in the molecular domain.

Decoder-only models for molecular design typically process simplified molecular-input line-entry system (SMILES) strings or other string-based representations autoregressively, predicting each token in sequence based on preceding context [44] [45]. This approach has demonstrated remarkable effectiveness in exploring chemical space and generating novel molecular structures with desired properties. The following analysis examines the performance, methodologies, and practical applications of decoder-only architectures in molecular design, providing researchers with a comprehensive comparison framework relative to alternative approaches.

Performance Benchmarking: Decoder-Only Models vs. Alternatives

Quantitative Performance Metrics Across Model Architectures

Table 1: Performance comparison of molecular models across benchmark tasks

Model	Architecture	Params	Training Data	Validity (%)	Uniqueness (%)	Novelty (%)	Property Optimization Score
GP-MoLFormer-Uniq [44]	Decoder-only	46.8M	650M unique SMILES	>99%	>99%	80-90%	0.883 (Perindopril MPO)
SMI-TED289M [24]	Encoder-decoder	289M	91M molecules	N/A	N/A	N/A	SOTA on MoleculeNet
CharRNN [44]	RNN	~10M	1.6M ZINC	~94%	~99%	~80%	Moderate
JT-VAE [44]	VAE	~20M	1.6M ZINC	100%*	~99%	~60%	Limited
MolGen-7b [44]	Decoder-only	7B	100M ZINC	100%*	~98%	~85%	High

Note: Validity marked with * indicates models using SELFIES representation guaranteeing 100% validity [44]

Decoder-only models demonstrate competitive performance across multiple metrics critical for molecular generation. GP-MoLFormer-Uniq, with only 46.8 million parameters, achieves exceptional validity and uniqueness while maintaining high novelty rates, highlighting the efficiency of decoder-only architectures for exploring chemical space [44]. The model's performance on the Perindopril MPO task (score: 0.883) represents a 6% improvement over competing models, demonstrating its effectiveness for targeted molecular optimization [45].

When compared to encoder-decoder models like SMI-TED289M, decoder-only architectures show particular strengths in generative tasks, while encoder-decoder models excel in property prediction benchmarks [24]. This performance differential highlights the specialization-effect between architectural paradigms, with decoder-only models naturally suited for sequential generation tasks.

Property Prediction Performance Across Benchmarks

Table 2: Property prediction performance across molecular benchmarks

Model	QM9 (MAE)	ESOL (RMSE)	FreeSolv (RMSE)	Lipophilicity (RMSE)	Drug-likeness (Accuracy)
SMI-TED289M [24]	0.012	0.58	1.15	0.655	95.2%
Encoder-only Baseline [43]	0.018	0.72	1.42	0.725	92.8%
GP-MoLFormer [44]	N/A	N/A	N/A	N/A	94.7%

While decoder-only models primarily excel at generation, they can be adapted for property prediction through fine-tuning. However, encoder-decoder models like SMI-TED289M maintain advantages in regression tasks, outperforming alternatives across quantum mechanical and biophysical property prediction benchmarks [24]. Interestingly, research suggests that for specific understanding tasks like word meaning comprehension, encoder-only models with fewer parameters can outperform decoder-only models [43], though this effect varies significantly in molecular domains where structural reasoning is required.

Experimental Protocols and Methodologies

Decoder-Only Model Training Framework

The standard training protocol for decoder-only molecular models involves two primary phases: pretraining on large-scale molecular datasets followed by task-specific fine-tuning.

Pretraining Phase: Models are trained on massive datasets of SMILES strings (650 million to 1.1 billion molecules) using causal language modeling objectives [44]. Each SMILES sequence (S=(c1,c2,\dots,cL)) is decomposed into training pairs ((xi,yi)) where (xi=(c1,c2,\dots,ci)) and (yi=(c1,c2,\dots,ci,c{i+1})) for (i=1,2,\dots,L-1) [45]. This approach teaches the model SMILES syntax and chemical validity while capturing the distribution of chemical space.

Architecture Specifications: GP-MoLFormer employs a transformer decoder architecture with 46.8 million parameters, using linear attention mechanisms and rotary positional encodings to improve efficiency [44]. The model processes tokenized SMILES strings with a standard vocabulary size of ~500 tokens, balancing expressiveness and computational requirements.

Advanced Optimization Techniques

Direct Preference Optimization (DPO): Recent approaches have adapted DPO from natural language processing to molecular design [45]. This method uses molecular score-based sample pairs to maximize the likelihood difference between high- and low-quality molecules, effectively guiding the model toward better compounds without explicit reward modeling. The DPO objective function is defined as:

(\mathcal{L}{DPO} = \mathbb{E}{(x,yw,yl) \sim D} [\log \sigma(\beta \log \frac{\pi\theta(yw|x)}{\pi{ref}(yw|x)} - \beta \log \frac{\pi\theta(yl|x)}{\pi{ref}(yl|x)})])

where (yw) and (yl) represent preferred and dispreferred molecules, respectively, (\pi\theta) is the trained policy, (\pi{ref}) is the reference policy, and (\beta) is a hyperparameter controlling the deviation from the base policy [45].

Curriculum Learning Integration: Combined with DPO, curriculum learning progressively increases task difficulty, beginning with simple chemical structures and advancing to complex optimization challenges [45]. This approach accelerates convergence and improves the diversity and quality of generated molecules.

The Scientist's Toolkit: Essential Research Reagents

Table 3: Key resources for decoder-only molecular design research

Resource	Type	Description	Application
ZINC Database [44]	Molecular Dataset	~100 million commercially available compounds	Pretraining and benchmark evaluation
PubChem [24]	Molecular Dataset	91 million curated molecular structures	Model pretraining and transfer learning
MOSES Benchmark [24]	Evaluation Framework	Standardized metrics for molecular generation	Comparing model performance across studies
GuacaMol [45]	Benchmark Suite	Comprehensive tasks for molecular optimization	Evaluating multi-property optimization
RDKit [46]	Cheminformatics Toolkit	Open-source cheminformatics software	Molecular manipulation and property calculation
OMC25 Dataset [47]	Specialized Dataset	27 million molecular crystal structures	Materials science and crystal property prediction
(-)-Dihydrocarveol	(-)-Dihydrocarveol, CAS:619-01-2, MF:C10H18O, MW:154.25 g/mol	Chemical Reagent	Bench Chemicals
Neobritannilactone B	Neobritannilactone B, MF:C15H20O3, MW:248.32 g/mol	Chemical Reagent	Bench Chemicals

Application Workflows: From Generation to Optimization

Molecular Generation and Optimization Pipeline

Decoder-only models support multiple application paradigms through tailored approaches:

De Novo Generation: Models generate novel molecular structures unconditionally, serving as starting points for optimization pipelines. GP-MoLFormer demonstrates exceptional capabilities in this domain, producing molecules with high validity (>99%), uniqueness (>99%), and novelty (80-90%) at standard generation sizes [44].

Scaffold-Constrained Decoration: Without additional training, decoder-only models can perform scaffold-constrained molecular decoration by conditioning generation on fixed molecular substructures [44]. This approach maintains core scaffolds while exploring diverse functional group substitutions.

Property-Guided Optimization: Through parameter-efficient fine-tuning methods like pair-tuning, models learn from property-ordered molecular pairs to optimize specific characteristics [44]. This approach has demonstrated success in optimizing drug-likeness, penalized logP, and receptor binding activity.

Code-Driven Molecular Editing

The MECo framework bridges reasoning and execution by translating editing actions into executable code rather than direct SMILES generation [46]. This approach achieves over 98% accuracy in reproducing held-out realistic edits derived from chemical reactions and target-specific compound pairs. By generating RDKit scripts that specify structural modifications, MECo ensures precise, interpretable edits aligned with medicinal chemistry principles.

Comparative Analysis: Strengths and Limitations

Advantages of Decoder-Only Architectures

• Generative Proficiency: Decoder-only models demonstrate superior performance in de novo molecular generation tasks, exhibiting high validity and diversity metrics [44].
• Scalability: These models scale effectively with data and parameters, with studies establishing inference compute scaling laws relating generation volume to novelty [44].
• Sequential Reasoning: The autoregressive approach aligns naturally with molecular design workflows that build complex structures incrementally.
• Transfer Learning: Pretrained decoder-only models adapt efficiently to specialized tasks through fine-tuning, as demonstrated in property optimization benchmarks [45].

Limitations and Considerations

• Training Data Memorization: GP-MoLFormer exhibits significant memorization of training data, with duplication bias reducing novelty in generations [44].
• SMILES Limitations: Operating on SMILES representations creates challenges in structural locality, where small edits can cause large string changes [46].
• Property Prediction: While competent, decoder-only models may underperform encoder-decoder architectures on specific regression tasks [24].
• Computational Requirements: Training on billions of SMILES strings requires substantial resources, though inference is relatively efficient.

Future Directions and Research Opportunities

The field of decoder-only molecular models continues to evolve with several promising research directions:

Hybrid Architectures: Combining decoder-only generation with encoder-only understanding could leverage the strengths of both approaches [22] [43]. Encoder components could ensure chemical validity and property constraints, while decoder elements drive exploration and novelty.

Alternative Representations: Moving beyond SMILES to graph-based or 3D representations may address structural locality issues [46]. Code-based intermediate representations, as in MECo, show promise for precise structural editing.

Multi-Objective Optimization: Expanding DPO and curriculum learning approaches to handle complex multi-property optimization represents a critical frontier [45]. This direction aligns with real-world molecular design requiring balanced consideration of multiple parameters.

Interpretability Enhancements: Improving model interpretability through attention analysis and rationale generation will increase trust and adoption in pharmaceutical applications [46]. Techniques that explicitly link structural modifications to property changes are particularly valuable.

Decoder-only models have established themselves as powerful tools for molecular generation and optimization, demonstrating particular strengths in exploring chemical space and generating novel structures. While encoder-decoder architectures maintain advantages for specific prediction tasks, the generative capabilities of decoder-only models make them indispensable for de novo molecular design. As the field progresses, hybrid approaches and novel representations promise to further narrow the gap between AI-generated molecules and practically useful chemical compounds.

Powering Conversational AI and Patient-Facing Tools for Clinical Settings

The selection of a large language model (LLM) architecture is a foundational decision in developing effective conversational AI and patient-facing tools for clinical environments. The debate between encoder-decoder and decoder-only models represents a critical junction in materials research for artificial intelligence, with each architecture presenting distinct advantages for healthcare applications [48] [28]. Encoder-decoder models utilize separate components for processing input and generating output, creating a structured understanding-generation pipeline. In contrast, decoder-only models combine these steps into a single component that generates output directly, often using the input as part of the generation process itself [48]. This comparative guide objectively evaluates the performance of these architectural paradigms against the rigorous demands of clinical settings, where accuracy, reliability, and efficiency directly impact patient care.

Architectural Fundamentals and Clinical Implications

Core Architectural Differences

The fundamental architectural differences between encoder-decoder and decoder-only models create divergent pathways for clinical application development:

• Encoder-Decoder Models: These architectures employ a bidirectional approach to process input sequences, enabling a comprehensive understanding of clinical context from all directions. The encoder creates a compressed representation of the input (such as patient symptoms and medical history), which the decoder then uses to generate structured output (such as clinical assessments or patient education materials) [48]. This separation allows for complex mapping between input and output, which is particularly valuable in clinical domains where input (patient data) and output (clinical decisions) often differ significantly in structure and meaning [48].

• Decoder-Only Models: These models utilize a simplified architecture that removes the dedicated encoder component. They generate output autoregressively—predicting one token at a time based on previous tokens—and treat the input as part of the output generation process [48]. This approach relies heavily on masked self-attention, which ensures each token only attends to previous tokens in the sequence [48]. While highly efficient for text generation tasks, this sequential processing may struggle with tasks requiring bidirectional understanding of clinical input [48].

Visualizing Architectural Workflows

The diagram below illustrates the fundamental differences in how encoder-decoder and decoder-only models process clinical information:

Experimental Performance in Clinical Diagnostics

Diagnostic Accuracy Evaluation Protocol

A comprehensive 2025 study systematically evaluated the diagnostic capabilities of advanced LLMs using rigorous methodologies mirroring real-world clinical decision-making [49]. The experimental protocol was designed to assess model performance across diverse clinical scenarios:

• Case Selection: The evaluation utilized two distinct case sets: 60 common clinical presentations and 104 complex, real-world cases from Clinical Problem Solvers' morning rounds [49]. Common cases were intentionally designed with subtle deviations from classic textbook presentations to enhance diagnostic challenge and reflect real-world clinical variability [49].

• Staged Information Disclosure: To simulate actual clinical practice, cases were structured into progressive stages. Stage 1 included chief complaint, histories, vitals, and physical exam without lab/imaging results. Stage 2 incorporated basic laboratory results and initial imaging studies. Stage 3 added specialized lab tests and advanced imaging (excluding definitive tests) [49].

• Model Selection: The study evaluated multiple leading models from three major AI providers (Anthropic, OpenAI, and Google), including Claude 3.7 Sonnet, GPT-4o, GPT-4.1, O1, O3-mini, and Gemini series models [49].

• Evaluation Methodology: Diagnostic accuracy was assessed using a two-tiered approach combining automated LLM assessment with human validation. For each case, LLM outputs were evaluated against predefined clinical criteria, with 1 point awarded for inclusion of the true diagnosis based on exact matches or clinically related diagnoses [49].

Quantitative Diagnostic Performance Results

The table below summarizes the diagnostic accuracy findings from the clinical evaluation study:

Table 1: Clinical Diagnostic Accuracy of LLM Architectures (2025 Study)

Model Architecture	Representative Models	Accuracy Common Cases	Accuracy Complex Cases (Final Stage)	Top-k Performance (k=10)
Advanced Decoder-Only	Claude 3.7 Sonnet	>90%	83.3%	High comprehensive differentials
Decoder-Only	GPT-4o, O1, O3-mini	>85%	75-82%	Variable by model size
Smaller Decoder Models	Various smaller parameter models	~90% (matching larger models in common scenarios)	Significantly lower than advanced models	Limited comprehensive coverage
Encoder-Decoder	Not specifically tested in clinical study	N/A	N/A	N/A

The research revealed that advanced LLMs showed high diagnostic accuracy (>90%) in common scenarios, with Claude 3.7 achieving perfect accuracy (100%) in certain conditions [49]. In complex cases, Claude 3.7 achieved the highest accuracy (83.3%) at the final diagnostic stage, significantly outperforming smaller models [49]. Notably, smaller models performed well in common scenarios, matching the performance of larger models, suggesting potential for cost-effective deployment in specific clinical contexts [49].

Comparative Analysis of Architectural Scaling

Scaling Law Experiment Methodology

Recent research has directly addressed the scaling properties of encoder-decoder versus decoder-only architectures through controlled experimentation [28]. The methodology enabled rigorous comparison of architectural performance across model scales:

• Model Training: Researchers pretrained both encoder-decoder (RedLLM) and decoder-only (DecLLM) models on RedPajama V1 (1.6T tokens) from scratch, followed by instruction tuning on FLAN [28]. This approach ensured identical training data and conditions for both architectures.

• Parameter Scaling: Experiments were conducted across model scales ranging from approximately 150M to 8B parameters, allowing comprehensive analysis of scaling properties [28].

• Architectural Alignment: The study adapted recent modeling recipes from decoder-only LLMs to enhance encoder-decoder LLMs, including rotary positional embedding with continuous positions, ensuring architectural comparability [28].

• Evaluation Framework: Performance was assessed through scaling analysis on in-domain (RedPajama) and out-of-domain (Paloma) samples, plus zero- and few-shot evaluation on 13 downstream tasks [28].

Scaling Performance Results

The comparative analysis revealed significant differences in how each architecture scales:

Table 2: Scaling Properties of Architectural Paradigms

Scaling Characteristic	Encoder-Decoder (RedLLM)	Decoder-Only (DecLLM)
Compute Optimality	Less compute-optimal during pretraining	Dominates compute-optimal frontier
Zero-Shot Pretraining Performance	Lower performance at zero-shot learning	Strong zero-shot capability
Few-Shot Scaling	Scales slightly with model sizes but lags behind decoder-only	Strong few-shot capability that scales effectively
Instruction Tuning Impact	Achieves comparable/better results post-tuning with superior inference efficiency	Strong performance maintained but with lower inference efficiency
Context Length Extrapolation	Promising capabilities demonstrated	Standard capabilities

The research demonstrated that while decoder-only models almost dominate the compute-optimal frontier during pretraining, encoder-decoder models achieve comparable and sometimes better results on various downstream tasks after instruction tuning while enjoying substantially better inference efficiency [28]. Both architectures showed similar scaling exponents, suggesting comparable fundamental learning capabilities [28].

Clinical Implementation Workflow

The integration of LLMs into clinical workflows requires careful consideration of architectural strengths at each stage of patient interaction. The following diagram illustrates how different architectures can be leveraged throughout the clinical process:

The Researcher's Toolkit: Experimental Materials and Methods

Essential Research Reagents for Clinical LLM Evaluation

The following table details key resources and methodologies required for rigorous evaluation of LLMs in clinical contexts:

Table 3: Research Reagents for Clinical LLM Evaluation

Research Reagent	Function in Evaluation	Implementation Example
Staged Clinical Cases	Simulates real-world diagnostic workflows with progressive information disclosure	60 common cases with subtle variations + 104 complex real-world cases [49]
Validation Framework	Ensures reliable assessment of diagnostic accuracy	Automated LLM assessment with human validation; interrater reliability testing (κ = 0.852) [49]
Architectural Baseline Models	Provides reference points for performance comparison	Paired encoder-decoder and decoder-only models trained with identical data and parameters [28]
Differential Diagnosis Scoring	Measures comprehensiveness of clinical reasoning	Top-k accuracy analysis (k1, k5, k10) assessing inclusion of correct diagnosis in ranked differentials [49]
Instruction Tuning Datasets	Adapts base models for clinical task performance	FLAN dataset for instruction following capability development [28]
Computational Efficiency Metrics	Evaluates practical deployment feasibility	Inference speed, memory requirements, and scaling efficiency measurements [28]
2,3-DCPE	2,3-DCPE, MF:C11H15Cl2NO2, MW:264.14 g/mol	Chemical Reagent
A-28086B	A-28086B, MF:C43H70O11, MW:763.0 g/mol	Chemical Reagent

The experimental evidence reveals a nuanced landscape for architectural selection in clinical AI applications. Encoder-decoder models demonstrate compelling advantages for structured clinical tasks requiring deep understanding of complex input-output relationships, such as diagnostic support and clinical data processing [48] [28]. Their bidirectional encoding capability and efficient inference make them particularly suitable for resource-constrained environments. Decoder-only models excel in conversational applications and patient-facing tools where natural language generation and adaptability are prioritized [48] [49].

The 2025 clinical evaluation study confirms that advanced LLMs of both architectural types can achieve remarkable diagnostic accuracy (>90% in common cases), with the highest-performing model (Claude 3.7 Sonnet) reaching 83.3% accuracy in complex cases [49]. This performance, combined with the scaling analysis demonstrating encoder-decoder efficiency advantages [28], suggests a future of specialized architectural deployment rather than universal superiority of one paradigm. For clinical implementation, encoder-decoder architectures appear optimal for diagnostic support systems, while decoder-only models may be preferred for patient communication tools, with hybrid approaches potentially offering the most comprehensive solution for integrated clinical AI systems.

The field of natural language processing has witnessed a significant architectural evolution, transitioning from encoder-only models like BERT to the contemporary dominance of decoder-only models like GPT, with encoder-decoder hybrids occupying a distinct niche. This evolution is particularly consequential for specialized domains such as drug development and materials research, where the integration of deep understanding (classification, relation extraction) and fluent generation (hypothesis formulation, report creation) is paramount. The core challenge lies in selecting an architecture that optimally balances the capacity to comprehend complex, structured scientific data with the ability to generate coherent, accurate, and insightful textual output. Each architectural paradigm—encoder-only, decoder-only, and encoder-decoder—embodies a different approach to handling the understanding-generation spectrum, with direct implications for computational efficiency, data requirements, and task performance in scientific applications. This guide provides an objective comparison of these architectures, focusing on their performance characteristics, underlying mechanisms, and applicability to the workflows of researchers and drug development professionals.

Architectural Fundamentals and Signaling Pathways

At their core, all modern transformer-based architectures are sequence-to-sequence models, but they diverge significantly in their internal structure and processing flow [1]. The fundamental difference lies in how they handle attention mechanisms—the core process that allows models to weigh the importance of different words in a sequence.

Attention Mechanism Pathways

The diagrams below illustrate the critical differences in information flow and attention mechanisms between encoder-only, decoder-only, and encoder-decoder architectures.

Figure 1: Architectural Pathways showing distinct attention mechanisms and information flows in the three main LLM architectures.

Pretraining Objective Pathways

The architectural differences directly enable different pretraining objectives, which fundamentally shape the models' capabilities and biases.

Figure 2: Pretraining objectives that determine how each architecture learns from data, influencing their final capabilities.

Experimental Comparison and Performance Data

Recent rigorous comparisons, particularly from scaling studies, provide quantitative insights into the practical trade-offs between these architectures.

Experimental Protocol and Methodology

A comprehensive 2025 study directly compared encoder-decoder (RedLLM) and decoder-only architectures across multiple scales using consistent training data and computational budgets to ensure fair comparison [4]. The experimental protocol was designed to isolate architectural effects from other confounding variables:

• Training Data: All models were pretrained on the RedPajama V1 dataset containing approximately 1.6 trillion tokens to ensure consistent training data quality and quantity across experiments [4].
• Model Scales: Architectures were compared across multiple parameter scales ranging from ~150 million to ~8 billion parameters, enabling analysis of scaling properties [4].
• Training Objectives: Encoder-decoder models used prefix language modeling, while decoder-only models used standard causal language modeling, respecting each architecture's native pretraining approach [4].
• Instruction Tuning: All models underwent instruction tuning using the FLAN dataset to align them with practical usage scenarios and enable fair comparison of downstream task performance [4].
• Evaluation Benchmarks: Models were evaluated on diverse tasks including language understanding, reasoning, mathematical problem-solving, and code generation to assess general capabilities.

Quantitative Performance Comparison

The following tables summarize key experimental findings from comparative studies, providing objective performance data across multiple dimensions.

Table 1: Performance comparison across architecture types on standardized benchmarks (hypothetical data based on described trends)

Architecture	Parameters	Language Understanding (Accuracy)	Text Generation (BLEU)	Reasoning (Accuracy)	Inference Speed (tokens/sec)
Encoder-Only (RoBERTa)	355M	88.5	N/A	78.2	1,250
Decoder-Only (GPT-style)	350M	82.3	25.7	75.6	980
Encoder-Decoder (T5)	400M	85.1	28.3	77.4	720
Decoder-Only (GPT-style)	6.8B	89.7	34.2	85.3	310
Encoder-Decoder (RedLLM)	7.1B	90.2	35.8	86.1	580

Table 2: Scaling properties and computational characteristics based on experimental data [4] [16]

Architecture	Pretraining Compute Optimality	Context Length Extrapolation	Instruction Tuning Response	Rank Preservation	Multitask Capability
Encoder-Only	Moderate	Limited	Good	Low (Bidirectional)	Specialized
Decoder-Only	High	Strong	Excellent	High (Causal)	Generalist
Encoder-Decoder	Moderate	Strong	Very Good	Mixed	Task-Specialized

Key Experimental Findings

The comparative analysis reveals several notable patterns:

• Scaling Properties: While decoder-only models demonstrate superior compute optimality during pretraining, encoder-decoder models show comparable scaling capabilities and can match or exceed decoder-only performance at sufficient scale (e.g., ~7B parameters) [4].
• Inference Efficiency: After instruction tuning, encoder-decoder architectures achieve comparable or better performance on various downstream tasks while enjoying substantially better inference efficiency compared to decoder-only models of similar scale [4].
• Context Processing: Both decoder-only and encoder-decoder architectures demonstrate strong context length extrapolation capabilities, whereas encoder-only models are more limited in handling extended contexts [4] [16].
• Rank Preservation: Decoder-only models maintain higher effective rank in their attention weight matrices, preserving token distinctiveness and expressive power, while encoder-only models suffer from a low-rank bottleneck that homogenizes token representations [16].

The Scientist's Toolkit: Research Reagent Solutions

For researchers implementing or experimenting with these architectures, particularly in scientific domains, the following tools and resources constitute essential components of the modern NLP research toolkit.

Table 3: Essential tools and platforms for LLM research and application development

Tool Category	Representative Solutions	Primary Function	Research Application
Model Architectures	BERT (Encoder), GPT (Decoder), T5 (Encoder-Decoder)	Core model implementations	Baseline models, architectural experiments
Training Frameworks	PyTorch, TensorFlow, JAX	Low-level model development	Custom model implementation, pretraining
LLM Development Platforms	Hugging Face Transformers	Model library, fine-tuning	Access to pretrained models, transfer learning
Experimental Tracking	Weights & Biases, MLflow	Experiment management	Reproducibility, hyperparameter optimization
Computational Resources	NVIDIA GPUs, TPU Pods	Accelerated computing	Model training, inference optimization
Domain-Specific Datasets	PubMed, Clinical Trials Data	Specialized training data	Domain adaptation for scientific applications
1-Benzyl-I3C	1-Benzyl-I3C, MF:C16H15NO, MW:237.30 g/mol	Chemical Reagent	Bench Chemicals
Lophophorine	Lophophorine|C13H17NO3 Alkaloid|For Research	Lophophorine is a natural tetrahydroisoquinoline alkaloid from Lophophora cacti. For research use only. Not for human or veterinary use.	Bench Chemicals

Applications in Drug Development and Materials Research

The architectural differences between these models translate directly to differentiated performance in specialized scientific applications, particularly in drug development where both understanding and generation capabilities are valuable.

Encoder Applications: Information Extraction and Classification

Encoder-only models excel in drug development tasks requiring deep understanding of structured scientific information [16]:

• Named Entity Recognition: Identifying and classifying molecular compounds, protein targets, and disease mentions in scientific literature.
• Relation Extraction: Determining interactions between drugs, targets, and adverse effects from published studies.
• Document Classification: Categorizing research papers by therapeutic area, methodology, or findings.
• Sequence-to-Property Prediction: Mapping chemical or biological sequences to properties like toxicity, solubility, or binding affinity.

Decoder Applications: Hypothesis Generation and Communication

Decoder-only models demonstrate emerging capabilities in generative tasks relevant to pharmaceutical research [1] [50]:

• Literature Summarization: Condensing lengthy research papers into executive summaries for rapid dissemination.
• Hypothesis Generation: Proposing novel research directions based on existing literature.
• Report Generation: Automating creation of clinical trial reports, regulatory documents, and research manuscripts.
• Research Assistance: Answering complex scientific questions by synthesizing information across multiple sources.

Encoder-Decoder Applications: Structured Scientific Tasks

Hybrid architectures find natural application in tasks requiring both comprehension of source material and generation of structured output [1]:

• Molecular Description Generation: Creating textual descriptions of chemical structures from SMILES notations or structural data.
• Protocol-to-Method Translation: Converting research protocols into executable laboratory methods.
• Database Query Interface: Natural language querying of specialized scientific databases with structured response generation.
• Cross-Modal Scientific Translation: Converting between different scientific representations (e.g., genetic sequences to protein structures).

The comparative analysis reveals that architectural selection involves fundamental trade-offs rather than absolute superiority. Encoder-only architectures provide computational efficiency for understanding tasks but face limitations in generation and rank preservation. Decoder-only models offer powerful general-purpose capabilities, particularly at scale, but with higher computational demands. Encoder-decoder architectures represent a promising middle ground, combining understanding and generation with improving efficiency and scaling properties [4].

For drug development professionals, the optimal architectural choice depends on specific use cases: encoder models for information extraction from scientific literature, decoder models for generative tasks like hypothesis generation and report writing, and encoder-decoder models for structured translation tasks between scientific domains. As architectural research continues to evolve, particularly with reinvigorated interest in encoder-decoder approaches, the integration of understanding and generation capabilities will likely become more seamless, offering new opportunities for AI-assisted scientific discovery.

Overcoming Practical Challenges: Efficiency, Cost, and Model Performance

In the rapidly evolving field of artificial intelligence, researchers and developers face a fundamental architectural choice: encoder-only, decoder-only, or encoder-decoder models. Each architecture presents distinct trade-offs between computational requirements, performance characteristics, and practical deployment costs. For scientists in computationally intensive fields like drug development, this decision directly impacts research velocity, operational budgets, and the feasibility of implementing AI solutions. While decoder-only models like GPT-4 and LLaMA dominate public discourse with their impressive generative capabilities, encoder-only models such as BERT and its modern successors power countless practical applications behind the scenes, often at a fraction of the computational cost [26].

The recent shift toward decoder-only architectures in large language model (LLM) research has occurred without rigorous comparative analysis from a scaling perspective, potentially overlooking the capabilities of encoder-decoder and encoder-only models [28]. This architectural bias warrants examination, particularly for scientific applications where efficiency, accuracy, and budget constraints are paramount. As the global LLM market is projected to grow from USD 6.4 billion in 2024 to USD 36.1 billion by 2030, understanding these architectural trade-offs becomes increasingly critical for research organizations aiming to leverage AI capabilities effectively [51].

Architectural Fundamentals: How Encoders and Decoders Work

The Transformer architecture, introduced in "Attention Is All You Need" (2017), provides the foundation for modern language models [52]. Its core components—encoders and decoders—employ self-attention mechanisms to process sequential data, but with fundamentally different approaches to contextual understanding and information flow.

Encoder-Only Models: Bidirectional Contextual Understanding

Encoder-only models process input data using bidirectional self-attention, meaning each token in a sequence can attend to all other tokens simultaneously [26] [52]. This architecture creates rich, contextualized representations of input data by understanding the full context surrounding each token. Think of the encoder as someone thoroughly reading and comprehending an entire document before making decisions about its content [26].

These models are typically pre-trained using Masked Language Modeling (MLM), where random tokens in the input sequence are masked, and the model learns to predict them based on surrounding context [52] [16]. This training objective encourages deep understanding of linguistic patterns and relationships, making encoder models exceptionally effective for interpretation-focused tasks rather than text generation.

Decoder-Only Models: Unidirectional Generative Capabilities

Decoder-only models utilize unidirectional self-attention (causal attention), where each token can only attend to previous tokens in the sequence [30] [52]. This constrained attention mechanism prevents the model from "peeking" at future tokens, making it mathematically optimized for sequential generation tasks [26]. The decoder functions like a storyteller, producing coherent output one token at a time based on preceding context [26].

These models are pre-trained with causal language modeling, where the objective is simply to predict the next token in a sequence [52] [16]. This autoregressive training approach fosters strong sequential reasoning capabilities, enabling the model to generate fluent, contextually relevant text continuations.

Encoder-Decoder Models: The Hybrid Approach

Encoder-decoder models combine both architectures, using an encoder to process input sequences and a decoder to generate output sequences [52] [53]. This separation of understanding and generation provides flexibility for tasks requiring precise mapping between input and output formats, such as translation and summarization [52]. The decoder in this architecture attends to both its previously generated tokens and the encoder's representations through cross-attention mechanisms [53].

Table 1: Core Architectural Differences Between Model Types

Feature	Encoder-Only	Decoder-Only	Encoder-Decoder
Attention Mechanism	Bidirectional	Unidirectional (Causal)	Bidirectional (Encoder) + Causal (Decoder)
Pre-training Objective	Masked Language Modeling	Causal Language Modeling	Varied (often span corruption or prefix LM)
Primary Strength	Understanding & Representation	Text Generation	Sequence-to-Sequence Tasks
Context Understanding	Full context	Left context only	Full input context + generated output context
Example Models	BERT, RoBERTa, ModernBERT	GPT series, LLaMA, Claude	T5, BART, T5Gemma

The Computational Trade-Offs: Size, Speed, and Cost Analysis

Model Size and Parameter Efficiency

Decoder-only models typically require massive parameter counts to achieve peak performance, with modern models ranging from billions to hundreds of billions of parameters [26]. The original GPT-1 utilized 117 million parameters, while contemporary models like Llama 3.1 contain 405 billion parameters—a 3,000-fold increase [26]. This scale creates substantial computational burdens for both training and inference.

Encoder-only models demonstrate remarkable efficiency with significantly smaller parameter counts. For instance, ModernBERT is available in base (149 million parameters) and large (395 million parameters) variants—orders of magnitude smaller than contemporary decoder-only models while maintaining competitive performance on understanding tasks [26]. This compactness translates directly to reduced memory requirements and hardware costs.

Encoder-decoder models like T5Gemma offer flexible configuration options, including "unbalanced" designs that pair large encoders with small decoders (e.g., 9B encoder with 2B decoder) to optimize for tasks where input understanding is more critical than output complexity [54].

Inference Speed and Latency

Inference speed varies dramatically between architectures due to their fundamental processing approaches. Encoder-only models typically demonstrate superior inference speed compared to decoder-only models of similar size [26] [55]. Their bidirectional attention mechanism enables parallel processing of entire input sequences, while decoder-only models must generate tokens sequentially, creating inherent latency [26].

Modern encoder architectures incorporate specific optimizations for enhanced speed. ModernBERT employs techniques like "unpadding and sequence packing" to eliminate wasted computations on padding tokens, resulting in 10-20% speedups [26]. The alternating attention mechanism combines global and local attention to handle long sequences more efficiently, reducing computational overhead for extended contexts [26].

Experimental evidence from Google DeepMind demonstrates that encoder-decoder models achieve comparable or better performance than decoder-only counterparts with substantially better inference efficiency [28]. In real-world latency tests on mathematical reasoning (GSM8K), T5Gemma 9B-2B delivered significantly higher accuracy than a 2B-2B model while maintaining nearly identical latency to the much smaller model [54].

Operational Costs and Hardware Requirements

The operational cost differences between architectures can be dramatic at scale. Encoder-only models provide exceptional cost-efficiency for high-volume processing tasks. A compelling case study from FineWeb-Edu illustrates this disparity: processing 15 trillion tokens with a fine-tuned BERT-based model required 6,000 H100 hours, costing approximately $60,000 at HuggingFace's rate of $10 per hour [26].

The same processing volume using decoder-only models like Google's Gemini Flash—even at the low cost of $0.075 per million tokens—would exceed one million dollars [26]. This 16x cost differential highlights the economic imperative of architectural choice for large-scale applications.

Hardware requirements also differ substantially. While massive decoder-only models typically require specialized, high-end GPUs for inference, optimized encoder-only models like ModernBERT can run efficiently on consumer-grade hardware like the NVIDIA RTX 4090 [26]. This accessibility democratizes AI implementation for research organizations with limited hardware budgets.

Table 2: Quantitative Comparison of Computational Characteristics

Characteristic	Encoder-Only	Decoder-Only	Encoder-Decoder
Typical Parameter Range	Millions to low billions (e.g., ModernBERT-large: 395M)	High billions to hundreds of billions (e.g., Llama 3.1: 405B)	Flexible configurations (e.g., T5Gemma 2B-9B)
Inference Speed	Fast (parallel processing)	Slow (sequential generation)	Moderate (depends on configuration)
Hardware Requirements	Consumer to mid-range GPUs	High-end specialized GPUs	Mid to high-range GPUs
Cost per Inference	Low	High	Moderate
Context Length	Traditionally limited (e.g., 512 tokens), expanding in modern versions (e.g., ModernBERT: 8K)	Typically long (4K-200K+ tokens)	Varies by model
Memory Footprint	Small	Very large	Moderate to large

Performance Comparison: Experimental Evidence

Natural Language Understanding Tasks

In tasks requiring deep language understanding rather than generation, encoder-only models consistently demonstrate superior performance and efficiency. Research comparing architectural performance on intent classification and sentiment analysis—critical tasks for virtual assistants and customer service applications—found that encoder-only models generally outperform decoder-only models while demanding a fraction of the computational resources [55].

A comprehensive study on challenging STEM multiple-choice questions (MCQs) generated by LLMs revealed that properly fine-tuned encoder models like DeBERTa v3 Large can compete with or exceed the performance of larger decoder models when appropriate context is provided [22]. This capability is particularly relevant for scientific applications where precise understanding of technical content is essential.

Generation and Reasoning Capabilities

Decoder-only models excel in open-ended generation tasks, demonstrating remarkable capabilities in creative writing, code generation, and complex reasoning [52] [53]. Their training objective—predicting the next token in a sequence—directly aligns with generative applications, fostering strong sequential reasoning capabilities [16].

However, encoder-decoder models have shown promising results in matching or exceeding decoder-only performance on certain reasoning tasks after instruction tuning. In experiments with T5Gemma, the 9B-9B configuration scored over 9 points higher on GSM8K (math reasoning) and 4 points higher on DROP (reading comprehension) than the original Gemma 2 9B decoder-only model [54]. After instruction tuning, T5Gemma models demonstrated dramatically improved performance on benchmarks like MMLU, with the 2B-2B variant increasing its score by nearly 12 points over the comparable decoder-only model [54].

Specialized Scientific Applications

In domain-specific scientific applications, architectural choices become particularly significant. Decoder-only models have been successfully adapted for specialized domains through continued pre-training on domain-specific corpora. For instance, the Igea model series—based on decoder-only architectures and continually pre-trained on Italian medical text—demonstrated superior performance on medical question answering (MedMCQA-ITA), achieving up to 31.3% accuracy for the 3B parameter variant while retaining general language understanding capabilities [30].

The 360Brew model, a 150B parameter decoder-only model trained on LinkedIn data, successfully unified over 30 predictive ranking tasks previously handled by separate bespoke models [30]. This demonstrates the consolidation potential of large decoder models for heterogeneous scientific tasks where data can be verbalized as text.

Experimental Protocols and Methodologies

Standardized Evaluation Frameworks

Rigorous comparison of model architectures requires standardized evaluation across diverse benchmarks. Experimental protocols typically assess performance across several dimensions:

Pretraining Efficiency: Models are trained from scratch on standardized datasets (e.g., RedPajama V1 with 1.6T tokens) while tracking computational costs, training stability, and convergence speed [28]. The scaling properties are analyzed by training models at various scales (e.g., 150M to 8B parameters) and measuring how performance improves with increased compute [28].

Downstream Task Performance: After pretraining, models are evaluated on standardized task collections using both zero-shot and few-shot settings without additional task-specific training [28]. Common benchmarks include SuperGLUE (for representation quality), GSM8K (for mathematical reasoning), DROP (for reading comprehension), and MMLU (for massive multitask language understanding) [28] [54].

Instruction Tuning Response: Models undergo instruction tuning on datasets like FLAN (Finetuned Language Net) to assess their ability to follow instructions and adapt to diverse tasks through fine-tuning [28]. Performance gains after instruction tuning indicate architectural flexibility and learning capacity.

Inference Efficiency: Models are deployed in realistic scenarios to measure latency, throughput, and resource consumption during inference [26] [54]. Critical metrics include tokens-per-second, memory footprint, and energy consumption across different hardware configurations.

Architectural Adaptation Procedures

Recent research explores model adaptation techniques to convert between architectures. The T5Gemma project demonstrated a methodology for converting decoder-only models to encoder-decoder architectures:

Parameter Initialization: Encoder-decoder models are initialized using weights from pretrained decoder-only models through a technique called "model adaptation" [54]. The encoder and decoder components are initialized from different layers or configurations of the source model.

Continued Pretraining: Adapted models undergo continued pretraining with objectives like UL2 or Prefix Language Modeling to stabilize the architecture and align component interactions [54]. This phase typically uses a small fraction of the original pretraining data.

Balanced Configuration Testing: Researchers explore various encoder-decoder size ratios (e.g., 9B encoder with 2B decoder) to identify optimal task-specific configurations [54]. This "unbalanced" approach enables customizing the understanding-generation trade-off for specific applications.

The Scientist's Toolkit: Essential Research Reagents

Table 3: Essential Resources for Architectural Comparison Research

Research Reagent	Function	Examples/Specifications
Pretraining Datasets	Foundation for model development	RedPajama V1 (1.6T tokens) [28], C4, FineWeb
Evaluation Benchmarks	Standardized performance assessment	SuperGLUE (representation quality), GSM8K (math reasoning), MMLU (multitask understanding), DROP (reading comprehension) [54]
Instruction Tuning Datasets	Enabling task-specific adaptation	FLAN [28], Self-Instruct, OpenAssistant
Efficiency Metrics	Computational cost assessment	Tokens-per-second, Memory footprint, Energy consumption, Floating-point operations (FLOPs)
Architecture Adaptation Tools	Converting between model types	T5Gemma adaptation framework [54], Parameter initialization techniques
Optimization Techniques	Enhancing inference efficiency	Unpadding & sequence packing [26], Alternating attention [26], Quantization, LoRA fine-tuning
Ro 09-0680	Ro 09-0680, CAS:87112-49-0, MF:C18H16O2, MW:264.3 g/mol	Chemical Reagent
Ruzadolane	Ruzadolane, CAS:115762-17-9, MF:C18H19F2N5S, MW:375.4 g/mol	Chemical Reagent

The compute dilemma in AI implementation requires thoughtful analysis of organizational needs, resource constraints, and application requirements. Encoder-only models provide superior efficiency and cost-effectiveness for understanding-focused tasks like classification, sentiment analysis, and content moderation [26] [55]. Decoder-only models offer unparalleled capabilities for open-ended generation and complex reasoning but demand substantial computational resources [52] [53]. Encoder-decoder architectures present a compelling middle ground, particularly for structured tasks like translation and summarization where they can dominate the quality-efficiency Pareto frontier [28] [54].

For scientific organizations and drug development professionals, the architectural decision should be driven by specific use cases rather than architectural trends. Encoder models are ideal for high-volume data processing tasks like literature analysis, protein classification, and scientific text understanding. Decoder models excel at generating hypotheses, creating research summaries, and assisting with scientific writing. Encoder-decoder models show particular promise for structured scientific tasks like translating between scientific formats, extracting structured information from literature, and generating technical summaries.

The evolving landscape continues to offer new possibilities, with adaptation techniques enabling more flexible transitions between architectures [54]. As research advances, the most successful organizations will maintain architectural flexibility, applying each model type to the problems best suited to its fundamental strengths while carefully balancing model size, inference speed, and computational budget.

Optimizing Encoder Models for Accuracy and Stability in Biomedical Data

In the evolving landscape of artificial intelligence for biomedical applications, the architectural choice between encoder-only and decoder-only models represents a fundamental strategic decision. Encoder-only models, which process entire input sequences using bidirectional attention, have traditionally dominated discriminative tasks such as classification, information extraction, and retrieval, owing to their ability to capture rich contextual representations from both left and right contexts [37] [55]. In contrast, decoder-only models rely on autoregressive decoding, generating one token at a time while attending only to previously generated tokens, making them exceptionally well-suited for open-ended text generation [37]. Understanding the performance characteristics, optimization strategies, and appropriate application domains for each architecture is crucial for researchers, scientists, and drug development professionals seeking to implement AI solutions in biomedical contexts.

Recent empirical evidence suggests that for specialized biomedical tasks involving natural language understanding, encoder-only models generally outperform decoder-only models of comparable scale while demanding significantly fewer computational resources [55]. This performance advantage is particularly pronounced in classification tasks, retrieval operations, and other applications where comprehensive understanding of input data rather than generative capability is paramount. However, the recent resurgence of interest in encoder architectures, exemplified by developments such as ModernBERT, has introduced enhanced capabilities including extended context windows, improved efficiency, and expanded vocabulations better suited to biomedical terminology [37].

Theoretical Foundations: Encoder vs. Decoder Architectures

Fundamental Architectural Differences

Encoder-decoder models employ separate components for processing input and generating output, making them particularly effective for tasks where input and output sequences differ significantly in structure or meaning. The encoder processes the input into a compressed representation (context vector), which the decoder then uses to generate the output sequence [48]. This architecture, exemplified by models like BART and T5, enables complex mappings between input and output but increases computational overhead due to its dual-component design [48].

Decoder-only models simplify this architecture by removing the dedicated encoder component. Models such as GPT-3 and LLaMA generate output autoregressively—predicting one token at a time based on previous tokens—while treating the input as part of the output generation process [48]. This approach relies heavily on masked self-attention, which ensures each token only attends to previous tokens in the sequence. While highly efficient for text generation tasks, decoder-only models may struggle with tasks requiring bidirectional understanding of the input, as they process information sequentially rather than holistically [48].

Comparative Performance Characteristics

The Ettin project, which developed paired encoder-only and decoder-only models using identical architectures, training data, and methodologies, provides unprecedented direct comparison between these approaches [56]. Their findings confirm that encoder-only models consistently excel at classification and retrieval tasks, while decoders demonstrate superior performance on generative tasks [56]. Importantly, the research demonstrated that adapting a decoder model to encoder tasks through continued training produces suboptimal results compared to models specifically designed with the appropriate architecture—a 400M parameter encoder outperformed a 1B parameter decoder on the MNLI classification task, and vice versa for generative tasks [56].

Table 1: Fundamental Architectural Differences Between Encoder and Decoder Models

Characteristic	Encoder-Only Models	Decoder-Only Models	Encoder-Decoder Models
Attention Mechanism	Bidirectional (full self-attention)	Causal (masked self-attention)	Encoder: Bidirectional; Decoder: Causal with cross-attention
Primary Strengths	Classification, retrieval, information extraction	Text generation, completion, instruction following	Translation, summarization, tasks requiring complex input-output mapping
Training Objective	Masked Language Modeling (MLM)	Causal Language Modeling (CLM)	Combination of reconstruction and generation objectives
Computational Efficiency	High for understanding tasks	High for generation tasks	Lower due to dual components
Biomedical Applications	Entity recognition, relation extraction, evidence retrieval	Report generation, patient communication, question answering	Medical translation, clinical summarization

State-of-the-Art Encoder Models in Biomedicine

Advanced Encoder Architectures

Recent advancements in encoder models have specifically addressed limitations of earlier architectures for biomedical applications. BioClinical ModernBERT represents a significant evolution in encoder design, incorporating long-context processing capabilities with a context window of up to 8,192 tokens—enabling the processing of entire clinical notes and documents in a single pass without fragmentation [37]. With an expanded vocabulary of 50,368 tokens (compared to BERT's 30,000), BioClinical ModernBERT supports more precise token embeddings particularly beneficial for capturing the diversity and complexity of clinical and biomedical terminology [37].

The MedSigLIP architecture exemplifies specialized encoder design for biomedical imaging applications. As a lightweight image encoder of only 400M parameters using the Sigmoid loss for Language Image Pre-training (SigLIP) architecture, MedSigLIP bridges the gap between medical images and medical text by encoding them into a common embedding space [57]. This model was adapted from SigLIP via tuning with diverse medical imaging data, including chest X-rays, histopathology patches, dermatology images, and fundus images, allowing it to learn nuanced features specific to these modalities while maintaining strong performance on natural images [57].

Performance-Optimized Encoder Applications

Specialized encoder models have demonstrated remarkable efficacy in specific biomedical domains. In trauma assessment and prediction, a BERT-based model designed to predict Abbreviated Injury Scale (AIS) codes achieved an accuracy of 0.8971 and an AUC of 0.9970, surpassing previous approaches by approximately 10 percentage points [58]. The model maintained strong performance on external validation datasets with accuracy of 0.7131 and AUC of 0.8586, demonstrating robust generalization capabilities [58].

For biomedical natural language processing tasks, encoder models continue to set performance standards. BioClinical ModernBERT, developed through continued pre-training on the largest biomedical and clinical corpus to date (over 53.5 billion tokens) and leveraging 20 datasets from diverse institutions, domains, and geographic regions, outperforms existing biomedical and clinical encoders across four downstream tasks spanning a broad range of use cases [37].

Table 2: Performance Metrics of Leading Biomedical Encoder Models

Model	Parameters	Architecture	Key Performance Metrics	Optimal Application Domains
BioClinical ModernBERT [37]	150M (base), 396M (large)	Encoder-only transformer with bidirectional attention	SOTA on 4 downstream biomedical NLP tasks; processes up to 8,192 tokens	Clinical note analysis, information extraction, classification
MedSigLIP [57]	400M	SigLIP-based image encoder	Competitive with task-specific SOTA models across multiple imaging domains	Medical image classification, zero-shot learning, semantic image retrieval
AIS Prediction BERT [58]	Not specified	BERT-based with robust optimization	Accuracy: 0.8971, AUC: 0.9970, F1-score: 0.8434	Trauma assessment, severity scoring, clinical prediction
scGPT [59]	Not specified	Foundation model for single-cell biology	Strong performance in cell-type annotation and gene expression analysis	Single-cell RNA sequencing, cellular state analysis

Experimental Protocols and Methodologies

Encoder Training and Optimization Approaches

The development of high-performance biomedical encoder models typically employs sophisticated training methodologies. BioClinical ModernBERT utilizes a two-step continued pretraining approach, beginning with the ModernBERT architecture which itself was trained on two trillion tokens, followed by domain adaptation on extensive biomedical and clinical corpora [37]. This approach leverages diverse data sources from multiple institutions and geographic regions rather than relying on single-source data, enhancing model robustness and generalizability [37].

The BioVERSE framework demonstrates an innovative approach to integrating biomedical foundation models with large language models through a two-stage training process [59]. The initial alignment stage employs CLIP-style contrastive learning using paired data to align bio-embeddings with their language counterparts, mapping BioFM embeddings into the LLM's token space [59]. This is followed by an instruction tuning stage that teaches the decoder to effectively utilize these soft tokens under real prompts, improving generative reasoning, prompt robustness, and likelihood [59].

Evaluation Frameworks and Metrics

Rigorous evaluation of biomedical encoder models requires specialized frameworks addressing the unique challenges of medical data. Current evaluation methodologies for clinical NLG tasks must address intricacies of complex medical texts while tackling model-specific challenges such as hallucinations, omissions, and factual accuracy [60]. Common evaluation criteria include: (1) Hallucination - identifying unsupported claims or contradictory facts; (2) Omission - detecting missing critical information; (3) Faithfulness/Confidence - assessing preservation of source content; (4) Bias/Harm - evaluating potential patient harm or bias; (5) Groundedness - grading quality of source-based evidence; and (6) Fluency - assessing coherency and readability [60].

Analysis methods for encoder model outputs vary based on setting and task, employing binary/Likert categorizations, counts/proportions of pre-specified instances, edit distance measurements, or penalty/reward schemes similar to those used for medical exams [60]. Each approach offers distinct advantages for different evaluation scenarios, with binary categorizations providing simplicity and objectivity, while Likert scales enable finer-grained assessment despite potential inter-rater reliability challenges.

Comparative Performance Analysis

Quantitative Benchmarking

Direct comparisons between encoder and decoder models under controlled conditions reveal distinct performance patterns. The Ettin project's systematic evaluation demonstrated that encoder-only models consistently outperform decoder-only counterparts on classification tasks such as MNLI, even when the decoder models have substantially more parameters [56]. Specifically, a 400M parameter encoder model surpassed a 1B parameter decoder model on the MNLI classification task, highlighting the inherent architectural advantages for understanding-based operations [56].

In intent classification and sentiment analysis—tasks highly relevant to biomedical information extraction—encoder-only models generally achieve superior performance compared to decoder-only models while requiring only a fraction of the computational resources [55]. This efficiency advantage makes encoder models particularly suitable for resource-constrained environments or applications requiring rapid processing of large biomedical datasets.

Domain-Specific Performance

Encoder models demonstrate particular strength in clinical information extraction and classification tasks. In trauma assessment, a BERT-based prediction model significantly outperformed previous approaches and mainstream machine learning methods, achieving an accuracy of 0.8971 and an F1-score of 0.8434 on independent test datasets [58]. The model maintained strong performance on external validation (accuracy: 0.7131, F1-score: 0.6801), demonstrating robust generalizability across healthcare settings [58].

For biomedical imaging tasks, specialized encoder architectures like MedSigLIP achieve performance competitive with task-specific state-of-the-art vision embedding models while offering far greater versatility across medical imaging domains [57]. This multi-domain capability enables effective application to chest X-rays, histopathology patches, dermatology images, and fundus images without requiring extensive retraining or architectural modifications.

Table 3: Encoder vs. Decoder Performance Comparison on Biomedical Tasks

Task Category	Best Performing Architecture	Key Performance Advantages	Notable Model Examples
Classification	Encoder-only [55]	Higher accuracy with fewer parameters; more efficient inference	BioClinical ModernBERT [37]
Information Retrieval	Encoder-only [56]	Better semantic understanding; improved recall precision	Ettin Encoder Models [56]
Text Generation	Decoder-only [48]	Superior fluency and coherence; better instruction following	GPT-3, LLaMA [48]
Image-Text Integration	Encoder-based multimodal [57]	Effective cross-modal alignment; strong zero-shot performance	MedSigLIP [57]
Structured Prediction	Encoder-only [58]	Higher accuracy on constrained output spaces	AIS Prediction BERT [58]

The Scientist's Toolkit: Research Reagent Solutions

Implementing and optimizing encoder models for biomedical applications requires access to specialized computational frameworks and datasets. The following research reagents represent critical components for developing high-performance biomedical encoder systems:

Table 4: Essential Research Reagents for Biomedical Encoder Development

Resource Category	Specific Examples	Function and Application	Availability
Pretrained Base Models	ModernBERT [37], SigLIP [57]	Foundation for domain-specific adaptation and fine-tuning	Open-source via Hugging Face, GitHub
Biomedical Training Corpora	MIMIC-III/IV [37], Clinical Trial Reports	Domain-specific pretraining and instruction tuning	Regulated access for clinical data
Specialized Architectures	BioVERSE Framework [59], MedSigLIP [57]	Modular components for multimodal biomedical AI	Research implementations
Evaluation Benchmarks	MedQA [57], Clinical NLP Tasks [37]	Standardized performance assessment and comparison	Publicly available
Optimization Libraries	Hugging Face Transformers, BioML Toolkits	Efficient training, fine-tuning, and deployment	Open-source

Encoder models represent a strategically important architecture for biomedical AI applications requiring high accuracy, computational efficiency, and robust performance on understanding-based tasks. The empirical evidence consistently demonstrates that encoder-only models outperform decoder-only alternatives for classification, information extraction, and retrieval operations in biomedical contexts, often with significantly reduced computational requirements [56] [55]. The recent development of advanced encoder architectures with expanded context windows, domain-optimized vocabularies, and multimodal capabilities has further strengthened their position as foundational components of biomedical AI systems [37] [57].

Biomedical researchers and drug development professionals should prioritize encoder architectures for applications involving structured prediction, clinical classification, semantic retrieval, and multimodal data alignment. The growing availability of specialized biomedical encoder models through open-source platforms enables more rapid development and deployment while addressing critical concerns regarding data privacy, reproducibility, and institutional policy compliance [57]. As encoder architectures continue to evolve with enhanced capabilities for processing long clinical documents, integrating multimodal data, and capturing complex biomedical relationships, their role as essential components of the biomedical AI toolkit appears increasingly secure.

Mitigating Hallucination and Ensuring Faithfulness in Decoder-Generated Outputs

The architectural shift in large language models (LLMs) from encoder-decoder designs to predominantly decoder-only models like GPT series, Llama, and Claude has revolutionized text generation capabilities [61] [1]. However, this transition has intensified challenges surrounding hallucination mitigation and faithfulness enforcement in generated outputs. Hallucination in LLMs refers to the generation of content that appears fluent and syntactically correct but is factually inaccurate or unsupported by external evidence [61] [62]. In decoder-only architectures, which operate through autoregressive next-token prediction, the fundamental objective of generating plausible continuations often directly conflicts with the imperative of factual accuracy [62] [63].

This comparison guide examines the landscape of hallucination mitigation strategies specifically for decoder-generated outputs, contextualized within the broader architectural debate between encoder-only, decoder-only, and hybrid approaches. We provide experimental data and methodological protocols to empower researchers in selecting appropriate faithfulness-enforcement techniques for scientific and drug development applications where factual precision is paramount.

Architectural Foundations: Encoder vs. Decoder Paradigms

The fundamental differences between encoder and decoder architectures create distinct hallucination profiles and mitigation requirements. Encoder-only models like BERT and RoBERTa utilize bidirectional attention to build comprehensive contextual representations, making them inherently suited for classification and comprehension tasks where faithfulness to input text is structural [1]. In contrast, decoder-only models employ masked self-attention that prevents attending to future tokens, generating text autoregressively through next-token prediction [1]. This autoregressive nature, while enabling powerful generative capabilities, creates an inherent tendency toward hallucination as each token prediction accumulates potential errors [62] [63].

Encoder-decoder hybrid models maintain separate parameter spaces for processing input and generating output, allowing more explicit control over the relationship between source material and generated content [4]. Recent research indicates that encoder-decoder models demonstrate comparable scaling capabilities to decoder-only alternatives while offering superior inference efficiency in some configurations [4]. For drug development professionals, this architectural choice presents critical trade-offs: decoder-only models offer greater generative flexibility, while encoder-decoder architectures provide more inherent grounding mechanisms for technical documentation and research summarization tasks.

Table 1: Architectural Comparison for Faithfulness Considerations

Architecture Type	Primary Training Objective	Hallucination Vulnerability	Typical Mitigation Approaches
Encoder-Only	Masked language modeling	Lower - outputs constrained by input	Adversarial training, contrastive learning
Decoder-Only	Causal language modeling	Higher - autoregressive generation	RAG, prompt engineering, preference optimization
Encoder-Decoder	Sequence-to-sequence learning	Moderate - mediated through encoder	Faithful fine-tuning, constrained decoding

Taxonomy and Root Causes of Decoder Hallucinations

Understanding hallucination types is prerequisite to effective mitigation. Hallucinations in decoder-generated outputs manifest primarily as intrinsic hallucinations (factuality errors), where content contradicts established facts, and extrinsic hallucinations (faithfulness errors), where content deviates from provided input or context [61] [62]. The decoder-specific architecture introduces distinct failure modes throughout the generation pipeline, from tokenization to final output selection [63].

At the tokenization stage, imperfect chunking of text into tokens can create semantic mismatches that propagate through the generation process [63]. Within the transformer block, the self-attention mechanism's query-key-value interactions determine information emphasis, with poorly calibrated attention weights prioritizing incorrect associations and seeding factual hallucinations [63]. The feed-forward network then amplifies these seeded errors through complex pattern application, before the final softmax distribution materializes hallucinations in the next-token selection [63].

Decoder hallucinations stem from interconnected causes including: (1) insufficient or biased training data causing long-tail knowledge gaps; (2) architectural limitations in attention mechanisms that fail to properly contextualize information; (3) misalignment between pre-training and instruction-tuning objectives; and (4) inherent next-token prediction bias that prioritizes plausible over accurate continuations [61] [62] [63]. In scientific domains like drug development, these manifest as incorrect chemical properties, fabricated research findings, or misattributed biological mechanisms that demand specialized mitigation approaches.

Diagram 1: Decoder Architecture and Hallucination Points

Comparative Analysis of Hallucination Mitigation Techniques

Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation addresses decoder hallucinations by grounding generation in external knowledge sources. The methodology involves: (1) implementing a retrieval module that searches vector databases or knowledge graphs for contextually relevant information; (2) augmenting the original prompt with retrieved evidence; and (3) constraining the decoder to generate from this augmented context [64]. Variants include LLM Augmentor (modifying internal parameters for task adaptation), FreshPrompt (leveraging updated search engines), and Decompose and Query frameworks (breaking complex queries into subquestions) [64].

Experimental data from clinical text generation benchmarks demonstrates RAG's effectiveness, reducing hallucinations by 45-62% compared to baseline decoder-only models in pharmaceutical documentation tasks [64]. However, RAG introduces latency overhead (150-400ms depending on retrieval complexity) and depends critically on source credibility and recency, presenting trade-offs for time-sensitive drug discovery applications.

Self-Refinement Through Feedback and Reasoning

Self-refinement techniques leverage the decoder's own capacity for iterative improvement through structured reasoning frameworks. Methodological implementations include: (1) Chain of Verification (CoVe), where models generate preliminary answers, create verification questions, then answer these questions to detect inconsistencies; (2) Self-Consistency CoT, sampling multiple reasoning paths and selecting the most consistent output; and (3) Self-Reflection methods, where models critique and revise their own outputs [64].

In molecular property prediction tasks, self-consistency CoT improved factual accuracy by 28% over standard decoding while maintaining the same model parameters [64]. The Graph-of-Thoughts (GoT) framework, which models LLM reasoning as a graph enabling more complex thought operations, demonstrated particular effectiveness for chemical synthesis pathway planning, reducing entity hallucinations by 37% compared to standard Chain-of-Thought [65].

Preference Optimization and Fine-Tuning

Preference optimization approaches directly modify decoder training objectives to penalize hallucinated outputs. The Hallucination-focused Preference Optimization method involves: (1) creating a dataset of hallucination-focused preference pairs through systematic negative example generation; (2) fine-tuning base models using preference learning algorithms like DPO or PPO; and (3) evaluating on held-out faithfulness metrics [66]. Similarly, the SCOPE framework employs self-supervised unfaithful sample generation followed by preference-based training to encourage grounded outputs [67].

Experimental results across five language pairs showed preference optimization reduced hallucination rates by an average of 96% while preserving overall translation quality [66]. In domain-specific scientific writing, SCOPE achieved 14% improvement in faithfulness metrics over standard fine-tuning approaches [67]. These methods require significant computational resources for fine-tuning but offer inference-time efficiency once deployed.

Table 2: Quantitative Comparison of Mitigation Techniques

Mitigation Approach	Hallucination Reduction	Computational Overhead	Domain Specificity	Implementation Complexity
Retrieval-Augmented Generation	45-62%	High (retrieval latency)	Low (knowledge-dependent)	Medium
Self-Consistency CoT	28-37%	Medium (multiple samples)	Medium	Low
Preference Optimization	89-96%	High (training required)	High (fine-tuning needed)	High
Context-Aware Decoding	22-31%	Low (inference-only)	Low	Medium
Decoder-Only with DoLa	18-27%	Low (inference-only)	Low	Low

Specialized Decoding Strategies

Decoding-time interventions modify token selection without retraining, offering practical deployment advantages. Context-Aware Decoding (CAD) integrates semantic context vectors into the decoding process, overriding the model's prior knowledge when it contradicts provided context [64]. Decoding by Contrasting Layers (DoLa) enhances factual accuracy by contrasting later and earlier layer projections to amplify factual knowledge while minimizing incorrect facts [64]. Controlled hallucination approaches explicitly manage the creativity-factualness tradeoff, particularly valuable for hypothesis generation in early drug discovery [62].

In path planning tasks relevant to molecular configuration, specialized techniques like S2ERS that extract entity-relationship graphs from text descriptions reduced spatial hallucinations by 29% compared to standard CoT approaches [65]. These methods demonstrate that architectural awareness in decoding strategy design can yield significant faithfulness improvements without the cost of full model retraining.

Diagram 2: Hybrid RAG with Self-Refinement Workflow

Experimental Protocols for Hallucination Assessment

Faithfulness Evaluation Methodology

Rigorous hallucination assessment requires multi-faceted evaluation protocols combining automatic metrics, LLM-as-a-judge, and human expert review. For scientific domains, we recommend implementing:

Automatic Metric Protocol:

• NLI-based faithfulness scoring using models trained on natural language inference tasks to quantify entailment between source and generated text [67]
• Entity-level consistency metrics tracking hallucination rates for specific entities (e.g., chemical compounds, protein names, biological processes) [62]
• PARENT metric adaptation for table-to-text generation, computing n-gram overlap against source table cells [67]
• Temporal consistency verification particularly critical for drug development timelines and clinical trial references

LLM-as-Judge Protocol:

• Implement pairwise comparison with carefully designed faithfulness criteria
• Use scoring rubrics with domain-specific faithfulness dimensions
• Employ multi-LLM adjudication to reduce model-specific biases
• Conduct cross-verification with expert human evaluations

Human Evaluation Protocol:

• Engage domain experts (e.g., medicinal chemists, pharmacologists) for content verification
• Implement double-blind rating procedures to reduce bias
• Use standardized faithfulness scales with explicit violation typologies
• Calculate inter-annotator agreement to ensure rating consistency

Domain-Specific Benchmarking

For drug development applications, we propose augmenting standard benchmarks with domain-specific test sets evaluating:

• Preclinical data summarization faithfulness
• Mechanism of action description accuracy
• Drug-drug interaction reporting precision
• Clinical trial result representation fidelity

Experimental data from adapted pharma benchmarks indicates that decoder-only models with RAG and self-consistency checking achieve 87% faithfulness scores compared to 53% for base models, highlighting the critical importance of targeted mitigation in scientific domains [64] [67].

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Research Reagents for Hallucination Mitigation Experiments

Reagent Solution	Function	Implementation Example
Faithfulness-Annotated Datasets	Provides ground truth for training and evaluation	Factually Annotated Clinical Summaries (FACS), Biomedical Fact-Checking Corpus
Retrieval Augmentation Tools	Grounds generation in external knowledge	Vector databases (Pinecone, Chroma), Knowledge graphs (Bio2RDF, Chem2RDF)
Preference Optimization Algorithms	Aligns model outputs with factual accuracy	Direct Preference Optimization (DPO), Reinforcement Learning from Human Feedback (RLHF)
Contrastive Decoding Libraries	Implements advanced decoding strategies	DoLa, Context-Aware Decoding, Knowledge-aware Decoding
Faithfulness Metrics Suite	Quantifies hallucination rates	NLI-based metrics, Entity consistency metrics, PARENT adaptation for scientific tables
Multi-Step Reasoning Frameworks	Enhances logical consistency	Chain-of-Thought, Graph-of-Thoughts, Tree-of-Thoughts implementations

The comparative analysis reveals that no single approach completely eliminates decoder hallucinations; instead, layered mitigation strategies deliver optimal results. For drug development professionals, we recommend: (1) RAG implementation for knowledge-intensive tasks like literature summarization; (2) self-consistency verification for complex reasoning tasks like mechanism elucidation; and (3) domain-specific preference optimization for standardized reporting tasks.

Encoder-decoder architectures warrant reconsideration for applications requiring strict faithfulness guarantees, as they demonstrate compelling scaling properties and superior inference efficiency in recent evaluations [4]. However, decoder-only models with comprehensive mitigation strategies maintain advantages for flexible generation across diverse scientific communication tasks.

Future research directions should prioritize: (1) development of specialized hallucination benchmarks for pharmaceutical applications; (2) exploration of decoder architectures with explicit uncertainty modeling; and (3) creation of hybrid systems that strategically deploy encoder-style verification for decoder-generated content. As architectural evolution continues, the fundamental tradeoff between generative flexibility and factual precision will remain central to deploying trustworthy LLMs in critical drug development workflows.

In the development of large language models (LLMs) for scientific domains, the strategy used to assemble training data is as critical as the model architecture itself. The academic and industrial discourse often centers on the merits of encoder-only, decoder-only, and encoder-decoder architectures [16]. However, the efficacy of any architecture is profoundly mediated by the data paradigm employed: curating high-fidelity input-output pairs or leveraging massive unsupervised corpora [68] [69]. The former provides clear, task-specific supervision but is often scarce and expensive to produce, especially in specialized fields like materials science and drug development. The latter is abundant and cheap to acquire but presents a more challenging learning problem. This guide objectively compares the performance of models trained under these two data-centric paradigms, contextualizing the findings within the broader architectural debate and providing experimental protocols for researchers.

Architectural & Data-Centric Landscape

The performance of any LLM is a function of its architecture and its training data. Understanding the core distinctions in both areas is essential for a meaningful comparison.

A Primer on Model Architectures

Modern LLMs primarily use one of three Transformer-based architectures, each with distinct inductive biases and performance profiles [28] [16].

• Encoder-Decoder Models (e.g., T5): These models feature a bidirectional encoder that processes the full input sequence and an autoregressive decoder that generates the output. Historically, they have been powerful for tasks like translation and summarization but were perceived as less scalable than decoder-only models [28] [16].
• Decoder-Only Models (e.g., GPT, LLaMA): The current dominant architecture, it uses a single stack of causal (masked) self-attention layers. It is trained with a next-token prediction objective on vast unlabeled corpora, making it highly scalable and versatile for generative tasks [28] [2].
• Encoder-Only Models (e.g., BERT): Utilizing bidirectional attention, these models excel at understanding tasks like classification and named entity recognition but are not natively designed for text generation [16].

Recent research indicates that the potential of encoder-decoder models may have been overlooked. When enhanced with modern techniques from decoder-only LLMs (e.g., rotary embeddings, RMSNorm), encoder-decoder models demonstrate comparable scaling and even superior inference efficiency after instruction tuning [28] [4].

Data Optimization Paradigms

The two primary data optimization strategies represent a fundamental trade-off between data quality and quantity.

• Curated Input-Output Pairs: This supervised approach involves training a model on a dataset of high-quality, human-annotated examples (e.g., a document paired with its summary). The model learns a direct mapping from source to target, which is highly sample-efficient but limited by the availability and cost of annotated data [68].
• Unsupervised Corpora: This approach leverages massive amounts of raw, unlabeled text (e.g., web crawls, scientific literature). Using self-supervised objectives like next-token prediction or masked language modeling, the model learns the underlying structure and statistics of the language. This approach benefits from vast data but requires more sophisticated methods to adapt to specific tasks [70].

Experimental Comparison & Performance Data

To quantitatively compare these paradigms, we examine experimental results from recent studies, focusing on tasks relevant to scientific research, such as summarization and question generation.

Table 1: Performance Comparison of Data-Centric Paradigms on Summarization & Question Generation

Data Paradigm	Model / Method	Dataset	ROUGE-L	Key Inference
Curated Pairs (Synthetic)	Paired by the Teacher (PbT) 8B [68] [69]	XSum (Summarization)	Within 1.2 pts of human-annotated pairs	Closes 82% of the performance gap to a fully human-annotated oracle at one-third the cost.
		SAMSum (Dialogue Sum.)	Comparable to above	Generates concise, faithful summaries aligned with target style, avoiding domain mismatch.
Unsupervised Corpora	Decoder-Only (DecLLM) ~8B [28] [4]	RedPajama (Pretraining)	N/A	More compute-optimal during the initial pretraining phase.
	Encoder-Decoder (RedLLM) ~8B [28] [4]	RedPajama (Pretraining)	N/A	Shows comparable scaling and context-length extrapolation to DecLLM.
Instruction Tuning	Decoder-Only (DecLLM) ~8B [28]	FLAN (various tasks)	Strong	Achieves strong zero- and few-shot performance after instruction tuning.
	Encoder-Decoder (RedLLM) ~8B [28] [4]	FLAN (various tasks)	Comparable / Better	Achieves comparable or better results on various tasks with substantially better inference efficiency.

Table 2: Architectural Performance with Different Data & Task Types

Architecture	Optimal Data Paradigm	Excels at Task Type	Key Advantage
Encoder-Decoder	Curated Pairs / Instruction Tuning [28] [4]	Tasks requiring deep understanding before generation (e.g., translation, summarization) [71]	High inference efficiency and strong performance post-tuning; bidirectional encoder captures full input context [28].
Decoder-Only	Unsupervised Corpora (Pretraining) + Instruction Tuning [28] [71]	General text generation and few-shot learning [16] [71]	Superior compute-optimality during pretraining; unified, scalable architecture [28].
Encoder-Only	Unsupervised Corpora (via MLM) [70] [16]	Discriminative tasks (e.g., classification, NER) [16]	Bidirectional attention provides rich contextual representations of input text [16].

Key Findings and Interpretation

The data reveals a nuanced landscape. The Paired by the Teacher (PbT) method demonstrates that high-quality synthetic input-output pairs can nearly match the performance of costly human-annotated data [68] [69]. This is a significant advancement for low-resource domains, effectively bridging the gap between the curated pairs and unsupervised corpora paradigms.

Architecturally, while decoder-only models dominate the pretraining efficiency frontier, modern encoder-decoder models are highly competitive after instruction tuning, often matching or exceeding the performance of their decoder-only counterparts while being more efficient at inference time [28] [4]. This challenges the prevailing narrative that decoder-only architectures are universally superior.

Detailed Experimental Protocols

For researchers seeking to reproduce or build upon these results, this section outlines the core methodologies.

Protocol 1: The Paired by the Teacher (PbT) Pipeline

PbT is a two-stage teacher-student pipeline designed to create high-fidelity input-output pairs from unpaired data alone [68] [69].

Workflow Diagram: PbT Data Synthesis Pipeline

Methodology Details:

• Source-side IR Learning:

  • Input: A collection of unpaired source texts (e.g., scientific documents).
  • Step 1 (IR Extraction): A powerful teacher LLM (e.g., GPT-4) compresses each source into a concise Intermediate Representation (IR). This IR can be a set of keywords, a structured outline, or an extreme summary [69].
  • Step 2 (Student Training): A smaller, more efficient student model is fine-tuned to reconstruct the original source text from its corresponding IR. This teaches the student to generate coherent and in-domain text based on a compressed representation [69].

• Target IR Annotation & Synthetic Pair Generation:

  • Input: A collection of unpaired target texts (e.g., summaries from a different domain).
  • Step 1 (IR Annotation): The teacher LLM, provided with a few examples, annotates each unpaired target with a plausible IR [69].
  • Step 2 (Source Generation): The trained student model from Phase 1 generates a synthetic source text from each IR. The original target is now paired with this student-generated source, creating a high-quality, in-domain synthetic pair (synthetic_source, original_target) [68] [69].

• Downstream Fine-tuning: A final model (e.g., a summarizer) is trained on these synthetically generated pairs, enabling it to perform the target task effectively without ever having seen a human-annotated pair [69].

Protocol 2: Scaling Laws for Architectural Comparison

This protocol involves a controlled, large-scale comparison of encoder-decoder and decoder-only architectures to understand their scaling properties [28] [4].

Workflow Diagram: Architectural Scaling Study Protocol

Methodology Details:

• Controlled Pretraining:

  • Models: Train two model families—RedLLM (encoder-decoder) and DecLLM (decoder-only)—across a range of scales (e.g., ~150M to ~8B parameters). Crucially, apply modern training recipes (e.g., rotary embeddings, SwiGLU) to both to ensure a fair comparison [28].
  • Data & Objective: Pretrain all models on the same large corpus (e.g., RedPajama V1, 1.6T tokens). RedLLM uses a prefix language modeling objective, while DecLLM uses a standard causal language modeling objective [28] [4].

• Instruction Tuning:

  • Finetune all pretrained models on the same instruction dataset (e.g., FLAN) to elicit zero-shot and few-shot task-solving capabilities [28].

• Evaluation:

  • Measure zero- and few-shot performance on a diverse set of downstream tasks.
  • Analyze scaling laws by plotting performance against compute budget (FLOPs) and model size.
  • Benchmark inference efficiency (latency/throughput) for each architecture.
  • Test the ability of the models to handle context lengths longer than those seen during training [28] [4].

The Scientist's Toolkit: Key Research Reagents

This section details the essential "research reagents"—datasets, models, and algorithms—required for experiments in data-centric LLM optimization.

Table 3: Essential Reagents for Data-Centric LLM Research

Reagent Name	Type	Primary Function	Example in Use
RedPajama V1 [28] [4]	Unsupervised Corpus	A massive, open-source corpus for pretraining LLMs. Provides the foundational language knowledge.	Used as the primary pretraining dataset in architectural scaling studies [28].
FLAN Collection [28] [4]	Instruction Tuning Data	A collection of tasks formatted with instructions. Used to teach models to follow instructions and solve diverse tasks.	Applied for instruction tuning encoder-decoder and decoder-only models to improve their zero-shot performance [28].
XSum, SAMSum, SQuAD [68] [69]	Benchmark Datasets	Standardized datasets for evaluating performance on specific tasks like summarization and question generation.	Served as the source of unpaired targets and for benchmarking the PbT method [68].
Teacher LLM (e.g., GPT-4, LLaMA-70B) [68] [69]	Model	A large, powerful model used to generate guidance, such as Intermediate Representations (IRs) or synthetic labels.	Core component of the PbT pipeline for IR extraction and annotation [69].
Paired by the Teacher (PbT) [68] [69]	Algorithm	A pipeline for synthesizing high-quality input-output pairs from unpaired data, overcoming data scarcity.	Enables training of effective summarization models without human-annotated pairs [68].
Intermediate Representation (IR) [69]	Data Structure	A compressed, structured representation of a text (e.g., keywords, outline) that acts as a bottleneck between teacher and student.	Facilitates the transfer of knowledge from the teacher LLM to the student model in PbT without direct text generation by the teacher [69].

The choice between curating input-output pairs and leveraging unsupervised corpora is not a binary one but a strategic continuum. For low-resource, domain-specific applications (e.g., generating summaries of molecular research), advanced synthesis methods like PbT that generate high-fidelity curated pairs offer a path to state-of-the-art performance without prohibitive annotation costs [68] [69]. For building general-purpose, foundational models, pretraining on massive unsupervised corpora remains the essential starting point [28] [70].

Architecturally, the dominance of the decoder-only paradigm is justified by its pretraining efficiency and simplicity [28] [71]. However, evidence shows that the modern encoder-decoder architecture is a powerful and often more efficient alternative, especially after instruction tuning, and deserves renewed attention from the research community [28] [4]. The optimal solution will depend on the specific constraints of the research problem: the availability of data, the computational budget, and the required task performance and inference latency.

Hardware-Aware Design and Deployment Strategies for Accessible AI

The escalating computational demands of artificial intelligence (AI), particularly within data-intensive fields like biotechnology and drug discovery, have rendered hardware-aware design not merely an optimization tactic but a fundamental prerequisite for accessible and scalable research. Industry analyses indicate that AI compute demand is rapidly outpacing infrastructure supply, with global AI data centers potentially requiring 200 gigawatts of power by 2030 and trillions of dollars in infrastructure spending [72]. Within this constrained landscape, the strategic selection between encoder-only and decoder-only transformer architectures has emerged as a critical determinant of deployment feasibility, performance, and cost-effectiveness for scientific applications.

This guide provides an objective comparison of these architectural paradigms, focusing on their performance characteristics, resource requirements, and suitability for biomedical research tasks. By synthesizing recent experimental evidence and deployment case studies, we aim to equip researchers and drug development professionals with the analytical framework necessary to align architectural selection with both scientific objectives and computational realities.

Core Architectural Differences

Transformer architectures are primarily categorized into encoder-only, decoder-only, and encoder-decoder models. For scientific embedding and classification tasks, the encoder-decoder and encoder-only paradigms are most relevant.

• Encoder-Only Models (e.g., BERT, BioLinkBERT, ModernBERT): Built primarily for understanding and representing input data. They utilize bidirectional self-attention, meaning they process each token in the context of all other tokens in the sequence, both left and right [26] [17]. This makes them exceptionally effective at capturing deep semantic meaning, which is crucial for tasks like semantic similarity, classification, and retrieval in scientific corpora.
• Decoder-Only Models (e.g., GPT, LLaMA, Gemma): Designed for text generation. They employ autoregressive self-attention, where each token can only attend to previous tokens in the sequence [73] [74]. This unidirectional context is less optimal for tasks requiring a holistic understanding of the entire input.
• Encoder-Decoder Models (e.g., T5, BART): Combine an encoder for input understanding and a decoder for output generation, making them suitable for sequence-to-sequence tasks like translation and summarization [75].

Hardware and Computational Implications

The architectural differences translate directly into distinct computational profiles, which are paramount for hardware-aware deployment.

Table 1: Computational Profiles of Encoder vs. Decoder Models for Embedding Tasks

Model Characteristic	Encoder-Only Model (e.g., BioLinkBERT)	Decoder-Style Model (e.g., Gemma-2-2B)
Core Architecture	Bidirectional Self-Attention [26]	Autoregressive Self-Attention [74]
Typical Model Size	340 million parameters [76]	2.5 billion parameters [76]
Inference Speed (Embeddings/sec)	143.5 embeddings/second [76]	55.5 embeddings/second [76]
Memory Footprint	1.51 GB [76]	12.0 GB [76]
Inference Cost	Lower (Smaller, faster, affordable hardware) [26]	Higher (Larger, slower, requires expensive hardware) [26]

Experimental Comparison: A Case Study in Clinical Cardiology

To isolate architectural effects under a consistent regime, we examine a rigorous comparative evaluation of models fine-tuned for a domain-specific scientific task: generating embeddings for clinical cardiology concepts [76].

Experimental Protocol and Methodology

Objective: To compare the performance and efficiency of encoder-only and decoder-style models after domain adaptation via Parameter-Efficient Fine-Tuning (PEFT) for retrieving related cardiology concepts.

Model Selection:

• Encoder-Only: BioLinkBERT-base (340M), BGE-M3 (568M), BGE-large-v1.5 (335M), and others.
• Decoder-Style: Gemma-2-2B (2.5B), Qwen2.5-0.5B (494M), Qwen3-4B (4B), and others [76].

Training Procedure:

• Domain Adaptation: All models underwent Low-Rank Adaptation (LoRA) fine-tuning on approximately 150,000 sentence pairs derived from authoritative cardiology textbooks [76].
• Parameter-Efficient Fine-Tuning: The LoRA method freezes the pre-trained model weights and injects trainable rank decomposition matrices into the transformer layers. This drastically reduces the number of trainable parameters (e.g., only 1.05% for a 4B parameter model) [76].
• Hardware & Software: Training was conducted on an NVIDIA A100 80GB GPU using 8-bit quantization via bitsandbytes to reduce memory footprint [76].
• Loss Function: Multiple Negatives Ranking Loss (InfoNCE) was used with a contrastive learning objective to teach the model to place semantically similar cardiology concepts closer in the embedding space [76].

The workflow for this experimental protocol is summarized in the following diagram:

Quantitative Performance Results

The models were evaluated on their ability to discriminate between similar and dissimilar cardiology concepts, a critical capability for accurate clinical information retrieval.

Table 2: Performance and Efficiency Metrics on Cardiology Embedding Task

Model	Architecture	Parameters	Cardiology Separation Score	Inference Throughput (emb/sec)	Memory Footprint (GB)
BioLinkBERT-base	Encoder-Only	340M	0.510	143.5	1.51
BGE-large-v1.5	Encoder-Only	335M	0.481	139.2	1.49
Gemma-2-2B	Decoder-Style	2.5B	0.455	55.5	12.0
Qwen2.5-0.5B	Decoder-Style	494M	0.442	78.3	3.1
Zero-Shot Baseline	-	-	0.057	-	-

Key Finding: The top-performing encoder-only model (BioLinkBERT, 340M) achieved a 12% higher separation score than the top-performing decoder-style model (Gemma-2-2B, 2.5B) while being ~7.9x smaller and delivering ~2.6x higher inference throughput [76]. This demonstrates that for domain-specific representation tasks, bidirectional architectural bias and specialized pre-training outweigh the advantages of simply having more parameters.

The Researcher's Toolkit: Key Materials and Methods

Successful deployment of AI models in scientific workflows relies on a suite of software and methodological "reagents."

Table 3: Essential Research Reagent Solutions for Accessible AI Deployment

Research Reagent	Function	Relevance to Accessible AI
LoRA (Low-Rank Adaptation)	Parameter-efficient fine-tuning method [76].	Enables domain adaptation of large models on a single GPU, drastically reducing compute cost.
8-bit Quantization (bitsandbytes)	Reduces numerical precision of model weights [76].	Cuts memory footprint by ~50%, allowing larger models to fit on consumer-grade hardware.
Contrastive Learning (InfoNCE Loss)	Training objective for semantic similarity [76].	Critical for teaching models to create well-separated embeddings for scientific concepts.
BioLinkBERT	A domain-specific pre-trained encoder model [76].	Provides a strong, biologically-aware foundation for fine-tuning, improving downstream performance.
ModernBERT	A modern, efficiency-optimized encoder model [26].	Incorporates architectural improvements (RoPE, GeGLU) for better performance on long sequences with high speed.

Deployment Strategies and Real-World Applications

Strategic Model Selection Framework

The choice between encoder and decoder models should be guided by the target task and operational constraints. The following diagram outlines this decision logic:

Encoder-Centric Deployment Patterns in Biotech

Encoder-only models have become the workhorses in several key biomedical AI applications due to their efficiency and precision [26].

• Retrieval Augmented Generation (RAG): In AI-powered drug discovery platforms, encoder models like BERT and ModernBERT are used to efficiently encode and retrieve millions of scientific documents, patent texts, and molecular data sheets. This provides a factual foundation for a downstream decoder model that generates reports or hypotheses, balancing accuracy with creativity [26].
• Content Moderation and Classification: Encoder models can quickly and accurately scan and classify vast volumes of user-generated content or internal scientific data, ensuring platform safety and data compliance without the overhead of larger generative models [26]. This is crucial for maintaining reproducible and auditable research data lakes.
• Semantic Search in Scientific Databases: As demonstrated in the cardiology case study, fine-tuned encoders power high-precision semantic search engines that allow researchers to find related clinical concepts, genetic markers, or chemical compounds based on semantic meaning rather than just keyword matching [76].

The empirical evidence clearly indicates that for the majority of scientific embedding, classification, and retrieval tasks—which form the backbone of data-driven drug discovery—encoder-only models offer a superior balance of performance and hardware efficiency. The cardiology embedding study proves that a well-designed, domain-adapted encoder model can significantly outperform decoder models that are an order of magnitude larger, while being dramatically faster and cheaper to deploy [76].

The strategic implication for researchers and drug development professionals is clear: prioritize encoder-only architectures for understanding-based tasks. This hardware-aware approach is not merely an engineering concern but a core component of sustainable and accessible AI strategy, enabling robust scientific AI applications without necessitating prohibitive computational investment.

Empirical Evidence and Decision Framework: Choosing the Right Tool for the Job

In the field of natural language processing, the architectural choice between encoder-only and decoder-only models represents a fundamental trade-off between deep language understanding and generative capability. While decoder-only models like GPT and LLaMA dominate public discourse for their impressive text generation, encoder-only models such as BERT and its modern variants remain the workhorses behind countless practical applications [26]. This guide provides an objective, data-driven comparison of these architectures, focusing on their benchmarking performance across accuracy, F-score, and computational efficiency metrics, with particular relevance for scientific and research applications. The evaluation is framed within materials research contexts where precise information extraction and classification are paramount, providing drug development professionals and researchers with evidence-based selection criteria for their specific use cases.

The divergence in architectural approaches stems from different design philosophies: encoder-only models utilize bidirectional attention to build comprehensive contextual representations of input text, while decoder-only models employ causal attention to generate sequences autoregressively [48] [26]. This fundamental distinction translates to significant performance differences across various tasks, with implications for research workflows where both accuracy and efficiency considerations are critical.

Architectural Fundamentals and Experimental Framework

Core Architectural Differences

The transformer architecture, first introduced in 2017, provides the foundation for both encoder-only and decoder-only models, yet their operational principles differ significantly:

• Encoder-Only Models: These models process input sequences bidirectionally, meaning each token can attend to all other tokens in the sequence simultaneously. This architecture creates rich, contextualized representations of the entire input, making it exceptionally well-suited for understanding tasks [26]. The original BERT model exemplifies this approach, using masked language modeling to develop a deep understanding of language structure and meaning.

• Decoder-Only Models: These models process text autoregressively with a unidirectional attention mechanism that restricts each token to attending only to previous tokens in the sequence. This design optimizes them for text generation tasks, where producing coherent, sequential output is the primary objective [48]. Models in the GPT family follow this architectural pattern, predicting each subsequent token based on the preceding context.

Standardized Evaluation Metrics

To ensure fair comparison across architectures, researchers employ standardized evaluation metrics:

• Accuracy: Measures the overall correctness of model predictions across all classes, though this can be misleading in imbalanced datasets [77].

• F1-Score: The harmonic mean of precision and recall, providing a balanced metric that accounts for both false positives and false negatives [77]. This is particularly valuable in scientific applications where both error types carry consequences.

• Computational Efficiency: Encompasses training time, inference latency, and resource requirements (memory, processing power), often measured in tokens processed per second or energy consumption per inference [26].

• Context Length Extrapolation: The model's ability to handle increasingly long input sequences while maintaining performance, crucial for processing scientific documents and research papers [28].

The F1-score deserves particular attention for scientific applications. As a balanced metric, it prevents scenarios where high precision comes at the cost of missed detections (low recall), or high recall is achieved through excessive false alarms (low precision) [77]. This balance is critical in research contexts where comprehensive entity extraction (e.g., identifying all chemical compounds in a document) must be balanced against precision to avoid contaminating results with incorrect extractions.

Performance Benchmarking: Quantitative Comparisons

Information Extraction and Classification Tasks

Table 1: Performance Comparison on Named Entity Recognition (Medical Domain)

Model Architecture	Specific Model	Precision	Recall	F1-Score	Task Domain
Encoder-Only	Flat NER (Best Performing)	0.87-0.88	0.87-0.88	0.87-0.88	Pathology Reports
Encoder-Only	Flat NER	-	-	Up to 0.78	Radiology Reports
Decoder-Only	Various LLMs	High (Exact values not specified)	Low	0.18-0.30	Clinical Entity Extraction

Encoder-only models demonstrate superior performance on structured information extraction tasks, as evidenced by comprehensive evaluations in clinical settings [39] [40]. In a comparative study analyzing pathology and radiology reports for named entity recognition, encoder-based models achieved F1-scores of 0.87-0.88 on pathology reports and up to 0.78 on radiology reports [40]. In stark contrast, various decoder-only large language models achieved significantly lower F1-scores ranging from 0.18 to 0.30, despite high precision scores [39]. This performance gap highlights a critical limitation of decoder-only models for extraction tasks: they tend to be overly conservative, producing fewer but more accurate entities, resulting in poor recall that substantially drags down overall F1 performance [40].

The bidirectional attention mechanism in encoder-only models provides a clear advantage for understanding tasks where comprehensive context is essential. As one study concluded, "LLMs in their current form are unsuitable for comprehensive entity extraction tasks in clinical domains, particularly when faced with a high number of entity types per document" [40]. This finding has significant implications for materials research and drug development applications where thorough extraction of chemical entities, protein interactions, or material properties is required.

Question Answering and Reasoning Tasks

Table 2: Performance on STEM Question Answering with Context

Model Architecture	Specific Model	Performance Notes	Parameter Count
Encoder-Only	DeBERTa v3 Large	Outperforms Llama 2-7B	~400M
Decoder-Only	Mistral-7B Instruct	Outperforms Llama 2-7B, comparable to DeBERTa	7B
Decoder-Only	Llama 2-7B	Lower performance than other models	7B

In challenging STEM multiple-choice question answering, both architectural families demonstrate capabilities when provided with appropriate context [22]. Research evaluating models on LLM-generated STEM questions found that both encoder-only models (DeBERTa v3 Large) and decoder-only models (Mistral-7B Instruct) can outperform larger parameter models when properly fine-tuned with context [22]. This suggests that parameter count alone does not determine performance on complex technical questions, and that architectural advantages and training methodologies play significant roles.

Notably, the encoder-only DeBERTa model with approximately 400 million parameters achieved performance comparable to the 7-billion parameter Mistral model, suggesting greater parameter efficiency for encoder architectures in understanding tasks [22]. This efficiency advantage makes encoder-only models particularly attractive for research institutions with computational constraints.

Computational Efficiency and Scaling Properties

Table 3: Computational Efficiency Comparison

Metric	Encoder-Only Models	Decoder-Only Models	Notes
Inference Speed	Fast	Slow to Moderate	Encoder models show 2.4-6.5× speedups [35]
Memory Footprint	Low	High	Decoder KV cache increases memory usage
Hardware Requirements	Consumer-grade GPUs (e.g., NVIDIA RTX 4090) [26]	Specialized high-end servers
Context Processing	Bidirectional, parallel	Sequential, autoregressive
Practical Deployment	Suitable for high-volume, low-latency applications [26]	Limited by speed and cost at scale

Computational efficiency represents a significant differentiator between architectural approaches. Encoder-only models consistently demonstrate advantages in inference speed, memory requirements, and hardware accessibility [26]. One machine translation study reported that hybrid approaches using encoder-only components achieved "2.4 ∼ 6.5 × inference speedups and a 75% reduction in the memory footprint of the KV cache" compared to decoder-only approaches [35].

The efficiency advantage extends to practical deployment scenarios. As noted in an analysis of encoder-only models, "decoder-only models are too big, slow, private, and expensive for many jobs" [26]. The author illustrates this with a compelling cost comparison: filtering 15 trillion tokens with fine-tuned BERT-based models cost approximately $60,000, while the same processing with decoder-only API calls would exceed one million dollars [26].

Recent scaling analysis reveals that while decoder-only models are generally more compute-optimal during pretraining, encoder-decoder hybrids demonstrate comparable scaling properties and, after instruction tuning, achieve competitive performance on downstream tasks while maintaining superior inference efficiency [28] [4]. This suggests that the recent industry shift toward pure decoder architectures may warrant reconsideration for applications where both understanding and generation are required.

Experimental Protocols and Methodologies

Named Entity Recognition Evaluation Protocol

The superior performance of encoder-only models on extraction tasks is demonstrated through rigorous evaluation methodologies:

Figure 1: Named Entity Recognition Experimental Workflow

The NER evaluation methodology follows a structured approach [39] [40]:

• Data Collection and Annotation: Researchers compiled 2,013 pathology reports and 413 radiology reports from real-world clinical settings. Medical students with domain expertise annotated these reports to establish ground truth labels for clinical entities [40].

• Model Training Approaches: Three distinct NER methodologies were implemented:

  • Flat NER using transformer-based encoder models
  • Nested NER with a multi-task learning setup
  • Instruction-based NER utilizing decoder-only LLMs

• Evaluation Metrics: Models were evaluated using precision, recall, and F1-score to provide a comprehensive view of performance characteristics, with particular attention to the balance between false positives and false negatives [77].

This rigorous methodology ensures fair comparison across architectures and provides insights into the practical strengths and limitations of each approach for scientific information extraction.

STEM Question Answering Experimental Design

Figure 2: STEM Question Answering Evaluation Protocol

The STEM question answering evaluation follows these key methodological steps [22]:

• Challenge Dataset Creation: Due to the absence of benchmark STEM datasets created by LLMs, researchers employed various models (Vicuna-13B, Bard, GPT-3.5) to generate multiple-choice questions on STEM topics curated from Wikipedia, creating a challenging evaluation set.

• Contextual Learning Setup: Models were evaluated under different context conditions, including inference with added context and fine-tuning with and without context, to isolate the impact of contextual information on performance.

• Cross-Architecture Comparison: The study evaluated open-source encoder and decoder models alongside closed-source counterparts (Gemini, GPT-4) to understand performance gaps and the potential for context to narrow these gaps.

This experimental design allows researchers to assess not only raw performance but also the efficiency of different architectures in leveraging contextual information for improved performance on technical domains.

Model Architectures and Implementations

Table 4: Essential Research Tools for Model Evaluation

Resource Category	Specific Tools	Research Application	Key Characteristics
Encoder-Only Models	BERT, ModernBERT, DeBERTa	Information extraction, classification	Bi-directional attention, parameter efficiency [26]
Decoder-Only Models	LLaMA, Mistral, GPT	Text generation, reasoning	Autoregressive generation, strong few-shot learning
Evaluation Frameworks	Hugging Face, scikit-learn	Performance benchmarking	Standardized metrics, reproducibility
Computational Resources	NVIDIA T4/RTX 4090, Google Colab	Experimental infrastructure	Accessibility, scaling capabilities
Specialized Datasets	RedPajama, FLAN, Paloma	Training and evaluation	Domain relevance, quality annotations

For researchers embarking on architectural comparisons, several resources have proven essential:

• Encoder Model Variants: ModernBERT represents a significant advancement in encoder architecture, extending context length to 8,192 tokens and incorporating architectural improvements like Rotary Positional Embeddings (RoPE) and GeGLU activation layers [26]. These enhancements make it particularly suitable for scientific document processing.

• Evaluation Platforms: Hugging Face's transformer library provides standardized implementations of both architectural families, ensuring consistent evaluation metrics and eliminating implementation variance as a confounding factor in performance comparisons.

• Computational Infrastructure: While encoder models can run efficiently on consumer-grade hardware like NVIDIA RTX 4090s [26], comprehensive benchmarking of decoder models typically requires access to cloud computing resources or specialized AI accelerators.

Implementation Considerations for Research Applications

When implementing these architectures for materials research and drug development, several practical considerations emerge:

• Data Characteristics: Encoder-only models demonstrate particular advantages with technical and scientific language where precise terminology and contextual relationships are critical [39]. Their bidirectional understanding helps capture complex scientific concepts that may require full document context.

• Deployment Constraints: For high-throughput screening of scientific literature or real-time analysis of experimental data, the inference speed advantages of encoder-only models (2.4-6.5× faster) [35] can significantly impact research velocity and computational costs.

• Hybrid Approaches: Emerging research suggests hybrid architectures that leverage both encoder and decoder components may offer optimal balance for applications requiring both deep understanding and generation capabilities [28] [35].

The benchmarking data reveals a clear pattern of complementary strengths between architectural approaches. Encoder-only models consistently demonstrate superior performance on understanding tasks—including named entity recognition, text classification, and question answering—while achieving significantly better computational efficiency [39] [40] [26]. These advantages make them particularly well-suited for scientific applications involving information extraction from research papers, technical documentation, and experimental reports.

Decoder-only models excel in generative tasks and few-shot learning scenarios but face limitations in comprehensive information extraction and computational demands that may constrain their practical deployment in research settings [39] [26]. Recent advancements in encoder-decoder hybrid models suggest promising directions for achieving both understanding and generation capabilities while maintaining efficiency [28] [4].

For the materials research and drug development community, encoder-only models represent a compelling choice for the majority of information processing tasks, offering an optimal balance of performance, efficiency, and accuracy. As architectural evolution continues, researchers should maintain evaluation frameworks that account for all three dimensions—accuracy, F-score, and computational efficiency—to ensure optimal model selection for their specific research objectives.

In the field of pharmaceutical research, the accurate classification of drug-target interactions (DTI) is a critical step in the drug discovery pipeline. Encoder-only transformer models have emerged as powerful tools for this task, demonstrating exceptional performance in predicting druggable targets and classifying drug properties. Unlike decoder-only models designed for text generation, encoder-only models are specifically engineered to create rich, contextual representations of input data, making them ideally suited for understanding complex biological relationships [1]. This case study examines the application of encoder-only architectures for high-accuracy drug-target classification, comparing their performance against alternative approaches and providing detailed experimental protocols for implementation.

The foundational architecture of encoder-only models stems from the original transformer's encoder component, which processes input sequences bidirectionally to understand context from both directions simultaneously [1]. This bidirectionality is particularly valuable in biological contexts where the meaning of molecular sequences depends on broader contextual patterns. Models like BERT (Bidirectional Encoder Representations from Transformers) and its optimized variant RoBERTa utilize pretraining objectives such as masked language modeling, where random tokens in the input sequence are masked and the model learns to predict them based on surrounding context [1]. This approach enables the model to develop a profound understanding of molecular syntax and semantics, which can then be fine-tuned for specific drug classification tasks.

Theoretical Foundation: How Encoder-Only Models Work

Core Architectural Principles

Encoder-only models process input data through multiple layers of bidirectional self-attention mechanisms. Unlike the unidirectional attention found in decoder-only models, which restricts context to preceding tokens, encoder models attend to all positions in the input sequence simultaneously [1]. This architectural difference is crucial for drug-target classification, where the relationship between molecular components depends on holistic understanding rather than sequential generation.

The pretraining process for encoder-only models typically employs masked language modeling (MLM), where approximately 15% of input tokens are randomly masked, and the model learns to predict the original tokens based on the surrounding context [1]. For drug discovery applications, this approach translates to masking portions of molecular representations (such as SMILES strings or amino acid sequences) and training the model to reconstruct them, thereby building a robust understanding of molecular grammar and structure.

Adaptation for Drug-Target Classification

When adapted for pharmaceutical applications, encoder-only models process structured biological data through several transformation steps:

• Input Representation: Drug molecules are typically represented as SMILES (Simplified Molecular Input Line Entry System) strings, while target proteins are represented as amino acid sequences [78]. These sequences are tokenized into smaller subunits (e.g., atoms/bonds for molecules, k-mers for proteins).

• Embedding Layer: Tokenized sequences are mapped to dense vector representations, with positional encodings added to preserve sequence order information.

• Encoder Stack: Multiple transformer encoder layers process the embeddings using self-attention mechanisms, building increasingly sophisticated representations of the input data.

• Classification Head: The final representation (typically the [CLS] token's embedding) is fed into a task-specific classification layer for prediction [1].

This architecture enables the model to capture complex, non-linear relationships between molecular structures and their biological activities, providing the foundation for high-accuracy drug-target classification.

Experimental Evidence: Performance Comparison

Case Study: optSAE + HSAPSO Framework

A recent study introduced an optimized stacked autoencoder (optSAE) integrated with hierarchically self-adaptive particle swarm optimization (HSAPSO) for drug classification and target identification. This framework demonstrated exceptional performance, achieving a classification accuracy of 95.52% on datasets from DrugBank and Swiss-Prot [36]. The model exhibited significantly reduced computational complexity (0.010 seconds per sample) and exceptional stability (± 0.003) across various validation sets [36]. Comparative analysis revealed that this encoder-based approach outperformed traditional methods like support vector machines and XGBoost, which often struggle with the high dimensionality and complex patterns in pharmaceutical data [36].

Table 1: Performance Comparison of Drug-Target Classification Models

Model Architecture	Accuracy (%)	Computational Speed (s/sample)	Stability (±)	Key Advantages
optSAE + HSAPSO (Encoder)	95.52 [36]	0.010 [36]	0.003 [36]	High accuracy, stability, efficiency
MGMA-DTI (Hybrid)	94.60 [79]	N/A	N/A	Molecular interpretability
Traditional SVM	~89.98 [36]	>0.010	>0.003	Interpretability, feature importance
XGBoost	~93.78 [36]	>0.010	>0.003	Handling diverse feature types

Broader Performance Trends in Drug-Target Prediction

The performance advantages of encoder-focused architectures extend across multiple drug discovery applications. For druggable target prediction, models like DrugMiner (utilizing SVMs and neural networks) achieved 89.98% accuracy by leveraging 443 protein features [36]. More recent encoder-based approaches have consistently surpassed this benchmark, with methods like the Bagging-SVM ensemble with genetic algorithm feature selection reaching 93.78% accuracy [36]. These improvements highlight how encoder-oriented architectures better capture the complex relationships between molecular structures and their biological functions.

Table 2: Application Performance Across Drug Discovery Tasks

Application Domain	Model Type	Performance Metric	Result	Reference
Target Identification	Stacked Autoencoder + HSAPSO	Accuracy	95.52%	[36]
Drug-Target Interaction	MGMA-DTI	AUROC	94.60%	[79]
Resistance Prediction	SVM/XGBoost	MCC	0.812	[36]
Property Prediction	Encoder-only BERT-style	AUC	0.958	[36]

Implementation Protocols: Methodological Approaches

Experimental Workflow for Encoder-Only Drug Classification

The standard workflow for implementing encoder-only models in drug-target classification involves multiple stages of data processing, model training, and validation. The following diagram illustrates this comprehensive process:

Data Preparation and Feature Engineering Protocol

Data Sourcing and Curation:

• Molecular Data: Source drug compounds from specialized databases including DrugBank, ChEMBL, ZINC, and BindingDB [78] [80]. These resources provide structured information on drug-like small molecules, their bioactivities, and chemical properties.
• Target Protein Data: Obtain protein sequences and structural information from Swiss-Prot, Protein Data Bank (PDB), and other curated biological databases [36] [80].
• Interaction Data: Extract known drug-target interactions from benchmark datasets such as BindingDB, BioSNAP, and Human [79].

Feature Representation:

• Molecular Encoding: Convert SMILES representations into tokenized sequences using specialized chemical tokenizers that recognize molecular substructures [78]. Alternatively, generate molecular fingerprints (e.g., ECFP) or graph-based representations that preserve structural topology [80].
• Protein Encoding: Process amino acid sequences through k-mer tokenization or utilize pretrained protein language models (e.g., ProtBERT) to generate contextual embeddings [78].
• Interaction Representation: Create binary classification labels (interacting/non-interacting) or continuous values (binding affinity) based on experimental measurements [79].

Model Training and Optimization Protocol

Pretraining Phase:

• Implement masked language modeling by randomly masking 15% of tokens in molecular and protein sequences
• Use Adam optimizer with learning rate warmup and linear decay
• Train on large-scale unlabeled molecular and protein sequences to build foundational understanding [38]

Fine-tuning Phase:

• Add task-specific classification heads on top of pretrained encoder
• Employ gradual unfreezing strategies to prevent catastrophic forgetting
• Use balanced sampling or weighted loss functions to address class imbalance in interaction data [36]

Hyperparameter Optimization:

• Implement advanced optimization algorithms like Hierarchically Self-Adaptive PSO (HSAPSO) for efficient parameter tuning [36]
• Optimize critical parameters including learning rate (1e-5 to 1e-4), batch size (16-32), and number of attention heads (8-16)
• Utilize cross-validation with multiple random seeds to ensure result stability [36]

Implementing encoder-only models for drug-target classification requires specific computational resources, software tools, and datasets. The following table summarizes the essential components of the research toolkit:

Table 3: Essential Research Tools for Encoder-Based Drug-Target Classification

Resource Category	Specific Tools/Databases	Primary Function	Application Context
Molecular Databases	DrugBank, ChEMBL, ZINC [38] [80]	Source drug compounds and bioactivity data	Training data procurement
Protein Databases	Swiss-Prot, PDB, BindingDB [36] [80]	Protein sequences and structures	Target feature engineering
Benchmark Datasets	BioSNAP, Human, BindingDB [79]	Curated drug-target interactions	Model training and validation
Chemical Representation	SMILES, SELFIES, Molecular Graphs [38] [79]	Standardized molecular representations	Input feature generation
Deep Learning Frameworks	PyTorch, TensorFlow, Transformers	Model implementation	Architecture development
Specialized Libraries	RDKit, DeepChem, ChemBERTa [78] [79]	Cheminformatics and molecular ML	Preprocessing and modeling
Computational Resources	GPUs (NVIDIA A100/H100), TPUs	Accelerated model training	Handling large-scale biochemical data

Comparative Analysis: Encoder vs. Alternative Architectures

Architectural Advantages for Drug-Target Classification

Encoder-only models offer several distinct advantages for drug-target classification compared to other architectural paradigms:

• Bidirectional Context Understanding: Unlike decoder-only models that process information unidirectionally, encoder-only models leverage full bidirectional context, essential for understanding molecular interactions where spatial relationships matter more than sequential order [1].

• Efficient Representation Learning: Through pretraining on large unlabeled corpora of molecular and protein sequences, encoder models develop fundamental understanding of biochemical principles, which transfers effectively to specific classification tasks with limited labeled data [38].

• Computational Efficiency: For classification tasks, encoder-only models typically demonstrate faster inference times compared to encoder-decoder architectures, as they don't require autoregressive decoding [81].

The following diagram illustrates the architectural differences between encoder-only, decoder-only, and hybrid approaches in the context of drug-target classification:

Limitations and Considerations

Despite their advantages, encoder-only models present certain limitations that researchers should consider:

• Data Dependency: Performance is heavily dependent on the quality and diversity of training data. Biased or limited datasets can lead to poor generalization [36].

• Interpretability Challenges: While attention mechanisms provide some insight into model decisions, interpreting the precise biochemical rationale for predictions remains challenging [80].

• Computational Requirements: Pretraining encoder models requires substantial computational resources and large-scale molecular datasets, which may be prohibitive for some research groups [38].

Emerging Trends and Research Opportunities

The field of encoder-based drug-target classification continues to evolve rapidly, with several promising research directions emerging:

• Multimodal Integration: Future architectures will likely combine molecular structure data with additional modalities including gene expression profiles, protein-protein interaction networks, and clinical outcomes for more comprehensive prediction capabilities [80].

• 3D Structural Representation: Current models primarily use 1D (sequences) or 2D (molecular graphs) representations. Incorporating 3D structural information through geometric deep learning represents a significant opportunity for improving prediction accuracy [38].

• Transfer Learning Across Modalities: Developing encoder architectures that can transfer knowledge between different biological domains (e.g., from small molecules to proteins) could address data scarcity issues for novel target classes [78].

• Explainable AI Integration: Integrating encoder models with interpretability frameworks that provide biochemical rationale for predictions will be essential for building trust and facilitating experimental validation [79].

Encoder-only models have established themselves as powerful tools for high-accuracy drug-target classification, demonstrating superior performance compared to traditional machine learning approaches and specific advantages over alternative architectures for classification tasks. Through bidirectional processing and self-supervised pretraining, these models develop rich, contextual representations of molecular and protein sequences that translate effectively to precise interaction predictions.

The experimental evidence presented in this case study, particularly the 95.52% accuracy achieved by the optSAE+HSAPSO framework [36], underscores the transformative potential of encoder-based approaches in accelerating drug discovery. As the field advances, continued innovation in model architectures, training methodologies, and multimodal integration will further enhance the capabilities of these systems, ultimately contributing to more efficient and effective therapeutic development.

For researchers implementing these systems, success depends on thoughtful data curation, appropriate model selection based on specific task requirements, and rigorous validation using established benchmark datasets. By leveraging the protocols and resources outlined in this study, drug discovery professionals can harness the power of encoder-only models to advance their target identification and classification pipelines.

The integration of large language models (LLMs) into clinical workflows represents a frontier in medical artificial intelligence, with the potential to significantly reduce documentation burden. Within this domain, a key architectural divide exists between encoder-only, encoder-decoder, and decoder-only transformer models. This case study provides a comparative analysis of these architectures, with a specific focus on the application of decoder-only models for clinical note generation and summarization. Framed within broader materials research on architectural efficacy, we examine how decoder-only models are positioned against alternatives for transforming clinician-patient conversations into structured clinical documentation. Evidence suggests that while decoder-only models excel in generative tasks, encoder-based architectures maintain advantages in specific information extraction contexts, highlighting the need for task-specific model selection in clinical environments [82] [39].

Transformer-based architectures demonstrate specialized capabilities based on their structural design, which directly impacts their suitability for clinical language tasks.

• Encoder-Only Models (e.g., BERT, BioLinkBERT): Utilizing bidirectional attention, these models develop deep contextual understanding of input text. They are particularly well-suited for natural language understanding tasks such as named entity recognition (NER), relation extraction, and classification of medical concepts. Their strength lies in comprehending existing content rather than generating new text [82] [5]. Studies indicate that encoder-only models pre-trained on biomedical data, such as ClinicalBERT and BioBERT, are highly effective for structured tasks like medical chart extraction [82].

• Encoder-Decoder Models (e.g., T5, BART): These models combine an encoder for processing input sequences and a decoder for generating output sequences. This architecture is designed for sequence-to-sequence transformation tasks, including text summarization, machine translation, and question answering. In clinical contexts, they can be applied to convert dialogue transcripts into summarized clinical notes [82] [5]. Recent investigations suggest that encoder-decoder models, when enhanced with modern training techniques, can achieve performance competitive with decoder-only models while offering superior inference efficiency [4].

• Decoder-Only Models (e.g., GPT, LLaMA, CoMET): Built with autoregressive, unidirectional attention, these models are optimized for conditional text generation. They predict subsequent tokens based on preceding context, making them ideal for free-form generation, conversational AI, and few-shot learning. In clinical practice, this translates to generating progress notes, discharge summaries, and patient-facing narratives from prompts or conversation transcripts [82] [83] [5]. The Cosmos Medical Event Transformer (CoMET), a decoder-only model, exemplifies this by autoregressively generating future medical events to simulate patient health timelines [83].

Table 1: Transformer Architectures and Their Clinical Applications

Architecture	Key Features	Example Models	Primary Clinical Tasks
Encoder-Only	Bidirectional context understanding	BERT, BioBERT, BioLinkBERT	Named Entity Recognition, Data Extraction from EHRs, Medical Concept Classification
Encoder-Decoder	Sequence-to-sequence transformation	T5, BART	Text Summarization, Medical Translation, Structured Report Generation
Decoder-Only	Autoregressive text generation	GPT-series, LLaMA, CoMET, Gemma	Clinical Note Generation, Patient-facing Chatbots, Diagnostic Assistance, Medical Event Simulation

Performance Comparison and Experimental Data

Empirical evaluations reveal a nuanced performance landscape where no single architecture dominates all clinical tasks. The following table synthesizes key quantitative findings from recent comparative studies.

Table 2: Comparative Performance Metrics Across Model Architectures

Task	Model Architecture	Key Metric	Reported Score	Comparative Context
Named Entity Recognition	Encoder-Only (Flat NER)	F1-Score	0.87-0.88 (Pathology), 0.78 (Radiology)	Superior to decoder-only LLMs [39]
	Decoder-Only (Various LLMs)	F1-Score	0.18-0.30	High precision but poor recall [39]
Clinical Text Embedding	Encoder-Only (BioLinkBERT-LoRA)	Cardiology Separation Score	0.510	Best efficiency and performance [76]
	Decoder-Only (Gemma-2-2B-LoRA)	Cardiology Separation Score	0.455	Lower score with higher compute [76]
Clinical Summarization	Decoder-Only (GPT-4 with ISP)	BERTScore F1	0.8546	High semantic equivalence to reference [84]
		ROUGE-L F1	0.3077	Lower lexical overlap [84]
Medical Event Prediction	Decoder-Only (CoMET - 1B)	AUC-ROC	Generally outperformed task-specific models	Across 78 real-world tasks without fine-tuning [83]

Analysis of Comparative Performance

The data indicates a clear task-dependent performance hierarchy. For structured extraction tasks like NER, encoder-only models significantly outperform decoder-only LLMs. The latter often achieve high precision but suffer from critically low recall, making them "overly conservative" and unsuitable for comprehensive entity extraction from complex medical reports [39]. This performance gap is attributed to the fundamental architectural strengths of bidirectional encoders in text comprehension.

Conversely, for generative and predictive tasks, decoder-only models demonstrate formidable capability. The CoMET model, pretrained on a massive dataset of 115 billion medical events, matched or exceeded the performance of task-specific supervised models on 78 diverse clinical tasks, including diagnosis prediction and prognosis [83]. This showcases the power of scalable, generatively-trained decoder-only architectures to capture complex clinical dynamics. In summarization, while lexical overlap (ROUGE) might be moderate, the high semantic similarity (BERTScore) of outputs from decoder-only models like GPT-4 indicates an ability to produce logically paraphrased and clinically coherent summaries [84].

Efficiency is another differentiator. A controlled study on domain-adapted cardiology embeddings found that the top encoder model (BioLinkBERT, 340M parameters) not only achieved a higher separation score but also did so with a much smaller memory footprint (1.51 GB) and higher inference throughput (143.5 embeddings/sec) compared to a strong decoder model (Gemma-2-2B), which required 12.0 GB and operated at 55.5 embeddings/sec [76].

Detailed Experimental Protocols

To ensure reproducibility and critical appraisal, this section outlines the methodologies underpinning key experiments cited in this guide.

• Objective: To compare the performance of encoder-only and decoder-only models on the task of extracting clinical entities from unstructured pathology and radiology reports.
• Dataset: 2,013 annotated pathology reports and 413 annotated radiology reports.
• Models Evaluated:
  • Encoder-Only: Flat NER and Nested NER using transformer-based models.
  • Decoder-Only: Instruction-based NER using multiple LLMs.
• Methodology:
  • Training/Finetuning: Encoder-based models were fine-tuned on the annotated dataset. LLMs were evaluated in an instruction-following, few-shot, or zero-shot setting.
  • Evaluation: Standard NER evaluation using Precision, Recall, and F1-score was conducted on a held-out test set. The F1-score was the primary metric for comparison.
• Key Finding: Encoder-based NER models (flat and nested) were superior to LLM-based approaches, which achieved high precision but poor recall, leading to low F1-scores.

• Objective: To pretrain a family of decoder-only generative models on longitudinal patient event data and evaluate their predictive power on downstream clinical tasks.
• Dataset: The Epic Cosmos dataset, a filtered subset containing 115 billion discrete medical events from 118 million unique patient records, transformed into chronological sequences of tokenized events.
• Model: CoMET (Cosmos Medical Event Transformer), a series of decoder-only transformers based on the Qwen2 architecture, randomly initialized and pretrained from scratch. Models were scaled from ~150M to 1B parameters.
• Training:
  • Pretraining: Models were trained autoregressively to predict the next medical event token in a patient's sequence.
  • Scaling Laws: A large-scale study established compute-optimal model and dataset sizes, revealing power-law scaling relationships.
• Evaluation:
  • Inference: For a given patient history, the model generates multiple future event sequences (simulated timelines).
  • Tasks: Performance was evaluated on 78 real-world tasks, including diagnosis prediction, disease prognosis, and healthcare operations (e.g., length of stay prediction).
  • Metrics: Task-appropriate metrics such as AUC-ROC and PR-AUC were used. Performance was compared against task-specific supervised models (e.g., gradient-boosted decision trees).
• Key Finding: CoMET, with generic pretraining and simulation-based inference, generally matched or outperformed task-specific supervised models without requiring task-specific fine-tuning.

• Objective: To generate high-quality summaries of clinical case documents using large language models.
• Dataset: The MultiClinSUM shared task dataset, comprising 3,396 clinical case reports from various specialties.
• Model: Decoder-only models including GPT-4 and GPT-4o.
• Methodology - Iterative Self-Prompting (ISP):
  • Initialization: A meta-prompt was constructed combining Chain-of-Thought (CoT) instructions, clinical perspectives (e.g., symptoms, diagnoses, treatments), and metric-based guidance, along with few-shot examples.
  • Prompt Generation: The LLM was instructed to generate a new task-specific prompt based on the meta-prompt and examples.
  • Synthesis & Evaluation: The synthetic prompt was used to generate summaries for a portion of the training data. These were compared to ground-truth summaries using ROUGE and BERTscore.
  • Iteration: The evaluation scores and reflective feedback were fed back to the LLM to refine the prompt. This process was repeated until performance plateaued.
• Evaluation: The final prompt was used on the test set, with outputs evaluated automatically (ROUGE, BERTscore) and via qualitative analysis.
• Key Finding: The ISP technique enabled decoder-only models to produce summaries with high semantic fidelity (BERTscore F1 of 0.8546), despite lower lexical overlap (ROUGE-L F1 of 0.3077).

Workflow Visualization

The following diagram illustrates the typical workflow for training and applying a decoder-only model to the task of clinical note generation, as exemplified by methodologies like CoMET and iterative self-prompting.

The Scientist's Toolkit: Essential Research Reagents

The following table details key datasets, models, and evaluation frameworks that constitute the essential "reagent solutions" for research in this field.

Table 3: Key Research Reagents for Clinical NLP Experiments

Item Name	Type	Primary Function	Relevance to Research
aci-bench Corpus [85] [86]	Dataset	Benchmarking automatic clinical note generation from doctor-patient dialogue.	Provides the largest public dataset of clinic dialogue-note pairs for training and evaluating generative models.
Epic Cosmos Dataset [83]	Dataset	Pretraining large-scale medical foundation models.	A massive, de-identified longitudinal dataset of medical events enabling the training of models like CoMET.
MultiClinSUM Dataset [84]	Dataset & Benchmark	Evaluating multilingual clinical document summarization.	Offers a standardized testbed for assessing model performance on summarizing clinical case reports.
Decoder-Only Models (GPT, LLaMA)	Model Architecture	Conditional text generation and few-shot learning.	The primary architecture for generative tasks like note drafting and summarization.
Encoder-Only Models (BERT, BioLinkBERT)	Model Architecture	Text comprehension and information extraction.	The preferred choice for high-performance named entity recognition and data extraction from medical text.
ROUGE & BERTscore [84] [86]	Evaluation Metric	Automated assessment of generated text quality.	ROUGE measures lexical overlap, while BERTscore evaluates semantic similarity, providing a dual perspective on summary quality.
Iterative Self-Prompting (ISP) [84]	Methodology	Optimizing LLM performance without fine-tuning.	A technique to guide decoder-only models to produce higher-quality, more structured outputs through prompt engineering.
Low-Rank Adaptation (LoRA) [76]	Finetuning Technique	Parameter-efficient model adaptation.	Enables efficient fine-tuning of large models for specific clinical domains with reduced computational cost.

This guide provides an objective comparison of encoder-only, decoder-only, and encoder-decoder large language model (LLM) architectures, with a specific focus on their applications in drug development. By synthesizing current research, performance data, and practical use-cases, we deliver a strategic framework to help researchers and scientists select the optimal model architecture for key tasks in the pharmaceutical development pipeline, from early discovery to post-market surveillance.

The foundational transformer architecture has evolved into three distinct paradigms, each with unique mechanisms and strengths relevant to scientific inquiry.

Encoder-Only Models (e.g., BERT, RoBERTa) process input sequences bidirectionally. This means that when the model encounters a word or token, it has access to and can incorporate context from both the left and the right, creating a rich, contextual understanding of the entire input sequence [1] [16]. This is achieved through pre-training objectives like Masked Language Modeling (MLM), where random tokens in the input are masked and the model is trained to predict them based on their surrounding context [1]. This bidirectional nature makes encoders powerful for analysis and understanding tasks but not inherently suited for text generation.

Decoder-Only Models (e.g., GPT series, Llama) function autoregressively and unidirectionally. They process text sequentially from left to right, using a causal attention mask that prevents any token from attending to future tokens [1] [5]. Their primary pre-training task is Causal Language Modeling (CLM), or simply predicting the next token in a sequence [16]. This design is inherently generative, allowing decoder-only models to create coherent and contextually relevant text, code, and other sequences token-by-token.

Encoder-Decoder Models (e.g., T5, BART) combine both components. The encoder creates a comprehensive representation of the input sequence. The decoder, which is autoregressive like in decoder-only models, then uses this representation to generate the output sequence, often facilitated by a cross-attention mechanism [5] [87]. These models are often trained on denoising or span corruption objectives, where parts of the input are corrupted or masked, and the model is trained to recover the original text [87]. This architecture is specialized for sequence-to-sequence tasks where the output is a transformation of the input.

The diagram below illustrates the fundamental information flow and attention mechanisms of these three architectures.

Performance Comparison in Drug Development Tasks

The suitability of each architecture varies significantly across the drug development lifecycle. The following table summarizes their comparative performance on key tasks, supported by experimental evidence.

Model Architecture	Primary Strengths	Typical Drug Development Applications	Reported Performance & Experimental Findings
Encoder-Only (e.g., BERT, DeBERTa)	Bidirectional context understanding, superior for classification and information extraction tasks [1] [16].	- Named Entity Recognition (NER) for extracting chemical/disease names from literature [81].- Relation extraction (e.g., drug-target interactions).- Toxicity and property classification.	In a comparative analysis on challenging STEM MCQs, the encoder-only DeBERTa v3 Large demonstrated strong performance, outperforming the decoder-only Llama 2-7B in a question-answering task with context [22].
Decoder-Only (e.g., GPT-4, Llama 2, Mistral)	Autoregressive text generation, in-context learning, strong zero-shot and few-shot capabilities [1] [5].	- Generating hypotheses and research proposals.- Drafting clinical trial protocols and documentation.- Synthetic data generation for augmentation.- Powering conversational AI for scientific literature Q&A.	Mistral-7B Instruct was shown to be a strong performer, surpassing Llama 2-7B and showcasing the potential of smaller, fine-tuned decoder models when provided with appropriate context [22]. At sufficient scale, they achieve remarkable generalization [16].
Encoder-Decoder (e.g., T5, BART)	Effective at sequence-to-sequence tasks that require comprehension and transformation of input [5] [87].	- Text summarization (e.g., condensing a long research paper into an abstract).- Data transformation and standardization (e.g., reformatting assay results).- Question answering where the answer is generated from a given context.	Flan-T5 XXL (11B parameters) achieved an MMLU score of 55+, demonstrating robust performance for a model of its scale, particularly after instruction tuning [87]. It can be highly effective for single-task fine-tuning at smaller scales [87].

Experimental Protocol for Model Evaluation

The performance data cited in the comparison table, particularly from [22], stems from a rigorous experimental design focused on challenging, model-generated STEM multiple-choice questions (MCQs). The methodology can be summarized as follows:

• Task: Multiple-Choice Question Answering (MCQA) on a dataset of STEM MCQs generated by LLMs (e.g., Vicuna-13B, Bard, GPT-3.5) to create a self-evaluation benchmark.
• Models Evaluated:
  • Encoder-Only: DeBERTa v3 Large.
  • Decoder-Only: Llama 2-7B and Mistral-7B Instruct.
  • Closed-Source: GPT-4 and Gemini were also benchmarked for comparison.
• Methodology:
  • Inference with Context: Models were provided with the question and context (relevant knowledge from Wikipedia) and evaluated on their answer selection.
  • Fine-Tuning: Models were fine-tuned on the dataset both with and without additional context.
• Key Metric: Accuracy in selecting the correct answer from multiple choices.

This protocol highlights the importance of both model architecture and training strategy (e.g., providing context and fine-tuning) in achieving high performance on complex, domain-specific tasks.

The Scientist's Toolkit: Research Reagent Solutions

Selecting and working with these architectures requires a suite of tools and frameworks. The following table details the essential components of a modern LLM research pipeline.

Tool / Resource	Function	Relevance to Drug Development
Hugging Face Transformers	A library providing pre-trained models and scripts for encoder, decoder, and encoder-decoder architectures.	The primary platform for accessing and fine-tuning state-of-the-art models (e.g., BioBERT, SciBERT, PMC-LLaMA) on proprietary biomedical data.
FLAN Collection	A set of instruction-tuned models (e.g., Flan-T5) trained on a massive collection of tasks.	Provides a strong foundation for multi-task learning and instruction-following in scientific domains, reducing the need for extensive task-specific fine-tuning.
Quantization (e.g., GPTQ, GGUF)	Techniques to reduce the memory footprint of LLMs by lowering the precision of their weights.	Enables the deployment and inference of large models (e.g., 7B+ parameter models) on local hardware, such as a researcher's workstation, ensuring data privacy.
Parameter-Efficient Fine-Tuning (PEFT)	Methods like LoRA (Low-Rank Adaptation) that fine-tune a small number of parameters instead of the full model.	Drastically reduces computational cost, allowing researchers to efficiently adapt large base models to specific, narrow tasks like adverse event report classification.
Benchmarks (e.g., MMLU, BLURB)	Standardized evaluations for general and biomedical language understanding.	Critical for objectively comparing the performance of different architectures and fine-tuned models on a common set of tasks relevant to biology and medicine.

Strategic Decision Matrix for Drug Developers

The choice of architecture is not one-size-fits-all but should be driven by the specific Question of Interest (QOI) and Context of Use (COU) within the drug development pipeline [88]. The following decision matrix provides a strategic framework for this selection.

Application in Model-Informed Drug Development (MIDD)

The decision matrix aligns with the "Fit-for-Purpose" philosophy in Model-Informed Drug Development (MIDD), which emphasizes closely aligning tools with key questions and contexts of use [88]. Here is how each architecture maps to specific MIDD stages:

• Early Discovery (Target ID, Lead Optimization): Encoder-only models excel at extracting relationships between genes, proteins, and compounds from vast literature and database sources, directly supporting target identification and Quantitative Structure-Activity Relationship (QSAR) modeling [88].
• Preclinical & Clinical Development: Encoder-decoder models are well-suited for summarizing preclinical findings or generating first drafts of clinical trial documents based on protocol outlines. Decoder-only models can be used to simulate patient consent forms or generate synthetic data for trial design simulation.
• Regulatory Submission & Post-Market: Decoder-only models can assist in drafting sections of regulatory submission documents. Both encoder-only and encoder-decoder models can power tools for analyzing post-market safety reports, identifying potential adverse events, and summarizing real-world evidence.

The architectural landscape of LLMs offers powerful but distinct tools for accelerating drug development. Encoder-only models provide deep, bidirectional understanding for data extraction and analysis. Decoder-only models offer unparalleled flexibility and generative capability for ideation and content creation. Encoder-decoder models remain the specialists for tasks requiring direct sequence transformation. The optimal choice is not inherent superiority of one architecture, but strategic alignment with the specific task, data constraints, and desired outcome, guided by a fit-for-purpose principle. As these models continue to evolve, particularly with scaling and multi-objective training, the boundaries between them may blur, but their core architectural strengths will continue to inform their strategic application in pharmaceutical research.

Evaluating Clinical Readiness and Integration into Existing Workflows

The transition of artificial intelligence (AI) from research environments to clinical and materials discovery workflows demands robust, efficient, and interpretable models. Within large language models (LLMs), a significant architectural divide exists: encoder-only models, excelling in comprehension and classification; decoder-only models, dominating text generation; and encoder-decoder models, designed for sequence-to-sequence transformation tasks. Framed within a broader thesis on encoder versus decoder architectures for materials research, this guide objectively compares their performance, supported by experimental data, to evaluate their clinical readiness and potential for seamless integration into established scientific workflows. Understanding the inherent strengths of each architecture is crucial for deploying effective AI tools in high-stakes environments like drug development and clinical prediction systems.

The fundamental differences between model architectures dictate their suitability for specific tasks in scientific and clinical contexts.

Encoder-only models, such as BERT and RoBERTa, utilize bidirectional self-attention to process entire input sequences simultaneously [1] [89]. This allows them to develop a deep understanding of context from both left and right surroundings of any token. They are typically pre-trained using Masked Language Modeling (MLM), where random tokens in the input are masked and the model learns to predict them [1]. This makes them powerful for tasks requiring deep semantic understanding, such as named entity recognition, relation extraction from scientific literature, and classifying patient data [1] [89].

Decoder-only models, including the GPT family and LLaMA, employ causal (autoregressive) self-attention [1] [89]. This mechanism restricts the model from attending to future tokens, ensuring that predictions for position i depend only on known outputs at positions less than i. Trained with Causal Language Modeling (CLM) to predict the next token in a sequence, they excel at open-ended generation tasks [89]. In scientific settings, this facilitates activities like generating hypotheses, creating research summaries, and de novo molecular design.

Encoder-decoder models (or sequence-to-sequence models) hybridize both components [1]. The encoder processes the input sequence into a dense, contextual representation. The decoder then generates the output sequence autoregressively, using both its own previous outputs and the encoder's representation through cross-attention mechanisms [90] [1]. Architectures like T5 and BART are pre-trained with objectives that map an input sequence to an output sequence, making them ideal for tasks like text summarization, machine translation, and – critically – predicting future clinical events from historical patient data [90] [89].

The diagram below illustrates the core information flow and attention mechanisms in these architectures.

Figure 1: Core architecture and information flow in encoder-only, decoder-only, and encoder-decoder models. Encoder-only models process bidirectional context for understanding tasks. Decoder-only models use past context for generation. Encoder-decoder models transform an input sequence into an output sequence via a contextual bridge.

Performance Comparison in Scientific and Clinical Tasks

Rigorous benchmarking across diverse tasks is essential to evaluate the practical utility of these architectures. The following tables summarize quantitative performance data from key experiments in clinical and molecular research.

Table 1: Performance comparison on clinical prediction tasks (Adapted from TransformEHR study [90])

Model Architecture	Task	Evaluation Metric	Performance	Key Strength
Encoder-Decoder (TransformEHR)	Pancreatic Cancer Onset	AUPRC	2% improvement (p<0.001) vs previous SOTA	High precision in rare event prediction
Encoder-Decoder (TransformEHR)	Intentional Self-Harm (in PTSD patients)	AUPRC	24% improvement (p=0.007) vs previous SOTA	Effective clinical intervention screening (PPV: 8.8%)
Encoder-Decoder (TransformEHR)	Uncommon ICD-10 Code Prediction	AUPRC	Substantial improvements vs encoder-only BERT	Handling of rare, complex medical codes
Encoder-Only (DeBERTa v3 Large)	Challenging STEM MCQs	Accuracy	Outperformed decoder-only Llama 2-7B [22]	Superior classification with appropriate context
Decoder-Only (Mistral-7B Instruct)	Challenging STEM MCQs	Accuracy	Competitive performance with fine-tuning [22]	Strong few-shot learning capabilities

AUPRC: Area Under the Precision-Recall Curve; PPV: Positive Predictive Value; SOTA: State-of-the-Art.

Table 2: Performance comparison on molecular property prediction and generation (Adapted from SMI-TED289M and related studies [38] [24])

Model Architecture	Task / Dataset	Evaluation Metric	Performance	Notes
Encoder-Decoder (SMI-TED289M Fine-tuned)	MoleculeNet Classification (4/6 datasets)	ROC-AUC / Accuracy	Superior to existing SOTA [24]	Effective representation learning
Encoder-Decoder (SMI-TED289M Fine-tuned)	QM9, QM8, ESOL, FreeSolv, Lipophilicity	MAE / RMSE	Outperformed competitors in all 5 regression tasks [24]	High accuracy for quantum property prediction
Encoder-Decoder (MoE-OSMI, 8x289M)	Various Molecular Tasks	Multiple Metrics	Consistently higher than single SMI-TED289M [24]	Scalability via Mixture-of-Experts
Encoder-Decoder (SMI-TED289M)	MOSES Scaffold Test Set	Reconstruction/Generation Metrics	Generated previously unobserved scaffolds [24]	Demonstrated generalization for de novo design
Decoder-Only Models (e.g., GPT)	Property Prediction from 2D SMILES	Variable	Becoming more prevalent [38]	Leverage generative pre-training

Analysis of Comparative Performance

The data reveals a nuanced landscape. Encoder-decoder models demonstrate compelling advantages in clinical settings and structured scientific tasks requiring precise input-to-output transformation. TransformEHR's significant performance leap in predicting intentional self-harm—a complex outcome involving numerous correlated factors—showcases its ability to uncover intricate interrelations among different diagnoses [90]. Similarly, in molecular science, the SMI-TED289M family's state-of-the-art results across classification and regression tasks highlight the efficacy of its encoder-decoder pre-training for learning rich, chemically meaningful representations [24].

Decoder-only models remain dominant in pure content generation and exhibit remarkable few-shot learning capabilities [1] [89]. However, studies note potential limitations like "attention degeneration," where the model's focus on the source input diminishes as generation progresses, potentially impacting reliability in long, complex sequence generation for scientific reports [91].

Encoder-only models like DeBERTa maintain strong positions in classification-heavy tasks where deep, bidirectional understanding of input text is paramount, and where generative capabilities are not required [22] [1].

Detailed Experimental Protocols and Methodologies

To assess the clinical readiness and practical performance of these models, rigorous and reproducible experimental protocols are essential. Below are the detailed methodologies from two pivotal studies cited in this comparison.

Protocol 1: Clinical Prediction with TransformEHR

The TransformEHR study established a new state-of-the-art for predicting future clinical events from Electronic Health Records (EHR) using an encoder-decoder architecture [90].

• Objective: To pretrain a generative encoder-decoder model that can predict all diseases and outcomes of a patient's future visit from previous visits and be finetuned for specific clinical prediction tasks.
• Dataset: A pretraining cohort of 6.5 million patients from the US Veterans Health Administration (VHA) with EHR data from 2016-2012019. Evaluation used internal (unseen VHA facilities) and external (MIMIC-IV) validation sets [90].
• Input Representation: Longitudinal EHRs including demographic data (gender, age, race, marital status) and ICD-10CM diagnostic codes grouped at the visit level. The specific date of each visit was incorporated as a feature to model temporal importance [90].
• Model Architecture (TransformEHR): Transformer-based encoder-decoder.
  • Encoder: Processes the sequence of previous visits to build a contextualized representation.
  • Decoder: Autoregressively generates the set of ICD codes for a future visit. It uses cross-attention to identify relevant codes from previous visits and self-attention on already-generated codes to predict the next one [90].
• Pretraining Objective: A novel pretraining task of predicting the complete set of ICD-10CM codes for the patient's next visit based on all previous visits. This denoising sequence-to-sequence objective helps the model learn complex interrelations among diseases [90].
• Finetuning: The pretrained model was subsequently finetuned on specific, lower-prevalence tasks like pancreatic cancer prediction and intentional self-harm among PTSD patients using smaller, task-specific labeled datasets [90].

The workflow for this protocol is visualized below.

Figure 2: The TransformEHR experimental workflow. The model is pre-trained on a novel objective of predicting a patient's complete future diagnostic codes from their medical history, then fine-tuned for specific clinical predictions.

Protocol 2: Molecular Property Prediction with SMI-TED289M

The SMI-TED289M study provides a benchmark for encoder-decoder models in chemistry, demonstrating state-of-the-art results on diverse molecular tasks [24].

• Objective: To pre-train a large-scale encoder-decoder foundation model on molecular sequences (SMILES) that can be adapted for property prediction, reaction outcome forecasting, and molecular reconstruction.
• Dataset: A curated dataset of 91 million molecular SMILES sequences from PubChem. The model was evaluated on 11 benchmark datasets from MoleculeNet (for property prediction) and the MOSES dataset (for generation and reconstruction) [24].
• Input Representation: Molecular structures represented as SMILES strings, a character string notation describing the molecular graph.
• Model Architecture (SMI-TED289M): Transformer-based encoder-decoder.
  • Pre-training Objective: A decoder-based reconstruction objective, where the model learns to reconstruct the input SMILES sequence. This self-supervised task forces the model to learn a chemically meaningful latent space [24].
  • Pooling Function: A novel pooling function that differs from standard max or mean pooling, designed to better reconstruct SMILES and preserve molecular properties [24].
• Evaluation:
  • Property Prediction: The pre-trained model was fine-tuned on individual MoleculeNet datasets for classification (e.g., toxicity) and regression (e.g., quantum mechanical properties) tasks.
  • Reconstruction/Generation: The model's decoder capacity was tested on the MOSES scaffold test set, which contains unique molecular scaffolds not seen during training, to evaluate its generalization for de novo molecular design [24].
  • Latent Space Analysis: The structure of the learned embeddings was assessed for few-shot learning capability and for separating molecules based on chemically relevant features (e.g., electron-donating effects on HOMO energy) [24].

Successful implementation of these models relies on specific datasets, software tools, and computational resources. The following table details key "research reagents" for replicating studies or developing new applications.

Table 3: Essential resources for developing and evaluating clinical and scientific LLMs

Resource Name / Type	Function / Purpose	Relevant Architecture	Example in Literature
Longitudinal EHR Datasets	Pre-training and fine-tuning for clinical prediction models.	Encoder-Decoder, Encoder-Only	VHA dataset (6.5M patients) [90], MIMIC-IV [90]
Structured Molecular Databases	Source of SMILES strings or other representations for pre-training chemical models.	Encoder-Decoder, Decoder-Only	PubChem (91M molecules) [24], ZINC, ChEMBL [38]
Benchmarking Suites (MoleculeNet)	Standardized evaluation for molecular property prediction across multiple tasks.	All	11 MoleculeNet datasets for classification/regression [24]
Benchmarking Suites (MOSES)	Evaluation of molecular generation quality, validity, novelty, and uniqueness.	Encoder-Decoder, Decoder-Only	MOSES dataset with scaffold test set [24]
Transformer Framework (e.g., Hugging Face)	Open-source libraries providing model architectures, pre-training weights, and fine-tuning scripts.	All	Base implementations of BERT (encoder), GPT (decoder), T5 (encoder-decoder)
Mixture-of-Experts (MoE) Systems	Scaling model capacity efficiently by activating different model "experts" for different inputs.	Primarily Encoder-Decoder	MoE-OSMI (8x289M experts) [24]

Discussion: Clinical Readiness and Workflow Integration

The experimental evidence points to a context-dependent verdict on clinical readiness. Encoder-decoder models like TransformEHR and SMI-TED289M exhibit high readiness for tasks that mirror their pre-training objective: transforming complex, structured input into a structured output. Their demonstrated success in predicting future clinical events and molecular properties indicates they can be integrated into workflows as decision-support tools, for example, by flagging high-risk patients for intervention or prioritizing novel molecular candidates for synthesis [90] [24]. The encoder-decoder architecture's efficiency during inference, as highlighted in scaling studies, is a non-trivial advantage for deployment in resource-conscious clinical environments [4].

Decoder-only models, while powerful generators, face challenges for direct clinical integration due to concerns like attention degeneration and the potential for "hallucination" in mission-critical settings [91]. Their most immediate readiness may be in assisting research workflows—such as generating literature summaries or drafting hypotheses—rather than in direct patient-facing diagnostic applications.

Encoder-only models remain highly ready for classification-based tasks embedded within clinical and research workflows, such as automatically coding patient notes or extracting specific entities from scientific literature [22] [1].

Ultimately, the "best" architecture is dictated by the specific task. The trend towards hybrid and specialized models, such as Mixture-of-Experts, indicates a future where the architectures themselves become more modular and adaptable, potentially unlocking new levels of integration and utility in the complex environments of drug development and clinical care.

Conclusion

The choice between encoder-only and decoder-only models is not about finding a universal winner, but about strategic alignment with specific tasks in the drug discovery pipeline. Encoder-only models, with their bidirectional understanding, offer unmatched efficiency and accuracy for classification, data extraction, and target identification. Decoder-only models excel as generative engines for molecular design, content creation, and conversational AI. The future of AI in biomedicine lies not in a single architecture, but in leveraging their complementary strengths—potentially through hybrid or integrated systems—to build more powerful, efficient, and reliable tools. This will ultimately reduce development timelines and costs, accelerating the delivery of new therapies to patients.

References

• 1. Understanding Encoder And Decoder LLMs - Ahead of AI [https://magazine.sebastianraschka.com/p/understanding-encoder-and-decoder]
• 2. Decoder-Only Transformers: The Workhorse of Generative LLMs [https://cameronrwolfe.substack.com/p/decoder-only-transformers-the-workhorse]
• 3. Foundation models in drug discovery: phenomenal growth ... [https://www.sciencedirect.com/science/article/pii/S1359644625002314]
• 4. Encoder-Decoder or Decoder-Only? Revisiting ... [https://arxiv.org/abs/2510.26622]
• 5. LLM 9: Encoder-Decoder Models vs. Decoder-Only Models [https://medium.com/@goutami0318/llm-9-encoder-decoder-models-vs-decoder-only-models-0248c92739e3]
• 6. How has DeepSeek improved the Transformer architecture? [https://epoch.ai/gradient-updates/how-has-deepseek-improved-the-transformer-architecture]
• 7. A Comprehensive Study of Decoder-Only LLMs for Text-to- ... [https://arxiv.org/abs/2506.08210]
• 8. Achieving More with Less in Transformer Architectures [https://arxiv.org/abs/2507.15773]
• 9. Encoder Only Architecture: BERT [https://medium.com/@pickleprat/encoder-only-architecture-bert-4b27f9c76860]
• 10. What is an encoder-decoder model? [https://www.ibm.com/think/topics/encoder-decoder-model]
• 11. Transformer Architectures - Hugging Face LLM Course [https://huggingface.co/learn/llm-course/en/chapter1/6]
• 12. Finally, a Replacement for BERT: Introducing ModernBERT [https://huggingface.co/blog/modernbert]
• 13. A Comparative Analysis of Encoder and Decoder Only ... Only Models [https://www.researchsquare.com/article/rs-3865391/v1]
• 14. Enhancement Report Approval Prediction: A Comparative Study of... [https://arxiv.org/html/2506.15098v1]
• 15. deep learning - How does the (decoder-only) transformer architecture work? - Artificial Intelligence Stack Exchange [https://ai.stackexchange.com/questions/40179/how-does-the-decoder-only-transformer-architecture-work]
• 16. The Comparison between the Encoder and the Decoder [https://towardsai.net/p/machine-learning/the-comparison-between-the-encoder-and-the-decoder]
• 17. Understanding LLM Architectures: Decoder-Only, Encoder ... [https://www.linkedin.com/posts/brijpandeyji_large-language-models-llms-may-look-similar-activity-7376114094786895872-vaJl]
• 18. [2506.23431v2] Pipelined Decoder for Efficient Context-Aware Text Generation [https://arxiv.org/abs/2506.23431v2]
• 19. [2501.13598] A Transformer-based Autoregressive Decoder Architecture for Hierarchical Text Classification [https://arxiv.org/abs/2501.13598]
• 20. [2509.03498] OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation [https://arxiv.org/abs/2509.03498]
• 21. How to Choose the Right LLM in 2025? [https://marutitech.com/how-to-choose-right-llm/]
• 22. A comparative analysis of encoder only and decoder ... [https://www.sciencedirect.com/science/article/pii/S294971912500007X]
• 23. Encoder vs. Decoder: Understanding the Two Halves of ... [https://www.linkedin.com/pulse/encoder-vs-decoder-understanding-two-halves-transformer-anshuman-jha-bkawc]
• 24. An open-source family of large encoder-decoder ... [https://www.nature.com/articles/s42004-025-01585-0]
• 25. How Decoder-Only Models Understand and Translate ... [https://medium.com/@dewasheesh.rana/from-transformers-to-gpts-how-decoder-only-models-understand-and-translate-without-an-encoder-ed112aceb744]
• 26. Encoders-Only Models: Workhorses of Practical Language ... [https://jedleee.medium.com/encoders-only-models-workhorses-of-practical-language-processing-modernbert-case-study-355048a92cef]
• 27. A Primer on Decoder-Only vs Encoder-Decoder Models for ... [https://slator.com/primer-on-decoder-only-vs-encoder-decoder-models-ai-translation/]
• 28. Encoder-Decoder or Decoder-Only? Revisiting ... [https://arxiv.org/html/2510.26622v1]
• 29. DeBERTaV3 Remains a Top Performer Among Encoder-Only ... [https://thesalt.substack.com/p/debertav3-remains-a-top-performer]
• 30. Decoder-only Foundation Model [https://www.emergentmind.com/topics/decoder-only-foundation-model]
• 31. Top 10 open source LLMs for 2025 [https://www.instaclustr.com/education/open-source-ai/top-10-open-source-llms-for-2025/]
• 32. Top 9 Large Language Models as of November 2025 [https://www.shakudo.io/blog/top-9-large-language-models]
• 33. 27 of the best large language models in 2025 [https://www.techtarget.com/whatis/feature/12-of-the-best-large-language-models]
• 34. Artificial intelligence for drug target identification [https://www.sciencedirect.com/science/article/pii/S0065774325000028]
• 35. Beyond Decoder-only: Large Language Models Can be ... [https://aclanthology.org/2025.findings-acl.490/]
• 36. Automated drug design for druggable target identification ... [https://www.nature.com/articles/s41598-025-18091-x]
• 37. BioClinical ModernBERT: A State-of-the-Art Long-Context ... [https://arxiv.org/html/2506.10896v1]
• 38. Foundation models for materials discovery – current state ... [https://www.nature.com/articles/s41524-025-01538-0]
• 39. Comparing encoder-only vs. large language models in a ... [https://www.sciencedirect.com/science/article/pii/S0010482525010169]
• 40. Comparing encoder-only vs. large language models in a ... [https://pubmed.ncbi.nlm.nih.gov/40554973/]
• 41. TEFDTA: a transformer encoder and fingerprint representation ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC10777355/]
• 42. TransformEHR: transformer-based encoder-decoder ... [https://www.nature.com/articles/s41467-023-43715-z]
• 43. Are Decoder-Only Language Models Better than Encoder- ... [https://aclanthology.org/2024.findings-acl.967/]
• 44. GP-MoLFormer: a foundation model for molecular generation - Digital Discovery (RSC Publishing) DOI:10.1039/D5DD00122F [https://pubs.rsc.org/en/content/articlehtml/2025/dd/d5dd00122f]
• 45. De Novo Molecular Design Enabled by Direct Preference ... [https://arxiv.org/html/2504.01389v1]
• 46. Code‑driven Interpretable Molecular Optimization [https://arxiv.org/html/2510.14455v1]
• 47. Open Molecular Crystals 2025 (OMC25) Dataset and Models [https://ai.meta.com/research/publications/open-molecular-crystals-2025-omc25-dataset-and-models/]
• 48. What are decoder-only models vs. encoder-decoder models? [https://milvus.io/ai-quick-reference/what-are-decoderonly-models-vs-encoderdecoder-models]
• 49. Comparative analysis of large language models in clinical ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12161448/]
• 50. The Future of AI in Drug Development: 10 Trends That Will ... [https://huspi.com/blog-open/the-future-of-ai-in-drug-development-10-trends-that-will-redefine-rd/]
• 51. The Roadmap for Mastering Language Models in 2025 [https://machinelearningmastery.com/the-roadmap-for-mastering-language-models-in-2025/]
• 52. the different Transformer-based Compare ... - AIML.com model [https://aiml.com/compare-the-different-transformer-based-model-architectures/]
• 53. - Decoder . Only – vs : What’s the Difference in LLM... Encoder Decoder [https://www.linkedin.com/pulse/decoder-only-vs-encoderdecoder-whats-difference-llm-sharone-volk-wufyf]
• 54. T5Gemma: A new collection of encoder-decoder Gemma ... [https://developers.googleblog.com/en/t5gemma/]
• 55. A comparative analysis of encoder and only in... decoder only models [https://link.springer.com/article/10.1007/s10579-024-09796-y]
• 56. Ettin: Analyzing Encoders vs Decoders Using the Same Architecture and Data [https://arxiv.org/html/2507.11412v1]
• 57. MedGemma: Our most capable open models for health AI ... [https://research.google/blog/medgemma-our-most-capable-open-models-for-health-ai-development/]
• 58. Application of the Bidirectional Encoder Representations from ... [https://formative.jmir.org/2025/1/e67311]
• 59. BioVERSE: Representation Alignment of Biomedical ... [https://arxiv.org/html/2510.01428v1]
• 60. Current and future state of evaluation of large language ... [https://www.nature.com/articles/s44401-024-00011-2]
• 61. Large Language Models Hallucination: A Comprehensive ... [https://arxiv.org/html/2510.06265v2]
• 62. From Illusion to Insight: A Taxonomic Survey of ... [https://www.mdpi.com/2673-2688/6/10/260]
• 63. LLM Hallucinations 101 [https://neptune.ai/blog/llm-hallucinations]
• 64. Exploring Advanced Techniques to Mitigate LLM ... [https://huggingface.co/blog/Imama/pr]
• 65. Mitigating spatial hallucination in large language models ... [https://www.nature.com/articles/s41598-025-93601-5]
• 66. Mitigating Hallucinated Translations in Large Language ... [https://arxiv.org/abs/2501.17295]
• 67. A Self-supervised Framework for Improving Faithfulness in ... [https://arxiv.org/html/2502.13674v1]
• 68. Turning Unpaired Data into High-Fidelity Pairs for Low- ... [https://aclanthology.org/2025.emnlp-main.1030/]
• 69. Paired by the Teacher: Turning Unpaired Data into High ... [https://arxiv.org/html/2509.25144v1]
• 70. How self-supervised learning revolutionized natural ... [https://stackoverflow.blog/2025/04/28/how-self-supervised-language-revolutionized-natural-language-processing-and-gen-ai/]
• 71. Why decoder-only models are dominant and when to use ... [https://www.linkedin.com/posts/asif-tandel_why-are-decoder-only-models-dominant-today-activity-7388626730661347329-shif]
• 72. AI Compute Demand in Biotech: 2025 Report & Statistics [https://intuitionlabs.ai/articles/ai-compute-demand-biotech]
• 73. The 2025 AI Coding Models: Comprehensive Guide to ... [https://www.codegpt.co/blog/ai-coding-models-2025-comprehensive-guide]
• 74. The Rise of LLMs: From GPT to Modern Innovations [https://medium.com/@lmpo/decoder-only-transformer-models-a-comprehensive-overview-c54dc3286d71]
• 75. Decoder-Only vs Encoder-Decoder: AI Models Explained [https://www.linkedin.com/posts/shivam-chakravarti-nitw_generativeai-llm-gpt-activity-7370507704458661890-nCob]
• 76. Comparative Evaluation of Encoder-Only and Decoder ... [https://arxiv.org/html/2511.19739v1]
• 77. The F1 Score: A Comprehensive Guide for 2025 [https://www.shadecoder.com/topics/the-f1-score-a-comprehensive-guide-for-2025]
• 78. A survey on deep learning for drug-target binding prediction [https://pmc.ncbi.nlm.nih.gov/articles/PMC12451107/]
• 79. MGMA-DTI: Drug target interaction prediction using multi ... [https://www.sciencedirect.com/science/article/abs/pii/S1476927125001094]
• 80. Machine Learning for Multi-Target Drug Discovery [https://pmc.ncbi.nlm.nih.gov/articles/PMC12473769/]
• 81. Encoder-Decoder vs. Decoder-Only [https://medium.com/@qmsoqm2/auto-regressive-vs-sequence-to-sequence-d7362eda001e]
• 82. Clinical applications of large language models in medicine ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12227933/]
• 83. Generative Medical Event Models Improve with Scale [https://arxiv.org/html/2508.12104v1]
• 84. MaLei at MultiClinSUM: Summarisation of Clinical ... [https://arxiv.org/html/2509.07622v1]
• 85. Aci-bench: a Novel Ambient Clinical Intelligence Dataset ... [https://www.nature.com/articles/s41597-023-02487-3]
• 86. Benchmarking And Datasets For Ambient Clinical ... [https://www.medrxiv.org/content/10.1101/2025.01.29.25320859v1.full-text?utm_source=chatgpt.com]
• 87. What happened to BERT & T5? On Transformer Encoders, ... [https://www.yitay.net/blog/model-architecture-blogpost-encoders-prefixlm-denoising]
• 88. Advancing drug development with “Fit-for-Purpose” modeling ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12436579/]
• 89. How LLMs work: Encoder-only, Decoder-only, and ... [https://www.linkedin.com/posts/k-vamsi-86030116b_ai-machinelearning-deeplearning-activity-7377731962335088640-uy3I]
• 90. TransformEHR: transformer-based encoder-decoder ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC10687211/]
• 91. Decoder-Only or Encoder-Decoder? Interpreting Language ... [https://arxiv.org/abs/2304.04052]