To answer that, we tested 8 LLM training methods. Here's what actually worked.
Our experiments covered three types of training:
- Pretraining Methods: Building domain knowledge before task-specific training
- Fine-Tuning Methods: Adapting models to specific tasks and formats
- Alignment Methods: Optimizing models to match human preferences and values
Each type has different purposes and trade-offs. As part of our experiments, we evaluated all 8 methods on 8,877 Malaysian finance questions.
Our Results: All Methods at a Glance
Here's our comprehensive comparison across all 8 methods we tested:
| Method | Category | Data Required | Training Resources | Training Speed | Key Advantage | Best For |
|---|---|---|---|---|---|---|
| DAPT | Pretraining | Large domain corpus | Very High | Slow | Broad domain knowledge | Specialized domains with unique terminology |
| TAPT | Pretraining | Domain data + task-focused text | High | Medium | Bridges domain to specific task | Preparing for task-specific fine-tuning |
| FFT | Fine-Tuning | Task-specific Q&A pairs | High | Medium | Strong adaptation (updates all weights) | Standard task adaptation |
| LoRA | Fine-Tuning | Task-specific Q&A pairs | Low | Fast | Parameter efficient (1-2% weights) | Resource-constrained scenarios |
| KD | Fine-Tuning | Labeled or unlabeled + teacher | Medium | Medium | Model compression | Edge deployment, latency reduction |
| RLHF (PPO) | Alignment | Preference Q&A pairs + unlabeled | Very High | Very Slow | Reward model can score unlabeled data | When you have lots of unlabeled data |
| RLHF (DPO) | Alignment | Preference Q&A pairs only | Medium | Medium | Simpler alternative to PPO, no reward model | Constrained compute, preference alignment |
| RLAIF | Alignment | AI-generated preference Q&A pairs | High | Very Slow | Scalable feedback (faster data collection) | Large-scale alignment |
Critical Insights: What We Learned Across All 8 Methods
Through implementing all 8 training methods, we identified key insights every practitioner should know:
1. Language Gap Amplifies DAPT's Value When your base model has limited exposure to your target language (<1% of training data), DAPT becomes essential, not optional. You're filling both a language gap and domain knowledge gap simultaneously.
2. TAPT Bridges Domain to Task, But Validate ROI First TAPT adds an intermediate training stage to narrow the gap between domain knowledge and specific tasks. However, if the gap is already small, direct approaches (DAPT or fine-tuning) often prove more effective. Validate the ROI before investing compute.
3. LoRA Delivers Competitive Results 30% Faster Parameter-efficient fine-tuning (LoRA) achieves competitive performance while updating only 1-2% of weights. In our experiments, LoRA trained 1.3x faster than FFT with comparable task performance. With larger models, this efficiency gain becomes even more pronounced. When rapid experimentation or budget constraints matter, efficiency often beats that last bit of performance.
4. KD Enables Model Compression, But Requires Strong Teachers Knowledge distillation compresses large models for faster inference and edge deployment. However, teacher quality is paramount:students learn output style but can lose factual knowledge. In our experiments, the teacher achieved only 35.6% accuracy. After distillation, the student dropped to 16.9% accuracy, worse than the untrained base model (21.2%). A mediocre teacher creates a worse student.
5. RLHF-PPO Leverages Unlabeled Data via Reward Model PPO's key strength is the reward model can score any generated response, enabling improvement from unlabeled data. In one of our experiments, the reward model tended to favor longer responses:chosen examples averaged 7-11x longer than rejected. This led to catastrophic forgetting as PPO weight updates optimized for verbosity, overwriting FFT's factual knowledge: questions FFT answered correctly became wrong after RLHF. Ensure your preference data captures actual quality, not just length or formatting.
6. RLHF-DPO Offers Simpler Pipeline Without Reward Model DPO requires only 2 models (policy and reference) compared to PPO's 4, eliminating the reward model and value model entirely. This makes implementation faster and more accessible. However, it still demands substantial preference data:industry leaders use ~33K pairs (OpenAI's InstructGPT, Ouyang et al., 2022) to ~161K pairs (Anthropic's Claude, Bai et al., 2022). Preference quality matters:verbose ≠ correct.
7. RLAIF Scales Preference Generation via AI Judges RLAIF solves the human labeling bottleneck by using AI judges to generate preference pairs at scale:thousands of comparisons without human annotators. However, it inherits the AI judge's biases. The same alignment challenges persist; RLAIF makes them more affordable to encounter, not easier to solve.
Our Bottom Line: There's No Silver Bullet
After extensive experiments across 8 different training methods, here's what we learned:
The LLM training landscape offers a spectrum of methods, each with distinct trade-offs:
- Pretraining methods (DAPT, TAPT) build foundational knowledge but require massive compute
- Fine-tuning methods (FFT, LoRA, KD) adapt models efficiently with different cost-performance profiles
- Alignment methods (RLHF-PPO, RLHF-DPO, RLAIF) optimize for human preferences but need careful implementation
Our Key Insight: Match the method to your goal. DAPT builds domain knowledge. FFT/LoRA adapt to tasks. RLHF (PPO/DPO) aligns preferences. Each serves a different purpose:they're not competing, they're complementary.
What's Next for Us
The debate between training methods will continue, and new methods will emerge. But our experiments taught us what really matters: understand your task deeply, match your method to your constraints, and always validate that your training objective aligns with your evaluation criteria.
