Huggingface Transformers

1---
2name: huggingface-transformers
3description: Hugging Face Transformers best practices including model loading, tokenization, fine-tuning workflows, and inference optimization. Use when working with transformer models, fine-tuning LLMs, implementing NLP tasks, or optimizing transformer inference.
4---
5 
6# Hugging Face Transformers Best Practices
7 
8Comprehensive guide to using the Hugging Face Transformers library including model loading, tokenization, fine-tuning workflows, pipeline usage, custom datasets, and deployment optimization.
9 
10---
11 
12## Quick Reference
13 
14**When to use this skill:**
15- Loading and using pre-trained transformers (BERT, GPT, T5, LLaMA, etc.)
16- Fine-tuning models on custom data
17- Implementing NLP tasks (classification, QA, generation, etc.)
18- Optimizing inference (quantization, ONNX, etc.)
19- Debugging tokenization issues
20- Using Hugging Face pipelines
21- Deploying transformers to production
22 
23**Models covered:**
24- Encoders: BERT, RoBERTa, DeBERTa, ALBERT
25- Decoders: GPT-2, GPT-Neo, LLaMA, Mistral
26- Encoder-Decoders: T5, BART, Flan-T5
27- Vision: ViT, CLIP, Stable Diffusion
28 
29---
30 
31## Part 1: Model Loading Patterns
32 
33### Pattern 1: Basic Model Loading
34 
35```python
36from transformers import AutoModel, AutoTokenizer
37 
38# Load model and tokenizer
39model_name = "bert-base-uncased"
40tokenizer = AutoTokenizer.from_pretrained(model_name)
41model = AutoModel.from_pretrained(model_name)
42 
43# For specific tasks
44from transformers import AutoModelForSequenceClassification
45model = AutoModelForSequenceClassification.from_pretrained(
46    model_name,
47    num_labels=3  # For 3-class classification
48)
49```
50 
51### Pattern 2: Loading with Specific Configuration
52 
53```python
54from transformers import AutoConfig, AutoModel
55 
56# Modify configuration
57config = AutoConfig.from_pretrained("bert-base-uncased")
58config.hidden_dropout_prob = 0.2  # Custom dropout
59config.attention_probs_dropout_prob = 0.2
60 
61# Load model with custom config
62model = AutoModel.from_pretrained("bert-base-uncased", config=config)
63 
64# Or create model from scratch with config
65model = AutoModel.from_config(config)
66```
67 
68### Pattern 3: Loading Quantized Models (Memory Efficient)
69 
70```python
71from transformers import AutoModel, BitsAndBytesConfig
72import torch
73 
74# 8-bit quantization (50% memory reduction)
75quantization_config = BitsAndBytesConfig(load_in_8bit=True)
76 
77model = AutoModel.from_pretrained(
78    "meta-llama/Llama-2-7b-hf",
79    quantization_config=quantization_config,
80    device_map="auto"  # Automatic device placement
81)
82 
83# 4-bit quantization (75% memory reduction)
84quantization_config = BitsAndBytesConfig(
85    load_in_4bit=True,
86    bnb_4bit_compute_dtype=torch.float16,
87    bnb_4bit_quant_type="nf4",
88    bnb_4bit_use_double_quant=True
89)
90 
91model = AutoModel.from_pretrained(
92    "meta-llama/Llama-2-13b-hf",
93    quantization_config=quantization_config,
94    device_map="auto"
95)
96```
97 
98### Pattern 4: Loading from Local Path
99 
100```python
101# Save model locally
102model.save_pretrained("./my-model")
103tokenizer.save_pretrained("./my-model")
104 
105# Load from local path
106model = AutoModel.from_pretrained("./my-model")
107tokenizer = AutoTokenizer.from_pretrained("./my-model")
108```
109 
110---
111 
112## Part 2: Tokenization Best Practices
113 
114### Critical Tokenization Patterns
115 
116```python
117from transformers import AutoTokenizer
118 
119tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
120 
121# ✅ CORRECT: All required arguments
122tokens = tokenizer(
123    text,
124    padding=True,  # Pad to longest in batch
125    truncation=True,  # Truncate to max_length
126    max_length=512,  # Maximum sequence length
127    return_tensors="pt"  # Return PyTorch tensors
128)
129 
130# Access components
131input_ids = tokens['input_ids']  # Token IDs
132attention_mask = tokens['attention_mask']  # Padding mask
133token_type_ids = tokens.get('token_type_ids')  # Segment IDs (BERT)
134 
135# ❌ WRONG: Missing critical arguments
136tokens = tokenizer(text)  # No padding, truncation, or tensor format!
137```
138 
139### Batch Tokenization
140 
141```python
142# Tokenize multiple texts efficiently
143texts = ["First text", "Second text", "Third text"]
144 
145tokens = tokenizer(
146    texts,
147    padding=True,  # Pad all to longest in batch
148    truncation=True,
149    max_length=128,
150    return_tensors="pt"
151)
152 
153# Result shape: [batch_size, max_length]
154print(tokens['input_ids'].shape)  # torch.Size([3, max_len_in_batch])
155```
156 
157### Special Token Handling
158 
159```python
160# Add special tokens
161tokenizer.add_special_tokens({
162    'additional_special_tokens': ['[CUSTOM]', '[MARKER]']
163})
164 
165# Resize model embeddings to match
166model.resize_token_embeddings(len(tokenizer))
167 
168# Encode with special tokens preserved
169text = "Hello [CUSTOM] world"
170tokens = tokenizer(text, add_special_tokens=True)
171 
172# Decode
173decoded = tokenizer.decode(tokens['input_ids'][0], skip_special_tokens=False)
174```
175 
176### Tokenization for Different Tasks
177 
178```python
179# Text classification (single sequence)
180tokens = tokenizer(
181    "This movie was great!",
182    padding="max_length",
183    truncation=True,
184    max_length=128,
185    return_tensors="pt"
186)
187 
188# Question answering (pair of sequences)
189question = "What is the capital of France?"
190context = "France is a country in Europe. Paris is its capital."
191 
192tokens = tokenizer(
193    question,
194    context,
195    padding="max_length",
196    truncation="only_second",  # Only truncate context
197    max_length=384,
198    return_tensors="pt"
199)
200 
201# Text generation (decoder-only models)
202prompt = "Once upon a time"
203tokens = tokenizer(prompt, return_tensors="pt")
204# No padding needed for generation input
205```
206 
207---
208 
209## Part 3: Fine-Tuning Workflows
210 
211### Pattern 1: Simple Fine-Tuning with Trainer
212 
213```python
214from transformers import (
215    AutoModelForSequenceClassification,
216    AutoTokenizer,
217    Trainer,
218    TrainingArguments
219)
220from datasets import load_dataset
221 
222# 1. Load dataset
223dataset = load_dataset("glue", "mrpc")
224 
225# 2. Load model
226model = AutoModelForSequenceClassification.from_pretrained(
227    "bert-base-uncased",
228    num_labels=2
229)
230tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
231 
232# 3. Tokenize dataset
233def tokenize_function(examples):
234    return tokenizer(
235        examples["sentence1"],
236        examples["sentence2"],
237        padding="max_length",
238        truncation=True,
239        max_length=128
240    )
241 
242tokenized_datasets = dataset.map(tokenize_function, batched=True)
243 
244# 4. Define training arguments
245training_args = TrainingArguments(
246    output_dir="./results",
247    evaluation_strategy="epoch",
248    learning_rate=2e-5,
249    per_device_train_batch_size=16,
250    per_device_eval_batch_size=16,
251    num_train_epochs=3,
252    weight_decay=0.01,
253    logging_dir="./logs",
254    logging_steps=100,
255    save_strategy="epoch",
256    load_best_model_at_end=True,
257    metric_for_best_model="accuracy",
258)
259 
260# 5. Define metrics
261from datasets import load_metric
262import numpy as np
263 
264metric = load_metric("accuracy")
265 
266def compute_metrics(eval_pred):
267    logits, labels = eval_pred
268    predictions = np.argmax(logits, axis=-1)
269    return metric.compute(predictions=predictions, references=labels)
270 
271# 6. Create Trainer
272trainer = Trainer(
273    model=model,
274    args=training_args,
275    train_dataset=tokenized_datasets["train"],
276    eval_dataset=tokenized_datasets["validation"],
277    compute_metrics=compute_metrics,
278)
279 
280# 7. Train
281trainer.train()
282 
283# 8. Save
284trainer.save_model("./fine-tuned-model")
285```
286 
287### Pattern 2: LoRA Fine-Tuning (Parameter-Efficient)
288 
289```python
290from transformers import AutoModelForCausalLM, AutoTokenizer
291from peft import LoraConfig, get_peft_model, TaskType
292 
293# Load base model
294model = AutoModelForCausalLM.from_pretrained(
295    "meta-llama/Llama-2-7b-hf",
296    load_in_8bit=True,  # 8-bit for memory efficiency
297    device_map="auto"
298)
299 
300# Configure LoRA
301lora_config = LoraConfig(
302    task_type=TaskType.CAUSAL_LM,
303    r=8,  # LoRA rank
304    lora_alpha=32,  # LoRA alpha
305    lora_dropout=0.1,
306    target_modules=["q_proj", "v_proj"],  # Which layers to adapt
307)
308 
309# Apply LoRA
310model = get_peft_model(model, lora_config)
311 
312# Check trainable parameters
313model.print_trainable_parameters()
314# Output: trainable params: 4.2M || all params: 6.7B || trainable%: 0.062%
315 
316# Train with Trainer (same as before)
317# Only LoRA parameters are updated!
318```
319 
320### Pattern 3: Custom Training Loop
321 
322```python
323import torch
324from torch.utils.data import DataLoader
325from transformers import AdamW, get_scheduler
326 
327# Prepare dataloaders
328train_dataloader = DataLoader(tokenized_datasets["train"], batch_size=16, shuffle=True)
329eval_dataloader = DataLoader(tokenized_datasets["validation"], batch_size=16)
330 
331# Optimizer
332optimizer = AdamW(model.parameters(), lr=2e-5)
333 
334# Learning rate scheduler
335num_epochs = 3
336num_training_steps = num_epochs * len(train_dataloader)
337lr_scheduler = get_scheduler(
338    "linear",
339    optimizer=optimizer,
340    num_warmup_steps=500,
341    num_training_steps=num_training_steps
342)
343 
344# Training loop
345device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
346model.to(device)
347 
348for epoch in range(num_epochs):
349    model.train()
350    for batch in train_dataloader:
351        batch = {k: v.to(device) for k, v in batch.items()}
352 
353        outputs = model(**batch)
354        loss = outputs.loss
355        loss.backward()
356 
357        optimizer.step()
358        lr_scheduler.step()
359        optimizer.zero_grad()
360 
361    # Evaluation
362    model.eval()
363    for batch in eval_dataloader:
364        batch = {k: v.to(device) for k, v in batch.items()}
365        with torch.no_grad():
366            outputs = model(**batch)
367        # Compute metrics
368```
369 
370---
371 
372## Part 4: Pipeline Usage (High-Level API)
373 
374### Text Classification Pipeline
375 
376```python
377from transformers import pipeline
378 
379# Load pipeline
380classifier = pipeline(
381    "text-classification",
382    model="distilbert-base-uncased-finetuned-sst-2-english"
383)
384 
385# Single prediction
386result = classifier("I love this product!")
387# [{'label': 'POSITIVE', 'score': 0.9998}]
388 
389# Batch prediction
390results = classifier([
391    "Great service!",
392    "Terrible experience",
393    "Average quality"
394])
395```
396 
397### Question Answering Pipeline
398 
399```python
400qa_pipeline = pipeline("question-answering", model="distilbert-base-uncased-distilled-squad")
401 
402result = qa_pipeline(
403    question="What is the capital of France?",
404    context="France is a country in Europe. Its capital is Paris, a beautiful city."
405)
406# {'score': 0.98, 'start': 49, 'end': 54, 'answer': 'Paris'}
407```
408 
409### Text Generation Pipeline
410 
411```python
412generator = pipeline("text-generation", model="gpt2")
413 
414outputs = generator(
415    "Once upon a time",
416    max_length=50,
417    num_return_sequences=3,
418    temperature=0.7,
419    top_k=50,
420    top_p=0.95,
421    do_sample=True
422)
423 
424for output in outputs:
425    print(output['generated_text'])
426```
427 
428### Zero-Shot Classification Pipeline
429 
430```python
431classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")
432 
433result = classifier(
434    "This is a course about Python programming.",
435    candidate_labels=["education", "technology", "business", "sports"]
436)
437# {'sequence': '...', 'labels': ['education', 'technology', ...], 'scores': [0.85, 0.12, ...]}
438```
439 
440---
441 
442## Part 5: Inference Optimization
443 
444### Optimization 1: Batch Processing
445 
446```python
447# ❌ SLOW: Process one at a time
448for text in texts:
449    output = model(**tokenizer(text, return_tensors="pt"))
450 
451# ✅ FAST: Process in batches
452batch_size = 32
453for i in range(0, len(texts), batch_size):
454    batch = texts[i:i+batch_size]
455    inputs = tokenizer(batch, padding=True, truncation=True, return_tensors="pt")
456    outputs = model(**inputs)
457```
458 
459### Optimization 2: Mixed Precision (AMP)
460 
461```python
462from torch.cuda.amp import autocast, GradScaler
463 
464scaler = GradScaler()
465 
466for batch in dataloader:
467    optimizer.zero_grad()
468 
469    # Forward pass in mixed precision
470    with autocast():
471        outputs = model(**batch)
472        loss = outputs.loss
473 
474    # Backward pass with scaled gradients
475    scaler.scale(loss).backward()
476    scaler.step(optimizer)
477    scaler.update()
478```
479 
480### Optimization 3: ONNX Export
481 
482```python
483from transformers import AutoModelForSequenceClassification, AutoTokenizer
484from optimum.onnxruntime import ORTModelForSequenceClassification
485 
486# Export to ONNX
487model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
488model.save_pretrained("./onnx-model", export=True)
489 
490# Load ONNX model (faster inference)
491ort_model = ORTModelForSequenceClassification.from_pretrained("./onnx-model")
492 
493# Inference (2-3x faster)
494tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
495inputs = tokenizer("Hello world", return_tensors="pt")
496outputs = ort_model(**inputs)
497```
498 
499### Optimization 4: Dynamic Quantization
500 
501```python
502import torch
503 
504# Quantize model to int8
505quantized_model = torch.quantization.quantize_dynamic(
506    model,
507    {torch.nn.Linear},  # Quantize Linear layers
508    dtype=torch.qint8
509)
510 
511# 4x smaller model, 2-3x faster inference on CPU
512```
513 
514---
515 
516## Part 6: Common Issues & Solutions
517 
518### Issue 1: CUDA Out of Memory
519 
520**Problem:** `RuntimeError: CUDA out of memory`
521 
522**Solutions:**
523 
524```python
525# Solution 1: Reduce batch size
526training_args = TrainingArguments(
527    per_device_train_batch_size=8,  # Was 32
528    gradient_accumulation_steps=4,  # Effective batch = 8*4 = 32
529)
530 
531# Solution 2: Use gradient checkpointing
532model.gradient_checkpointing_enable()
533 
534# Solution 3: Use 8-bit model
535from transformers import BitsAndBytesConfig
536quantization_config = BitsAndBytesConfig(load_in_8bit=True)
537model = AutoModel.from_pretrained("model-name", quantization_config=quantization_config)
538 
539# Solution 4: Clear cache
540import torch
541torch.cuda.empty_cache()
542```
543 
544### Issue 2: Slow Tokenization
545 
546**Problem:** Tokenization is bottleneck
547 
548**Solutions:**
549 
550```python
551# Solution 1: Use fast tokenizers
552tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=True)
553 
554# Solution 2: Tokenize dataset once, cache it
555tokenized_dataset = dataset.map(
556    tokenize_function,
557    batched=True,
558    num_proc=4,  # Parallel processing
559    remove_columns=dataset.column_names,
560    load_from_cache_file=True  # Cache results
561)
562 
563# Solution 3: Use larger batches for tokenization
564tokenizer(
565    texts,
566    padding=True,
567    truncation=True,
568    max_length=512,
569    return_tensors="pt",
570    batched=True,  # Process multiple texts at once
571    batch_size=1000
572)
573```
574 
575### Issue 3: Inconsistent Results
576 
577**Problem:** Model outputs different results for same input
578 
579**Solution:**
580 
581```python
582# Set seeds for reproducibility
583import random
584import numpy as np
585import torch
586 
587def set_seed(seed=42):
588    random.seed(seed)
589    np.random.seed(seed)
590    torch.manual_seed(seed)
591    torch.cuda.manual_seed_all(seed)
592    torch.backends.cudnn.deterministic = True
593    torch.backends.cudnn.benchmark = False
594 
595set_seed(42)
596 
597# Disable dropout during inference
598model.eval()
599 
600# Use deterministic generation
601outputs = model.generate(
602    inputs,
603    do_sample=False,  # Greedy decoding
604    # OR
605    do_sample=True,
606    temperature=1.0,
607    top_k=50,
608    seed=42  # For sampling
609)
610```
611 
612### Issue 4: Attention Mask Errors
613 
614**Problem:** `IndexError: index out of range in self`
615 
616**Solution:**
617 
618```python
619# ✅ ALWAYS provide attention mask
620tokens = tokenizer(
621    text,
622    padding=True,
623    truncation=True,
624    return_tensors="pt",
625    return_attention_mask=True  # Explicit (usually default)
626)
627 
628# Use it in model forward
629outputs = model(
630    input_ids=tokens['input_ids'],
631    attention_mask=tokens['attention_mask']  # Don't forget this!
632)
633 
634# For custom padding
635attention_mask = (input_ids != tokenizer.pad_token_id).long()
636```
637 
638---
639 
640## Part 7: Model-Specific Patterns
641 
642### GPT Models (Decoder-Only)
643 
644```python
645from transformers import GPT2LMHeadModel, GPT2Tokenizer
646 
647model = GPT2LMHeadModel.from_pretrained("gpt2")
648tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
649 
650# Set pad token (GPT doesn't have one by default)
651tokenizer.pad_token = tokenizer.eos_token
652 
653# Generation
654input_text = "The future of AI is"
655inputs = tokenizer(input_text, return_tensors="pt")
656 
657outputs = model.generate(
658    **inputs,
659    max_new_tokens=50,
660    num_beams=5,  # Beam search
661    early_stopping=True,
662    no_repeat_ngram_size=2,  # Prevent repetition
663    temperature=0.8,
664    top_p=0.9
665)
666 
667print(tokenizer.decode(outputs[0], skip_special_tokens=True))
668```
669 
670### T5 Models (Encoder-Decoder)
671 
672```python
673from transformers import T5ForConditionalGeneration, T5Tokenizer
674 
675model = T5ForConditionalGeneration.from_pretrained("t5-small")
676tokenizer = T5Tokenizer.from_pretrained("t5-small")
677 
678# T5 expects task prefix
679input_text = "translate English to German: How are you?"
680inputs = tokenizer(input_text, return_tensors="pt")
681 
682outputs = model.generate(
683    **inputs,
684    max_length=50
685)
686 
687print(tokenizer.decode(outputs[0], skip_special_tokens=True))
688# "Wie geht es dir?"
689```
690 
691### BERT Models (Encoder-Only)
692 
693```python
694from transformers import BertForMaskedLM, BertTokenizer
695 
696model = BertForMaskedLM.from_pretrained("bert-base-uncased")
697tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
698 
699# Masked language modeling
700text = "Paris is the [MASK] of France."
701inputs = tokenizer(text, return_tensors="pt")
702 
703# Get predictions for [MASK]
704outputs = model(**inputs)
705mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]
706mask_token_logits = outputs.logits[0, mask_token_index, :]
707 
708# Top 5 predictions
709top_5_tokens = torch.topk(mask_token_logits, 5, dim=1).indices[0].tolist()
710for token in top_5_tokens:
711    print(tokenizer.decode([token]))
712# capital, city, center, heart, ...
713```
714 
715---
716 
717## Part 8: Production Deployment
718 
719### FastAPI Serving Pattern
720 
721```python
722from fastapi import FastAPI
723from transformers import pipeline
724from pydantic import BaseModel
725import uvicorn
726 
727app = FastAPI()
728 
729# Load model once at startup
730classifier = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
731 
732class TextInput(BaseModel):
733    text: str
734 
735@app.post("/classify")
736async def classify_text(input: TextInput):
737    result = classifier(input.text)[0]
738    return {
739        "label": result['label'],
740        "confidence": result['score']
741    }
742 
743if __name__ == "__main__":
744    uvicorn.run(app, host="0.0.0.0", port=8000)
745```
746 
747### Batch Inference Optimization
748 
749```python
750import asyncio
751from typing import List
752 
753class BatchPredictor:
754    def __init__(self, model, tokenizer, max_batch_size=32):
755        self.model = model
756        self.tokenizer = tokenizer
757        self.max_batch_size = max_batch_size
758        self.queue = []
759        self.lock = asyncio.Lock()
760 
761    async def predict(self, text: str):
762        async with self.lock:
763            future = asyncio.Future()
764            self.queue.append((text, future))
765 
766            if len(self.queue) >= self.max_batch_size:
767                await self._process_batch()
768 
769        return await future
770 
771    async def _process_batch(self):
772        if not self.queue:
773            return
774 
775        texts, futures = zip(*self.queue)
776        self.queue = []
777 
778        # Process batch
779        inputs = self.tokenizer(list(texts), padding=True, truncation=True, return_tensors="pt")
780        outputs = self.model(**inputs)
781        results = outputs.logits.argmax(dim=-1).tolist()
782 
783        # Return results
784        for future, result in zip(futures, results):
785            future.set_result(result)
786```
787 
788---
789 
790## Quick Decision Trees
791 
792### "Which model should I use?"
793 
794```
795Task type?
796  Classification → BERT, RoBERTa, DeBERTa
797  Generation → GPT-2, GPT-Neo, LLaMA
798  Translation/Summarization → T5, BART, mT5
799  Question Answering → BERT, DeBERTa, RoBERTa
800 
801Performance vs Speed?
802  Best performance → Large models (355M+ params)
803  Balanced → Base models (110M params)
804  Fast inference → Distilled models (66M params)
805```
806 
807### "How should I fine-tune?"
808 
809```
810Have full dataset control?
811  YES → Full fine-tuning or LoRA
812  NO → Few-shot prompting
813 
814Dataset size?
815  Large (>10K examples) → Full fine-tuning
816  Medium (1K-10K) → LoRA or full fine-tuning
817  Small (<1K) → LoRA or prompt engineering
818 
819Compute available?
820  Limited → LoRA (4-bit quantized)
821  Moderate → LoRA (8-bit)
822  High → Full fine-tuning
823```
824 
825---
826 
827## Resources
828 
829- **Hugging Face Docs:** https://huggingface.co/docs/transformers/
830- **Model Hub:** https://huggingface.co/models
831- **PEFT (LoRA):** https://huggingface.co/docs/peft/
832- **Optimum:** https://huggingface.co/docs/optimum/
833- **Datasets:** https://huggingface.co/docs/datasets/
834 
835---
836 
837**Skill version:** 1.0.0
838**Last updated:** 2025-10-25
839**Maintained by:** Applied Artificial Intelligence
840