-
어떻게 동작하도록 하고 싶은지에 대한 설명 : 전처리 작업이 된 '영어', '문장' 데이터를 학습 시킨 후 이후 문장을 집어넣었을 때 어떤 '레이블'로 예측되는지 알아내는 작업
-
겪고 있는 문제에 대한 상세한 설명 (에러의 경우 에러 메시지 포함) : BERT 모델을 쓰고 싶은데 자꾸 런타임에러가 발생함
-
Python, PyTorch 및 사용하고 계신 주요 라이브러리의 버전 등을 포함한 환경 설명
-
실행하셨던, 혹은 문제 발생의 원인으로 의심하는 코드
!--> GPT에서 생성한 코드와 각종 커뮤니티에서 확인 한 결과를 조합해서 코드를 만들어내어 명확하지 않음.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from transformers import BertTokenizer, BertForSequenceClassification, Trainer, TrainingArguments
from torch.utils.data import Dataset
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
import torch
Load data
data = pd.read_excel("/Users/jakedb/Downloads/BERT/미래엔_2학년_pp.xlsx")
mps_device = torch.device("mps")
Define the BERT model and tokenizer
model_name = 'bert-base-uncased'
tokenizer = BertTokenizer.from_pretrained(model_name)
number_of_labels = data['label'].nunique()
model = BertForSequenceClassification.from_pretrained(model_name, num_labels=number_of_labels)
model.to(mps_device)
device = (
"cuda"
if torch.cuda.is_available()
else "mps"
if torch.backends.mps.is_available()
else "cpu"
)
print(f"Using {device} device")
Prepare the data
texts = data['text'].tolist()
labels = data['label'].tolist()
Initialize the LabelEncoder
encoder = LabelEncoder()
Fit the encoder and transform the labels
labels = encoder.fit_transform(labels)
Split the data into a training set and a test set
train_texts, test_texts, train_labels, test_labels = train_test_split(texts, labels, test_size=0.2, random_state=42)
Now you can tokenize your texts
train_encodings = tokenizer(train_texts, truncation=True, padding=True)
test_encodings = tokenizer(test_texts, truncation=True, padding=True)
Define a PyTorch Dataset
class EnglishTextDataset(Dataset):
def init(self, encodings, labels, device):
self.encodings = encodings
self.labels = labels
self.device = device
def __getitem__(self, idx):
item = {key: torch.tensor(val[idx]).to(self.device) for key, val in self.encodings.items()}
item['labels'] = torch.tensor(self.labels[idx]).to(self.device)
return item
def __len__(self):
return len(self.labels)
train_dataset = EnglishTextDataset(train_encodings, train_labels, device=device)
test_dataset = EnglishTextDataset(test_encodings, test_labels, device=device)
Define metrics
def compute_metrics(pred):
labels = pred.label_ids
preds = pred.predictions.argmax(-1)
precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='binary')
acc = accuracy_score(labels, preds)
return {
'accuracy': acc,
'f1': f1,
'precision': precision,
'recall': recall
}
Define the training arguments
training_args = TrainingArguments(
output_dir='./results',
num_train_epochs=3,
per_device_train_batch_size=16,
per_device_eval_batch_size=64,
warmup_steps=500,
weight_decay=0.01,
logging_dir='./logs',
)
Create the Trainer and train the model
trainer = Trainer(
model=model.to(device),
args=training_args,
train_dataset=train_dataset,
eval_dataset=test_dataset,
compute_metrics=compute_metrics
)
Train the model
trainer.train()
Evaluate the model
eval_result = trainer.evaluate()
print(eval_result)
위까지 코드 전체.
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Using mps device
/Users/jakedb/Downloads/yes/envs/tf29_py39/lib/python3.9/site-packages/transformers/optimization.py:411: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or setno_deprecation_warning=Trueto disable this warning
warnings.warn(
0%| | 0/16947 [00:00<?, ?it/s]Traceback (most recent call last):
File "/Users/jakedb/Downloads/BERT/main.py", line 105, in
trainer.train()
File "/Users/jakedb/Downloads/yes/envs/tf29_py39/lib/python3.9/site-packages/transformers/trainer.py", line 1645, in train
return inner_training_loop(
File "/Users/jakedb/Downloads/yes/envs/tf29_py39/lib/python3.9/site-packages/transformers/trainer.py", line 1938, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/Users/jakedb/Downloads/yes/envs/tf29_py39/lib/python3.9/site-packages/transformers/trainer.py", line 2759, in training_step
loss = self.compute_loss(model, inputs)
File "/Users/jakedb/Downloads/yes/envs/tf29_py39/lib/python3.9/site-packages/transformers/trainer.py", line 2784, in compute_loss
outputs = model(**inputs)
File "/Users/jakedb/Downloads/yes/envs/tf29_py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/jakedb/Downloads/yes/envs/tf29_py39/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1562, in forward
outputs = self.bert(
File "/Users/jakedb/Downloads/yes/envs/tf29_py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/jakedb/Downloads/yes/envs/tf29_py39/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1013, in forward
embedding_output = self.embeddings(
File "/Users/jakedb/Downloads/yes/envs/tf29_py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/jakedb/Downloads/yes/envs/tf29_py39/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 230, in forward
inputs_embeds = self.word_embeddings(input_ids)
File "/Users/jakedb/Downloads/yes/envs/tf29_py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/jakedb/Downloads/yes/envs/tf29_py39/lib/python3.9/site-packages/torch/nn/modules/sparse.py", line 162, in forward
return F.embedding(
File "/Users/jakedb/Downloads/yes/envs/tf29_py39/lib/python3.9/site-packages/torch/nn/functional.py", line 2210, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Placeholder storage has not been allocated on MPS device!
0%| | 0/16947 [00:00<?, ?it/s]
여기가 오류 메세지입니다.
질문1. 현재 커뮤니티에서 진행하라고 한 M2 Mac 환경 설정도 다 되었고, Using mps device 라고 정상으로 뜨는 상황인데, 계속 에러가 발생하는 이유는 무엇인가요?
질문2. model.to(device) 로 설정한 라인에서 계속 오류가 나오는데, 코드 전반적으로 틀린 부분이 많은가요?
코드나 인공지능 활용을 이제 막 배우는 단계입니다. 많은 도움 부탁드립니다. 감사합니다.
를 한 번 눌러주시길 부탁드려요!