아래 코드로 모델을 로드하려고하는데 colab에서는 잘 되는 코드가 윈도우11에서는 잘 되지 않습니다. 뭐가 문제인지 모르겠어요. CUDA도 11.8로 버전다운했고,
print(f"Was CUDA enabled? {torch.cuda.is_available()}") 했을 때 True가 나오는것도 확인했습니다.
전체 소스는 다음과 같습니다.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
print(f"Was CUDA enabled? {torch.cuda.is_available()}")
model_id = "beomi/polyglot-ko-12.8b-safetensors" # safetensors 컨버팅된 레포
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map={"":0})
이렇게 하고 실행하면 다음과 같은 에러가 표시됩니다.
[c:\Users\hwyoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\tqdm\auto.py:21](file:///C:/Users/hwyoo/AppData/Local/Programs/Python/Python311/Lib/site-packages/tqdm/auto.py:21): TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See Installation — Jupyter Widgets 8.0.7 documentation from .autonotebook import tqdm as notebook_tqdm
Was CUDA enabled? True False ===================================BUG REPORT=================================== ================================================================================ The following directories listed in your path were found to be non-existent: {WindowsPath('vs/workbench/api/node/extensionHostProcess')} The following directories listed in your path were found to be non-existent: {WindowsPath('/matplotlib_inline.backend_inline'), WindowsPath('module')} CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')} DEBUG: Possible options found for libcudart.so: set() CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md CUDA SETUP: Loading binary [c:\Users\hwyoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...](file:///C:/Users/hwyoo/AppData/Local/Programs/Python/Python311/Lib/site-packages/bitsandbytes/libbitsandbytes_cuda118.so...) argument of type 'WindowsPath' is not iterable CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected. CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA. CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO. CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
[c:\Users\hwyoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\bitsandbytes\cuda_setup\main.py:166](file:///C:/Users/hwyoo/AppData/Local/Programs/Python/Python311/Lib/site-packages/bitsandbytes/cuda_setup/main.py:166): UserWarning: Welcome to bitsandbytes. For bug reports, please run python -m bitsandbytes warn(msg)
--------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) File [c:\Users\hwyoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\utils\import_utils.py:1096](file:///C:/Users/hwyoo/AppData/Local/Programs/Python/Python311/Lib/site-packages/transformers/utils/import_utils.py:1096), in _LazyModule.get_module(self, module_name) 1095 try: -> 1096 return importlib.import_module("." + module_name, self.name) 1097 except Exception as e: File [c:\Users\hwyoo\AppData\Local\Programs\Python\Python311\Lib\importlib_init.py:126](file:///C:/Users/hwyoo/AppData/Local/Programs/Python/Python311/Lib/importlib/init.py:126), in import_module(name, package) 125 level += 1 --> 126 return _bootstrap._gcd_import(name[level:], package, level) File :1204, in _gcd_import(name, package, level) File :1176, in find_and_load(name, import) File :1147, in find_and_load_unlocked(name, import) File :690, in _load_unlocked(spec) File :940, in exec_module(self, module) File :241, in _call_with_frames_removed(f, *args, **kwds) File [c:\Users\hwyoo\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\models\gpt_neox\modeling_gpt_neox.py:38](file:///C:/Users/hwyoo/AppData/Local/Programs/Python/Python311/Lib/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:38) 31 from ...modeling_outputs import (
...
python -m bitsandbytes Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes and open an issue at: Issues · TimDettmers/bitsandbytes · GitHub
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
혹시 어떤 부분이 잘못된건지 알수 있을까요? 부탁드리겠습니다.
를 한 번 눌러주시길 부탁드려요!