늘보의 서랍

The 🤗 Tokenizers library - Hugging Face Course Introduction In Chapter 3, we looked at how to fine-tune a model on a given task. When we do that, we use the same tokenizer that the model was pretrained with — but what do we do when we want to train a model from scratch? In these cases, using a tokeni huggingface.co 코스 6에 해당하는 내용 1. 기존 토크나이저로 새 토크나이저 학습시키기 데이터셋을 불러오고 사용할 데이터를 확인. generator를 활용해 토..

The 🤗 Datasets library - Hugging Face Course Introduction In Chapter 3 you got your first taste of the 🤗 Datasets library and saw that there were three main steps when it came to fine-tuning a model: Load a dataset from the Hugging Face Hub. Preprocess the data with Dataset.map(). Load and compute huggingface.co 오늘은 chapter 5 내용을 리뷰하고자 한다. 1. Data 불러오기 데이터 로드 및 확인, field를 통해 어디에서 data를 가져올지 지정(j..

Sharing models and tokenizers - Hugging Face Course In the steps below, we’ll take a look at the easiest ways to share pretrained models to the 🤗 Hub. There are tools and utilities available that make it simple to share and update models directly on the Hub, which we will explore below. We encourage al huggingface.co 위 링크, Course 4의 Sharing pretrained models의 내용 요약입니다. 노트북 환경에서 notebook_login()을..

Sharing models and tokenizers - Hugging Face Course Using pretrained models The Model Hub makes selecting the appropriate model simple, so that using it in any downstream library can be done in a few lines of code. Let’s take a look at how to actually use one of these models, and how to contribute back to huggingface.co 위의 링크의 내용 1. 파이프라인 패키지를 통해 불러오기 - 가장 단순하지만 Task에 맞는 모델을 불러와야함. 2. 모델 아키텍쳐 패키..

Fine-tuning a pretrained model - Hugging Face Course Now we’ll see how to achieve the same results as we did in the last section without using the Trainer class. Again, we assume you have done the data processing in section 2. Here is a short summary covering everything you will need: Before actually writi huggingface.co 최근 허깅페이스의 코스를 학습 중이다. 허깅페이스는 NLP 문제들을 쉽게 해결할 수 있게 도와주는 API이자 HUB라고 보면 되겠다. ..