Benign huggingface download unavailable

rushikeshkotkar73

Calls base64.b64decode (Python)Imports Python base64 module

SHA-25690a9a1b739f834cdf7a2f40959f9eeff2a705b80acefb2cea9c19ff3958a9279

Evidence

Imports Python base64 module lines 1–7

1import regex as re
2import base64
3import os
4import tiktoken
5from typing import List, Optional, Union, Dict
6from transformers import PreTrainedTokenizer
7from transformers.utils import PaddingStrategy

Calls base64.b64decode (Python) lines 26–32

      mergeable_ranks = {}
      with open(vocab_file) as f:
          for line in f:
              token, rank = line.strip().split()
              rank = int(rank)
              token = base64.b64decode(token)
              mergeable_ranks[token] = rank

Credential-access vocabulary word: token lines 73–77

              temp += t
          else:
              raise TypeError("token should only be of type int, bytes or str")
      if temp:
          text += temp.decode("utf-8", errors="replace")

File opened for binary read lines 115–116

115        with open(self.vocab_file, 'rb') as fin:
116            proto_str = fin.read()

Python open write mode lines 118–119

118        with open(vocab_file, "wb") as writer:
119            writer.write(proto_str)