# Tokenize

**Repository Path**: yanglinzhuo/tokenize

## Basic Information

- **Project Name**: Tokenize
- **Description**: C++ tokenizer and related code for minddiffusion
- **Primary Language**: C++
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2023-06-02
- **Last Updated**: 2023-06-08

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

## 简介

本仓库为 `minddiffusion` 的 `C++` 模型推理代码。

## 依赖库配置

- Boost
- OpenCV
- utf8proc

### Boost

For Ubuntu:

```shell
sudo apt install libboost-dev
sudo apt install libboost-filesystem-dev
```

For CentOS or EulerOS:

```shell
sudo yum install boost boost-devel
```

### OpenCV

For Ubuntu:

```shell
sudo apt install libopencv-dev python3-opencv
```

For CentOS or EulerOS:

```shell
# Build OpenCV
# Install minimal prerequisites (Ubuntu 18.04 as reference)
sudo apt update && sudo apt install -y cmake g++ wget unzip
# Download and unpack sources
wget -O opencv.zip https://github.com/opencv/opencv/archive/4.x.zip
unzip opencv.zip
# Create build directory
mkdir -p build && cd build
# Configure
cmake  ../opencv-4.x
# Build
cmake --build .

# Make
make -j4

# Install
# Installation root directory can be changed with CMAKE_INSTALL_PREFIX
# configuration parameter, e.g. -DCMAKE_INSTALL_PREFIX=$HOME/.local to
# install to current user's local directory.
sudo make install
```


### utf8proc

```shell
git clone https://github.com/JuliaStrings/utf8proc.git
```

### MindSpore-Lite

从官网链接下载环境包：https://www.mindspore.cn/lite/docs/zh-CN/r2.0/use/downloads.html
注意和机器架构一致。

解压后，放在当前代码文件夹根目录，将文件夹重命名为 `lite`，或者修改 `CMakeLists.txt` 中 `set(LITE_HOME lite`
语句的 `lite` 改为当前文件夹名称。