English | [简体中文](README_CN.md)

# Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision

Unlike image classification tasks, **data annotation is relatively difficult and costly for semantic segmentation tasks**. Each pixel in the image is required to have a label, including objects with fine details, such as electric poles. Compared with the dense annotation of pixels, obtaining raw RGB data is relatively simpler. Thus, **how to make use of the large amount of unlabeled data to improve the performance of the model is a research hotspot in the field of semi-supervised semantic segmentation**.

[Cross pseudo supervision, CPS](https://arxiv.org/abs/2106.01226) is a **concise and high-performance** semi-supervised semantic segmentation algorithm. During training, two networks with the same structure but different initial states are used, and constraints are added to ensure that the outputs of the two networks for the same sample are similar. Specifically, the one-hot pseudo labels generated by one network will serve as the target for training another network. The cross entropy loss is used to make the supervision, just like the traditional supervised learning strategy used in semantic segmentation tasks. **This algorithm has achieved state-of-the-art (SOTA) results in two commonly-used benchmarks (PASCAL VOC, Cityscapes)**.

Some visualization results are as follows (RGB image on the left, prediction image in the middle, and ground-truth segmentation map on the right):

![](https://user-images.githubusercontent.com/52785738/229003524-103fb081-dd36-4b19-b070-156d58467fe2.png)

![](https://user-images.githubusercontent.com/52785738/229003602-05cb2be1-8224-4600-8f6a-1ec58b909e47.png)

## Contents
- [Installation](#Installation)
- [Models](#Models)
- [Dataset Preparation](#Dataset-Preparation)
- [Training, Evaluation and Prediction](#Training-Evaluation-and-Prediction)

## Installation

- [PaddlePaddle Installation](https://www.paddlepaddle.org.cn/install/quick)
    - Versions：PaddlePaddle develop (Nightly build), Python>=3.7

- To install PaddleSeg, use the following commands:

```shell
git clone -b develop https://github.com/PaddlePaddle/PaddleSeg
cd PaddleSeg
pip install -r requirements.txt
pip install -v -e .
```

## Models

We chose to reproduce the CPS.resnet50.deeplabv3+(1/2 Cityscapes) setting in the original paper, where 1/2 of the samples in the Cityscapes dataset are labeled. The mIoU is **78.39%**. The comparison is shown in the following table:

| CPS.resnet50.deeplabv3+(1/2 Cityscapes) | mIOU |
| --- | --- |
| original paper | 78.77% |
| reproduced | 78.39% |

Please download the pretrained weights from [this link](https://paddleseg.bj.bcebos.com/dygraph/cross_pseudo_supervision/cityscapes/deeplabv3p_resnet50_cityscapes0.5.pdparams).

## Data Preparation

The Cityscapes dataset was provided by the CPS source code. Download the dataset `city` to the `contrib/CrossPseudoService/data` folder from [OneDrive link](https://pkueducn-my.sharepoint.com/:f:/g/personal/pkucxk_pku_edu_cn/EtjNKU0oVMhPkOKf9HTPlVsBIHYbACel6LSvcUeP4MXWVg?e=139icd).

The dataset should be organized as follows:

```
data/
|-- city
    ├── config_new
    │    ├── coarse_split
    │    │   ├── train_extra_3000.txt
    │    │   ├── train_extra_6000.txt
    │    │   └── train_extra_9000.txt
    │    ├── subset_train
    │    │   ├── train_aug_labeled_1-16.txt
    │    │   ├── train_aug_labeled_1-2.txt
    │    │   ├── train_aug_labeled_1-4.txt
    │    │   ├── train_aug_labeled_1-8.txt
    │    │   ├── train_aug_unlabeled_1-16.txt
    │    │   ├── train_aug_unlabeled_1-2.txt
    │    │   ├── train_aug_unlabeled_1-4.txt
    │    │   └── train_aug_unlabeled_1-8.txt
    │    ├── test.txt
    │    ├── train.txt
    │    ├── train_val.txt
    │    └── val.txt  
    ├── generate_colored_gt.py
    ├── images
    │   ├── test
    │   ├── train
    │   └── val
    └── segmentation
        ├── test
        ├── train
        └── val
```

## Training, Evaluation and Prediction

Execute the following command to enter the `CrossPseudoSupervision` folder:
```shell
cd ./contrib/CrossPseudoSupervision
```

### Training

After preparing the environment and data, execute the following command to launch training:

```shell
export CUDA_VISIBLE_DEVICES=0
python train.py --config ./configs/deeplabv3p/deeplabv3p_resnet50_0.5cityscapes_800x800_240e.yml --log_iters 10 --save_dir ./output/ --batch_size 2
```

We recommend training the model with multiple GPUs on a single machine. Execute the following command to start training with four GPUs:

```shell
python -m paddle.distributed.launch --gpus="0,1,2,3" train.py --config ./configs/deeplabv3p/deeplabv3p_resnet50_0.5cityscapes_800x800_240e.yml \
--log_iters 10 --save_dir $SAVE_PATH$ --batch_size 8
```

- `SAVE_PATH`: The path to save files such as weights and logs.

**Note**:
1. The default configuration uses 1/2 labeled data. If you want to change the labeled ratio, you can modify `labeled_ratio` parameter in the configuration file. When you change the ratio of labeled data, the number of training epochs also needs to be adjusted according to the following table (modify `nepochs` parameter in the configuration file to adjust the training epochs):

| Ratio    | 1/16 | 1/8  | 1/4  | 1/2  |
| ---------- | ---- | ---- | ---- | ---- |
| nepochs | 128  | 137  | 160  | 240  |


### Evaluation

After training, execute the following commands to evaluate the model accuracy:

```shell
export CUDA_VISIBLE_DEVICES=0
python val.py \
       --config ./configs/deeplabv3p/deeplabv3p_resnet50_0.5cityscapes_800x800_240e.yml \
       --model_path $MODEL_PATH$
```

- `MODEL_PATH`: The path of model weights to load.

### Prediction

Execute the following commands to use sliding window inference for prediction:

```shell
export CUDA_VISIBLE_DEVICES=0
python predict.py \
       --config ./configs/deeplabv3p/deeplabv3p_resnet50_0.5cityscapes_800x800_240e.yml \
       --model_path $MODEL_PATH$ \
       --image_path $IMG_PATH$ \
       --save_dir $SAVE_PATH$ \
       --is_slide \
       --crop_size 800 800 \
       --stride 532 532
```

- `IMG_PATH`: The path of the picture or folder to be predicted.

You can also download the [pretrained weights](https://paddleseg.bj.bcebos.com/dygraph/cross_pseudo_supervision/cityscapes/deeplabv3p_resnet50_cityscapes0.5.pdparams) provided by us for prediction.
