Tracks and ranking
Scoring system
Overview: The evaluation of submitted solutions involves comparing their x4 super-resolved frames with the corresponding ground truth frames and reporting the inference time of the SR model. The primary goal is to improve the quality of low-resolution videos to high-resolution, prioritizing high quality while ensuring minimal runtime for live broadcast scenarios.
Quality Metric: The quality metric is calculated and normalized using the WS-PSNR results of the bicubic model. We define two fixed values, WS-PSNR-Bicubic and WS-PSNR-max, which establish the lower and upper limits for model performance, respectively. Utilizing these values, we introduce penalties for models that produce WS-PSNR values below the bicubic baseline, while giving emphasis to those that achieve values higher than the defined maximum. These WS-PSNR values are fixed differently for each track.
Runtime: The runtime evaluation metric assigns a full score to models that achieve a processing time of 0.016 seconds or less per 2K frame, as this speed enables a smooth 60 frames per second (fps), essential for high-quality live streaming. To reinforce this standard, we apply penalties in our evaluation criteria for runtimes that exceed 0.016 seconds. The slower the model, the larger the penalties.
Final score: The final score is calculated by assigning specific coefficients to the quality and runtime scores and then multiplying the result by 100, as outlined in the formula above. These coefficients vary according to the track, ensuring that the scoring accurately reflects the priorities of each track.
Tracks
Our challenge, 360 Super Resolution and Quality Enhancement, comprises two tracks, each tailored to different use cases:
Track #1: ICIP 24 challenge:
Description: This truck for contributors aiming to submit a challenge paper to the ICIP 24. It targets x2 upscaling of the downgraded source videos. Achieving good quality in x2 super resolution is relatively more feasible. This truck lies in finding the optimal trade-off between complexity and quality. The complexity score is given a higher weight in the final score.
Final score weights: α = 0.3 and β = 0.7
Quality bounds: WS-PSNR-Bicubic = 29.16 and WS-PSNR-max = 31
Submission portal: Codalab Track #1
Track #2: Innovation showcase:
Description: This track is a longer-duration compared to Track 1, giving participants more time to work on their ideas. It focuses on achieving x4 upscaling for the input video, which poses a significant challenge in maintaining high quality. Models that achieve superior quality, even at a slower pace, are emphasized in this track. The quality score holds a higher weight in the final score calculation.
Final score weights: α = 0.5 and β = 0.5
Quality bounds: WS-PSNR-Bicubic = 27.87 and WS-PSNR-max = 30
Submission portal: Codalab Track #2