MSU Cloud Video Transcoding Benchmark 2020

Cloud Video Transcoding Services Comparison
conducted by Moscow State University Graphics & Media Lab Video Group

Video group head:	Dr. Dmitriy Vatolin
Project head:	Dr. Dmitriy Kulikov
Measurements, analysis:	Dr. Mikhail Erofeev, Alexander Yakovenko, Egor Sklyarov, Anastasia Antsiferova

---

---

Contents

• About
• List of compared services
• Results
• Download HTML report
• Methodology
• List of test videos
• Contact
• Subscribe to receive notifications about our reports

---

About

We perform an independent comparison of cloud-based video-encoding services. This year test included encoding performance comparison using services' default settings. The results of our previous comparison showed that in the case of cloud services that provide the same quality, the file-size difference can reach 100% and the cost per minute difference is 700%.

Comparison main points

• Encoding time and objective videos quality are measured
  We perform a series of measurements to estimate average time needed for transcoding. Objective algorithms are used for estimating the resulting video quality
• Two encoding use cases: default encoding options and automatic sequence-based options
  This year we performed two types of tests: encode test videos using service default encoding settings and using automatic sequence-based settings (per-title encoding). If a service does not have automatic sequence-based settings, it is not be presented in this test
• FullHD videos of different type and content (including UGC)
  15 FullHD videos were used for tests. We performed two types of measurements: the main case involved encoding into FullHD, and the additional case included transcoding into several resolutions
• Anonymous accounts were used for testing
  For fair comparison, we used accounts which were not connected to us as the comparison organizers. This was done to prevent the services reserving separate servers/rules for comparison test videos processing. If the account was provided by the owner of the service — it is noted in list of compared services

---

List of compared services

	Service name	Sequence-based encoding	Anonymous account	Storage
1	Alibaba ApsaraVideo for Media Processing	Yes	Yes	Alibaba Cloud OSS
2	AWS Elemental MediaConvert	Partial2 4	Yes	Amazon S3 (us-east-1)
3	Coconut	Partial4	Yes	Amazon S3 (us-east-1)
4	Kingsoft Cloud	No	No1	Kingsoft Cloud
5	Qencode	Partial4	Yes	Amazon S3 (us-east-1)
6	Tencent Media Processing Service	Partial3	No1	Tencent Cloud COS
7	Zencoder	Yes	No1	Amazon S3 (us-east-1)

8^*. FFMpeg was also used for measurements just as an example to be compared with Cloud-encoding solutions.

Notes

• ¹ — account was provided by service team
• ² — we didn't test AWS Elemental MediaConvert sequence-based encoding, because it was released after the comparison was started
• ³ — service provides parameterless sequence-based bitrate selection but doesn't calculate output resolutions. The following resolution ladder was used: 1080p, 720p, 480p, 360p, 240p
• ⁴ — service provides sequence-based bitrate selection but doesn't calculate output resolutions. The following resolution-quality parameters were used:

  Resoultion	AWS QVBR	Coconut quality	Qencode min–max CRFs
  1080p	9	4	11–20
  720p	8	4	11–20
  720p	7	3	21–30
  480p	7	3	21–30
  360p	7	3	21–30
  240p	7	3	21–30

---

Overall conclusions

Default services options were used for comparison, so the leaders are judged not only for quality score, but also for speed and cost trade-off. Below are the winners by different categories: speed/quality trade-off shows how frequently a service was pareto-optimal for encoding speed and quality among all test videos, cost/quality trade-off shows how frequently a service was pareto-optimal for video encoding cost and quality among all test videos, and ease of use shows our subjective estimation on service convenience.

Category	Metric	HEVC	H.264
Best speed/quality trade-off	YUV-SSIM	Tencent Media Processing Service Zencoder Alibaba ApsaraVideo for Media Processing	Tencent Media Processing Service Alibaba ApsaraVideo for Media Processing
	YUV-PSNR	Tencent Media Processing Service Zencoder Alibaba ApsaraVideo for Media Processing
	Y-VMAF	Tencent Media Processing Service Alibaba ApsaraVideo for Media Processing AWS Elemental MediaConvert
Best cost/quality trade-off	All metrics (YUV-SSIM, YUV-PSNR, Y-VMAF)	Tencent Media Processing Service Kingsoft Cloud Coconut	Tencent Media Processing Service Kingsoft Cloud
Ease of use		Coconut

Encoding time

We performed three encodes for each sequence on different days and day times. The chart below shows a deviation of encoding time among all videos for each iteration.
Big delays could be caused by high load of service resources (and big queues) or long time of accessing our videos from storage (the table with services description above shows which storage was used for each service).

Alibaba ApsaraVideo for Media Processing showed the least average encoding time and deviation for both HEVC and H.264 encoding
AWS Elemental MediaConvert showed small encoding time and deviation for H.264 encoding Kingsoft Cloud showed small time deviation for both HEVC and H.264 encoding

Different resolution test

Three resolutions were used to transcode source videos: FullHD(1920x1080), HD(1280x720), SD(854x480). The chart below shows RD-curves for different resolutions at pyranha_rafting video sequence.

Resolution: 480p Figure 2. RD-curves for tennis_vlog, 480p	Resolution: 720 Figure 3. RD-curves for tennis_vlog, 720p
Resolution: 1080 Figure 4. RD-curves for tennis_vlog, 1080p	Resolution: all on one chart Figure 5. All resolutions RD-curves for tennis_vlog

Merged resolutions chart shows all RD-curves constructed into one curve by the rule: it contains a point from each curve if this points is not covered by quality of a point from another resolution curve.

Quality/encoding speed trade-off

Integrating RD-curves by bitrate and calculating relative scores (AWS Elemental MediaConvert was takes as a reference), we got relative quality/relative speed plots (detailed methodology).
The chart below shows relative quality and speed scores averaged for all videos (HEVC encoding, SSIM metric).
Tencent Media Processing Service, Zencoder and Alibaba ApsaraVideo for Media Processing were most frequently optimal among all services.

Cost/quality trade-off

The following charts show relative bitrate and encoding cost for same quality (SSIM) averaged for all videos.
Kingsoft Cloud, Qencode (for H.264 and HEVC encoding) and Coconut (for HEVC encoding) showed good quality for the least prices.

Encoding cost per minute, 0.01$ (September 2020)

Service name	HEVC encoding	H.264 encoding
Alibaba ApsaraVideo for Media Processing	SD: 4.22 HD: 5.63 FullHD: 11.25	SD: 0.84 HD: 1.13 FullHD: 2.25
AWS Elemental MediaConvert	SD: 2.72 HD: 5.44 FullHD: 5.44	SD: 0.85 HD: 1.7 FullHD: 1.7
Coconut	SD: 1.5 HD: 1.5 FullHD: 1.5	SD: 1.5 HD: 1.5 FullHD: 1.5
Kingsoft Cloud	SD: 0.72 HD: 1.4 FullHD: 2.8	SD: 0.18 HD: 0.37 FullHD: 0.71
Qencode	SD: 1 HD: 2 FullHD: 2	SD: 0.5 HD: 1 FullHD: 1
Tencent Media Processing Service	SD: 1.27 HD: 2.48 FullHD: 4.94	SD: 0.79 HD: 1.09 FullHD: 2.15
Zencoder	SD: 5 HD: 10 FullHD: 10	SD: 5 HD: 10 FullHD: 10

Overall comparison of HEVC and H.264 encoding

The plot below shows encoding speed and relative quality scores for HEVC and H.264 averaged for all test videos, SSIM metric. The results for different metrics are almost the same (see all charts in full report). H.264 encoding results of AWS Elemental MediaConvert was used a reference to get quality scores.
Tencent Media Processing Service, Kingsoft Cloud and Zencoder showed best quality for HEVC encoding, Alibaba ApsaraVideo for Media Processing showed also good quality for higher encoding speed.

Per-title encoding test

In addition to default encoding settings, per-sequence encoding optimization was tested for services, which supported this feature at the time we did the tests. This option optimizes encoding settings to each input video to get better encoding performance (better quality of less bitrate).

We didn't test AWS Elemental MediaConvert sequence-based encoding, because it was released after the comparison was started, and Qencode sequence-based encoding, because it produces only one output without any bitrate or resolution variation.

According to our tests, Tencent Media Processing Service showed the best quality for per-title encoding, Alibaba ApsaraVideo for Media Processing is at the second place.

---

Download full HTML report

MSU Cloud Benchmark 2020 Download free full version

Cloud encoding services Alibaba Cloud: ApsaraVideo for Media Processing, AWS Elemental MediaConvert, Coconut, Kingsoft Cloud, Qencode, Tencent Cloud: Media Processing Service, Zencoder 15 FullHD video sequences From Vimeo and YouTube UGC Three resolutions FullHD, HD, SD Two encoding use cases Encoding with default options of tested services and with automatic sequence-based options HTML report 5153 interactive charts

Video Transcoding Clouds Comparison 2019 Download free full version

6 cloud encoding services Alibaba, Amazon Elastic Transcoder, AWS Elemental MediaConvert, Coconut, Qencode, Zencoder 4 FullHD video sequences epson, fountains, hawk and hockey from Vimeo website Three resolutions FullHD, HD, SD Three encoding use cases (presets) Encoding with similar options, default options of tested services and with tuned options for our test videos HTML and PDF documents 118 interactive charts and 33 pages

---

Methodology

1. Preprocessing

We performed lossless compression with x264 (x265 for AWS MediaConvert) and passed compressed stream as input, as most of services don't support raw YUV420p.

2. Storage

To equalize network latency in speed measurements we used Amazon S3 storage (us-east-1 region) for most of the services as widely used solution. Some services (Alibaba Cloud, Tencent Cloud) don't support Amazon S3, so we used corresponding storage analogs (Alibaba Cloud OSS, Kingsoft Cloud, Tencent Cloud COS).

3. Encoding

We used services' API to run encoding tasks. We performed H.264 and HEVC transcoding with multiple bitrates for each resolution. In the first use case we used default options everywhere it was possible.

Encoding bitrates and resolutions

Bitrate, kbps	480p	720p	1080p
75	+
120	+
250	+	+
500	+	+
750	+	+
1000	+	+	+
2000	+	+	+
4000		+	+
6000		+	+
8000			+
10000			+
12000			+

4. Speed measurement

This year we tried to measure encoding time including files transmission. We used services' callbacks to get jobs finish time. Also, we encoded one job multiple times to estimate deviation and minimize single time delays.

5. Objective metrics calculation

We used MSU VQMT to calculate objective metrics (SSIM, PSNR, VMAF) against original YUV files.

6. Plots

We merge RD curves of different resolutions to evaluate overall service performance. After that we made all our plots same way as in the Main report. You could learn more about our methodology in presentation.

---

Videos

Name	Preview	Resolution	FPS	Frames	Description	Source
blue_hair		1920x1080	30	600	Shaky handheld vlog video. ID: Vlog_1080P-35cd	YouTube UGC
christmas_cats		1920x1080	25	1500	Concert record with superimposed complicated translucent CG effects	Vimeo
construction_site		1920x1080	30	1043	Shots of building under construction	Vimeo
crowd_run		1920x1080	50	500	A crowd of sportsmen runs while the camera slowly moves left and right	Xiph
football		1920x1080	30	599	People are playing football. ID: Sports_1080P-15d1	YouTube UGC
hard_rock		1920x1080	25	500	Poorly lit very noisy scene of people. ID: Musicvideo_1080P-6260	YouTube UGC
kindergarten_interview		1920x1080	30	1016	One man is interviewing other	Vimeo
park_mobile		1920x1080	24	359	Information video with people speaking in front of camera and old fashioned scenes	Vimeo
pyranha_rafting		1920x1080	24	1203	Panning shots of whitewater rafting	Vimeo
stone		1920x1080	30	598	Shots of stone in front of a camera. ID: HowTo_1080P-7cf2	YouTube UGC
street_musician		1920x1080	24	974	Handheld video of a musician performing and people listening. Heavy grain and black and white sections	Vimeo
summer_of_adventure		1920x1080	30	994	Summer camp commercial, consists of nature scenes, POV shots and a slideshow	Vimeo
tennis_vlog		1440x1080	30	599	Selfie video followed by a girl practicing hitting a baseball. ID: Vlog_1080P-19cc	YouTube UGC
the_forest		1920x1080	30	600	The Forest gameplay. ID: Gaming_1080P-72c8	YouTube UGC
wedding_party		1920x1080	24	1757	Bride and groom dancing. Colorful illumination with flashes of light	Vimeo

---

Contact Information

---

Subscribe to report updates

---

Materials about MSU Codec Comparison

---

See all MSU Video Codecs Comparisons

MSU video codecs comparisons resources:

• Introduction to Video Codecs Comparison
• Lossless Video пїЅodecs Comparison 2004 (October 2004)
• MPEG-4 SP/ASP Video Codecs Comparison (March 2005)
• JPEG 2000 Image Codecs Comparison (September 2005)
• First Annual MPEG-4 AVC/ H.264 Video Codecs Comparison (January 2005)
• FAQ of First MPEG-4 AVC/ H.264 Comparison
• Selected Comments for First MPEG-4 AVC/ H.264 Comparison
• Second Annual MPEG-4 AVC/H.264 Video Codec Comparison (December 2005)
• FAQ of Second Annual MPEG-4 AVC/H.264 Video Codec Comparison
• Subjective Comparison of Modern Video Codecs (February 2006)
• MPEG-2 Video Decoders Comparison (May 2006)
• WMP and JPEG2000 Comparison (October 2006)
• Third Annual MPEG-4 AVC/H.264 Comparison (December 2006) (All versions for free!)
• Lossless Video Codecs Comparison 2007 (March 2007)
• Fourth Annual MPEG-4 AVC/H.264 Comparison (December 2007) (All versions for free!)
• Options Analysis of MPEG-4 AVC/H.264 Codec x264 (December 2008)
• Fifth MPEG-4 AVC/H.264 Comparison (May 2009) (All versions for free!)
• Sixth MPEG-4 AVC/H.264 Comparison (May 2010)
• Seventh MPEG-4 AVC/H.264 Comparison (May 2011)
• Eighth MPEG-4 AVC/H.264 Comparison (May 2012)
• Ninth MPEG-4 AVC/H.264 Comparison (Dec 2013)
• Tenth Video Codec Comparison (HEVC) (Oct 2015)
• Eleventh Video Codec Comparison (HEVC) (Aug 2016)
• Twelfth Video Codec Comparison (HEVC) (Aug 2017)
• Thirteen Video Codec Comparison (HEVC) (Aug 2018)
• Fourteen Video Codec Comparison (HEVC) (Sept 2019)
• Cloud Encoding Servoces Comparison 2019 (Dec 2019)
• Fifteen Video Codec Comparison (HEVC) (Dec 2020)
• Codec Analysis for Companies:
  • Private Video Codec Comparison
  • MSU Video Quality Measurement Tool

---

Other Materials

---

Video resources:

3D and stereo video Projects on 3D and stereo video processing and analysis MSU S3D-video analysis 1-st report (5 movies, captured) 2-nd report (5 movies, captured) 3-rd report (5 movies, 2D-3D conversion) 4-th report (5 movies, captured) 5-th report (5 movies, 2D-3D conversion) 6-th report (10 movies, Stereo Window) 7-th report (10 movies, Stereo Window) 8-th report (25 movies) 9-th report (105 movies) MSU 3D Devices Testing Semiautomatic Visual-Attention Model Video Matting 3D Displays Video Generation 3D Displays Video Capturing Stereo Video Depth Map Generation MSU Video Quality Measurement tools Programs with different objective and subjective video quality metrics implementation MSU Video Quality Measurement Tool - objective metrics for codecs and filters comparison Implemented metrics short info: PSNR, Delta, MSAD, MSE, SSIM, VQM, MSU Blurring Metric, MSU Blocking Metric Why update Download & Purchase Documentation MSU Human Perceptual Quality Metric - several metrics for exact visual tests Methods info: ITU-R BT.500-11: DSIS, DSCQS I & II, SCACJ; EBU: SAMVIQ; MSUCQE; Custom Download Documentation	Codecs comparisons Objective and subjective quality evaluation tests for video and image codecs Annual Video-Codecs Comparison 2020 MSU Cloud Benchmark 2020 Cloud Encoding Services Comparison 2019 HEVC/AV1 Codec Comparison 2019 HEVC/AV1 Codec Comparison 2018 HEVC/AV1 Codec Comparison 2017 HEVC Codec Comparison 2016 HEVC Codec Comparison 2015 9-th MPEG4-AVC/H.264 Comparison 8-th MPEG4-AVC/H.264 Comparison 7-th MPEG4-AVC/H.264 Comparison 6-th MPEG4-AVC/H.264 Comparison 5-th MPEG4-AVC/H.264 Comparison Codec Analysis for Companies: Private Video Codec Comparison MSU Video Quality Measurement Tool Options Analysis of Codec x264 4-th MPEG4-AVC/H.264 Comparison Lossless Video Codecs Comparison 2007 3-rd MPEG4-AVC/H.264 Comparison HD Photo and JPEG 2000 Comparison MPEG-2 Decoders Crash Test Subjective Compar. of 4 Modern Codecs 2-nd MPEG-4 AVC/H.264 Comparison FAQ of 2-nd H.264 Comparison 1-st MPEG-4 AVC/H.264 Codecs Comparison FAQ of 1-st H.264 Comparison JPEG 2000 Image Codecs Comparison Lossless Video Codecs Comparison 2004 MPEG-4 SP/ASP Codecs Comparison Video codecs comparison (old) Ext. link: x264 parameters efficiency comparison
Public MSU video filters Here are available VirtualDub and AviSynth filters. For a given type of digital video filtration we typically develop a family of different algorithms and implementations. Generally there are also versions optimized for PC and hardware implementations (ASIC/FPGA/DSP). These optimized versions can be licensed to companies. Please contact us for details via video(at)graphics.cs.msu_ru. MSU Cartoon Restore MSU Noise Estimation MSU Frame Rate Conversion MSU Image Restoration MSU Denoising MSU Old Cinema MSU Deblocking MSU Smart Brightness and Contrast MSU Smart Sharpen MSU Noise generation MSU Noise estimation MSU Motion Estimation Information MSU Subtitles removal MSU Logo removal MSU Deflicker MSU Field Shift Fixer AviSynth plug-in MSU StegoVideo MSU Cartoonizer MSU SmartDeblocking MSU Color Enhancement MSU Old Color Restoration MSU TV Commercial Detector MSU filters FAQ MSU filters statistics	Filters for companies We are working with Intel, Samsung, RealNetworks and other companies on adapting our filters other video processing algorithms for specific video streams, applications and hardware like TV-sets, graphics cards, etc. Some of such projects are non-exclusive. Also we have internal researches. Please let us know via video(at)graphics.cs.msu_ru if you are interested in acquiring a license for such filters or making a custom R&D project on video processing, compression, computer vision. 3D Displays Video Generation 3D Displays Video Capturing Stereo Video Depth Map Generation Automatic Objects Segmentation Semiautomatic Objects Segmentation New Frame Rate Conversion New Deinterlacer MSU-Samsung Deinterlacing Project Digital TV Signal Enhancement Old Film Recovery Tuner TV Restore Panorama Video2Photo SuperResolution SuperPrecision MSU-Samsung image and video resampling MSU-Samsung Frame Rate Conversion Motion Phase filter Deshaker (video stabilization) Film Grain/Degrain filter Deblurring filter Video Content Search
Video codecs projects Different research and development projects on video codecs MSU Lossless Video Codec (Top!) MSU Screen Capture Lossless Codec (Top!) MSU MPEG-2 Video Codec x264 Codec Improvement	Other Other information Crazy gallery (filters screams :) License for commercial usage of MSU VideoGroup Public Software (please be careful: some soft like metrics has another license!)

Last updated: 02-March-2021

---

Server size: 8069 files, 1215Mb (Server statistics)

Project updated by
Server Team and MSU Video Group

Project sponsored by YUVsoft Corp.

Project supported by MSU Graphics & Media Lab