Download and Combine Media Segments of a HLS Stream Locally Using FFMpeg

NB: This article aims to introduce the concept of HLS streaming and FFMpeg at a foundational level.

I have had several experiences working with HLS streams and FFMpeg, and so in this article, I will be sharing how I was able to download and combine chunks of media segments of a variant of an HLS stream.

HLS was developed by Apple and it stands for "HTTP Live Streaming" which is the most widely used streaming protocol for playback today.

The HLS streams are delivered in the .ts format and these .ts files typically contains H.264 encoded video and AAC encoded audio, both compressed by an encoder. They are in turn packaged via RTMP protocol and sent to a media server which in turn un-packs the RTMP stream and re-packages it as HLS playlist to be sent to a CDN, cached for faster distribution to Players anywhere.

HLS streaming is an adaptive bitrate technology. This means, when video is encoded to HLS, multiple files are created for different bandwidths and different resolutions (Screen sizes). The streams are mapped to the client in real time using an .M3u8 index file based on screen size and available bandwidth.

FFMpeg is a multimedia framework with the ability to encode, decode, transcode, stream, and manipulate multimedia files. I find this tool really interesting for two reasons. First of which is that it is open source, and second is it can perform any and every streaming operation you may want to execute.

Download TS Segments

We will be working with the HLS test stream below:
https://mnmedias.api.telequebec.tv/m3u8/29880.m3u8

Running a curl from the terminal against that stream gives similar Manifest below:

#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=794000,RESOLUTION=640x360,CODECS="avc1.64001f,mp4a.40.2"
https://s2.content.video.llnw.net/smedia/42f4e71183054396907c0dea18241568/yd/yLpoUEilVrxhhYSkpsyGMnn9t0N3AYxWmoMsM7Faw/francstireurs_entrevue_ep472_seq24.mpegts/playlist-de5cb50573ac3952cd031f64973a614828771406.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=64000,RESOLUTION=398x224
https://s2.content.video.llnw.net/smedia/42f4e71183054396907c0dea18241568/pG/_eYmKLIBSVI6S27rSuF_ykGSlW0Qc8D1PALJty4Mk/francstireurs_entrevue_ep472_seq24.mpegts/playlist-3e7933be6a617888b736c1ebbb1f2a6bcda30afd.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=209000,RESOLUTION=416x234,CODECS="avc1.66.30,mp4a.40.2"
https://s2.content.video.llnw.net/smedia/42f4e71183054396907c0dea18241568/Sf/0Yz6m4a9H2LvdJs5cXZ2OkLVBsVETi3pQR4LF-Z3k/francstireurs_entrevue_ep472_seq24.mpegts/playlist-767499b7586bfe9b892916f8a24e71928f2e509f.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=429000,RESOLUTION=480x270,CODECS="avc1.66.30,mp4a.40.2"
https://s2.content.video.llnw.net/smedia/42f4e71183054396907c0dea18241568/DW/tAHYcWrGU7mj9nce4TbYlou5LPRT67U7UKqD1-L2c/francstireurs_entrevue_ep472_seq24.mpegts/playlist-b08bceed9acbde9e396ff9bda6c47b0e2767ffc3.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1196000,RESOLUTION=768x432,CODECS="avc1.64001f,mp4a.40.2"
https://s2.content.video.llnw.net/smedia/42f4e71183054396907c0dea18241568/Eg/o_fUnlyL800wzDHxFl6hhw-8UQc-ooyDeghAFJJhc/francstireurs_entrevue_ep472_seq24.mpegts/playlist-d4a7a4f0ec6d5166035d24a010a67a11eca19cf4.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2096000,RESOLUTION=960x540,CODECS="avc1.64001f,mp4a.40.2"
https://s2.content.video.llnw.net/smedia/42f4e71183054396907c0dea18241568/lB/Qc25Nb0TUdQe_w_ScrBJzqxkVpsKUEm1dGtLWhM3g/francstireurs_entrevue_ep472_seq24.mpegts/playlist-10a6530ae4e350c1c757c11287d1a0f7c2fad471.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=3096000,RESOLUTION=1280x720,CODECS="avc1.64001f,mp4a.40.2"
https://s2.content.video.llnw.net/smedia/42f4e71183054396907c0dea18241568/Ks/l56Xl_JnkEU-hQoz9-AHS5lrBjICWHZV0dybQSN3w/francstireurs_entrevue_ep472_seq24.mpegts/playlist-933ef269e85350d026f5e55af66a76a52737ff48.m3u8

The above manifest have 7 variants. Take for instance we need to download a local copy of the 1280x720 variant in an .mp4 container, we first need to get the media segments of the stream for that variant. To do so, first ensure you have a local installation of ffmpeg:

FFmpeg Installation for Linux: https://johnvansickle.com/ffmpeg/

FFmpeg Installation for Mac: Use homebrew - "brew install ffmpeg"

Verify FFmpeg installation by running the "ffmpeg" command directly from your terminal. If installation is successful, you should see something similar to this:

ffmpeg version N-95534-gac0f5f4c17 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple LLVM version 10.0.1 (clang-1001.0.46.4)
  configuration: --enable-gpl --enable-libx264 --enable-libvmaf --enable-version3
  libavutil      56. 35.101 / 56. 35.101
  libavcodec     58. 59.102 / 58. 59.102
  libavformat    58. 33.100 / 58. 33.100
  libavdevice    58.  9.100 / 58.  9.100
  libavfilter     7. 64.100 /  7. 64.100
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

Thereafter, run the command below to generate media segments in your current working directory:

ffmpeg -i https://mnmedias.api.telequebec.tv/m3u8/29880.m3u8 -map p:6 -c copy -t 60 -f segment -segment_list out.list out%03d.ts

Let me explain what those flags do:
"-i" specifies the input file which can be a stream url or any media file. "-map p:5" tells ffmpeg to select stream with program id 5, which in our case is the 720p variant (To know the different program id of a stream, run just ffmpeg -i <file/url>). "-c copy" tells ffmpeg to copy default video and audio properties and not make any encoding. "-f segment -segment_list" tells ffmpeg to segment the input file with output files in the format out%03d.ts. "-t" tells ffmpeg the duration to run the stream in seconds.
Running the above command will generate chunks of TS segments of about 3 seconds duration each within the current working directory.

Convert TS Segments to mp4

To combine the generated .ts file segments into a single .mp4 file:
from the working directory within your terminal, run the below command:

ffmpeg -f concat -safe 0 -i <(for f in ./*.ts; do echo "file '$PWD/$f'"; done) -c copy playlist.mp4

That will generate a playlist.mp4 file within same directory which can be played via any media player.