Run whisperCast

See also: Configure whisperCast?

whisperCast is a streaming server that can be configured for running by setting a number of command line flags. After starting it, configuring the media that it streams can be done via an RPC? interface, which can be invoked currently via an auto-generated html interface at http://<host>:<http port>/__admin__.

This document covers the command line flags and general file and directory configuration necessary for running whisperCast. For information regarding media and stream configuration, please visit the Configure whisperCast document.

Upon installation with make install (please check the Install document), several directories and a test/demo configuration is created. You can use that structure as a starting point, or you can use cpack to prepare a debian package to be installed on multiple of your servers.

In that demo structure, you can look in $INSTALL_PREFIX/bin/start_whispercast.sh to see an example for starting whisperCast:

$> LD_LIBRARY_PATH=$INSTALL_PATH/lib/:$LD_LIBRARY_PATH  $INSTALL_PATH/bin/whispercast --flagfile=$INSTALL_PATH/whispercast/etc/ini/whispercast.ini

It first makes sure that the necessary shared libraries (whisperLib, whisperStreamLib) are in the dynamic libraries' path, and then starts whisperCast with flags read from the files specified by the --flagfile flag.

The command line flags are of great importance of how whisperCast runs and behaves. You can get the long list of parameters by running

$> whispercast --help

Here are the parameters, grouped by their category and their meaning.

Directories:

• -base_media_dir
  Every file that we serve is under this directory. This is the root for all whisperCast/Elements/FileElements that you create. If you have data on multiple disks, make some links in here. (normally should be set to $INSTALL_PATH/whisperCast/var/www). Can be shared by multiple instances of whisperCast running on the same machine.
  
  
• -element_libraries_dir
  From this place we load the modules (shared libraries) that contain the implementations of elements that we can create. The sets of elements is extensible, and anyone can write a module with some elements to satisfy his own needs (normally should be set to $INSTALL_PATH/whisperCast/modules). Can be shared by multiple instances of whisperCast running on the same machine.
  
  
• -media_config_dir
  This is where the configuration files are placed (normally should be set to $INSTALL_PATH/whisperCast/etc/config). Must be unique for each whisperCast instance.
  
  
• -media_state_dir
  Here whisperCast saves its state - a checkpoint and some log files (normally should be set to $INSTALL_PATH/whisperCast/var/state). Pair -media_state_dir / -media_state_name must be unique for each whisperCast instance.
  
  
• -media_state_name
  whisperCast prefixes the state files with this identifier. Pair -media_state_dir / -media_state_name must be unique for each whisperCast instance.
  
  
• -media_state_checkpoint_interval_sec
  We checkpoint the state every s often.. (default: 1800)
  
  
• -rpc_js_form_path
  If specified, we make RPC interface available via auto-generated forms, and we read the extra js sources from here (normally should be set to $INSTALL_PATH/whisperCast/var/www/js/rpc)
  
  
• -rtmp_handshake_library
  If specifired, we load the rtmp handshake function from this library. In order to stream h264 encoded flv you need this library. It is available via a sepparate commercial package from whispersoft.
  
  

Basic network parameters:

• -http_port
  The port on which we accept HTTP connections (default: 8080). If you want to bind to the standard HTTP port (80) you may be tempted to start whisperCast as root, which we strongly NOT advise. Alternately, you may run it on port 8080 and use iptables to map port 80 to port 8080 (of course, other solutions, that do not imply running whisperCast as root, may also be available on your system).
  
  
• -rtmp_port
  The port on which we accept RTMP connections (default: 1935).
  
  
• -http_max_num_connections
  We accept at most these many concurrent HTTP connections (default: 1000). NOTE: your system may have limits set for the number of file descriptors opened by user / process (usually 1024). If you increase this flag, make sure that you do not run into that limit (look for man ulimit and /etc/security/limits.conf file).
  
  
• -rtmp_max_num_connections
  We accept at most these many concurrent RTMP connections (default: 1000). NOTE: your system may have limits set for the number of file descriptors opened by user / process (usually 1024). If you increase this flag, make sure that you do not run into that limit (look for man ulimit and /etc/security/limits.conf file).
  
  
• -media_path
  For HTTP, we export our media elements under this path (default "" - which corresponds to root);
  
  

Authorization:

• -admin_passwd
  The admin password for whisperCast. We strongly encourage you to put this in a file and use (--flagfile option).
  
  
• -authorization_realm
  If you set an admin password, you also need this, as it is the realm returned for a failed basic-authentication HTTP request.
  
  

HTTP parameters:

• -http_connection_max_media_outbuf_size
  We use at most this send buffer for all media sent on HTTP connections. We start dropping media tags when buffer exceedes this value. (In bytes)
  
  
• -http_connection_read_timeout
  Read timeout for HTTP server connections (default: 20000)
  
  
• -http_connection_write_timeout
  Write timeout for HTTP server connections (default: 20000)
  
  
• -http_connection_write_ahead_ms
  How much HTTP data (measured on time scale) to write ahead of the element time, in miliseconds (default 4000).
  
  
• -http_default_content_type
  We send this content type when we don't know what content type to send (default: "text/html")
  
  
• -http_extensions_to_content_types_file
  If set we read mappings from extensions to content types from here (normally should be set to $INSTALL_PATH/whisperCast/etc/config/extension2content_type).
  
  

RTMP parameters:

• -rtmp_connection_chunk_size
  Use this RTMP chunk size when talking with clients.
  
  
• -rtmp_connection_max_video_outbuf_size
  We use at most this end buffer for all video sent on RTMP connections. We start dripping video tags when buffer exceedes this value. (In bytes)
  
  
• -rtmp_connection_max_media_outbuf_size
  We use at most this send buffer for all media sent on RTMP connections. We start dropping media tags when buffer exceedes this value. (In bytes)
  
  
• -rtmp_connection_max_outbuf_size
  An absolute maximum buffer size for RTMP connections. We start dropping anything over this limit. (In bytes). Please note that we should have: -rtmp_connection_max_video_outbuf_size <= -rtmp_connection_max_media_outbuf_size <= -rtmp_connection_max_outbuf_size.
  
  
• -rtmp_connection_media_chunck_size
  When sending h264 encoded media we send chuncks of this size (In bytes)
  
  
• -rtmp_connection_min_outbuf_size_to_send
  We do not send to an RTMP connection when we have under this size, unless forced (default: 8192). Increasing it improves performance, decreasing it improves latency. We recommend keeping at least the default value. (In bytes)
  
  
• -rtmp_connection_play_reset_description
  What we return as a description in the server status of invoked rtmp play commands.
• -rtmp_connection_play_reset_detail
  What we return as a detail in the server status of invoked rtmp play commands.
  
  
• -rtmp_connection_seek_processing_delay_ms
  We delay the seek processing by this amount of time, to avoid seek bursts (default: 1000)
  
  
• -rtmp_connection_send_buffer_size
  The internal TCP write buffer size for the RTMP connections (with all TCP related things attached - In bytes) (default: 120000)
  
  
• -rtmp_connection_write_ahead_ms
  How much RTMP data (measured on time scale) to write ahead of the element time, in miliseconds
  
  
• -rtmp_connection_pause_timeout_ms
  After how long we close a paused connection (default: 60000)
  
  
• -rtmp_connection_write_timeout_ms
  Write timeout for RTMP server connections (default: 20000)
  
  
• -rtmp_decoder_mem_limit
  We limit the size of the objects received from client to this size (default: 131072)
  
  
• -rtmp_fancy_escape
  If true, decodes escaped stream names received on RTMP play command ('='->'/' and '#' -> '.'). This crap is necessary because of the way Flash FlvPlayback? object treats rtmp:// urls given to it. To such an object, instead of rtmp://host:port/'''whisperCast'''/element1/element2/file.flv you would need to specify rtmp://host:port/'''whisperCast'''/element1=element2=file#flv, and have this flag turned on. If you use the base Flash """NetStream?""" object (or the provided whisperPlayer-Basic), such a mess would not be necessary.
  
  

Saver Configuration:

Savers are special elements in whisperCast that can save media that is produced by other elements. For example, you can use them to save media files for live transmissions.

Savers save several partial files, of determined length for the duration they are configured to save.

Some of the flags here need to be kept consistent with the ones givven to whisperProc file processor, the that picks up partial files written by savers and combines them in one big file to be served.

• -saver_description_file
  Where to put the small description of our saves (default: "description.txt")
  
  
• -saver_dir_prefix
  We write saved media under this directory (inside -base_media_dir) (default: "saved")
  
  
• -saver_file_prefix
  We write partial files with this prefix (default: "part_")
  
  
• -saver_file_sufix
  We write partial files with this suffix (default: ".part_flv")
  
  
• -max_default_save_duration_sec
  When starting on user command a save, we stop automatically recording after these many seconds (even when no explicit stop command is givven, to prevent 'forgoten' save operations) (default: 10800)
  
  
• -media_saver_max_file_size
  We initiate a new partial file when we have written these many bytes in the old file (default: 5000000)
  
  

Stats Saving:

whisperCast can dump information about the consumed media. It can do it directly in a MySQL database (to be discontinued soon), or in files: a binary log file (recommended) or a text file (discoruaged).

MySQL stats saving:

(will soon be removed)

• -mysql_stats_database
  If specified, we connect to this database.
  
  
• -mysql_stats_host
  If specified, we dump stats to this server database server.
  
  
• -mysql_stats_passwd
  We connect to db using this password (if specified). We strongly encourage you to put this in a file and use (--flagfile option).
  
  
• -mysql_stats_port
  We connect to mysql stats server on this port (default: 3306).
  
  
• -mysql_stats_user
  We connect to db using this user (if specified).
  
  

File stats logging:

• -text_stats_file
  The filename (path) where to log stats in text mode (discouraged).
  
  
• -log_stats_dir
  The directory where the stats log files will be written.
  
  
• -log_stats_file_base
  The base name for stats log files.
  
  

Other:

• -stat_collection_interval_ms
  We collect stats about all our connection at this interval (in miliseconds) (default: 60000).
  
  
• -server_id
  The ID of the server, as it will be logged for stats purposes (default: "")
  
  

IP classification:

We normally classify IP-s based on various criteria. We support so far only IPv4 addresses. We don't use them much, so far, but here are the flags involved:

• -ip_classifiers
  Comma sepparated list of classifiers - first class 0, etc). Please refer to IP Clasification Syntax.
  
  
• -ip2location_classifier_db
  Where is the file for ip2location database (you need this file with a license from Ip2Location).
  
  

File AIO-related parameters:

For high-performance file access you should fine-tune these parameters:

• -disk_devices
  Comma separated list of disk devices (we create one aio manager) per disk instance. For example if your -base_media_dir is /media, and inside it you have directories /media/sda3, /media/sdb3 and /media/sdc3, each corresponding to a disk, you should specify, for optimum performance, --disk_devices=/media/sda3/,/media/sdb3/,/media/sdc3.
  
  
• -media_aio_block_size
  Alignment of aio memory blocks - should me multiple of disk block size (default: 4096).
  
  
• -media_aio_buffer_pool_size - Basically this is the maximum number of aio buffers that we can have allocated at one time. This determines also the number of concurrent aio operations (as we need one block for an operation at one time) (default: 1000).
  
  
• -media_aio_buffer_size - We read media from disk in chunks of this size - is specified in blocks of -media_aio_block_size bytes. The bigger you have these, the more bandwidth you get (and more concurrent file operations you have), but also the longer the seek latency. This is probably the biggest memory user for whisperCast, as it allocates upfront -media_aio_buffer_size * -media_aio_buffer_pool_size * -media_aio_block_size bytes of memory (for default values that is ~256MB) (default: 64).
  
  
• -max_concurrent_aio_ops - We start at most these many aio operations per thread (one thread per [disk, buffer range] pair) (default: 64).
  
  

FLV-related flags:

• -flv_max_tag_size
  Maximum accepted FLV tag size, as it probably reflects some totally screwed up - in bytes. (default: 5242880, but probably using something like 128000 would be safe too).
  
  
• -flv_coder_search_for_head
  We can try to find another FLV header in a corrupted FLV stream by looking in these many bytes after we detect corruption (default: 0 - i.e. we don't try).
  
  
• -flv_joiner_max_timestamp_difference
  If we see this much difference between two consecutive flv tags, we collapse them (this flag has no effect in whisperCast but is here because of the library linkage).
  
  

Other:

• -selector_high_alarm_precission
  Loose some CPU time and gain that extra mili-second precission for selector alarms. (default: false).
  
  

Logging and Messages:

• -alsologtostderr
  If this is turned on, we also write log lines to this stderr (default: false).
  
  
• -logdir
  We write the log file under this directory. (default: "/tmp").
  
  
• -loglevel
  We initialize the log at this maximum level. (i.e. we log messages with levels less or equal then this). Levels: 0 - FATAL, 1: ERROR, 2: WARNING, 3: INFO, 5: DEBUG (lower - deep debugging) (default: 3 - INFO).
  
  
• -logtid
  Log the thread ID in each message (default: false).
  
  
• -logcolors
  If this is turned on, the log will use bash colors for log messages (default: false).
  
  

Debugging:

• -loop_on_crash
  Causes the program to hang on bad signals waiting for your debugger. (default: false).
  
  
• -loop_on_exit
  If this is turned on, we loop before exiting (waiting for your debuger to attach) (default: false).
  
  
• -rtmp_connection_dlog_level
  Turn on deep debugging messages on RTMP connections (default: false).
  
  
• -http_connection_dlog_level
  Turn on deep debugging messages on HTTP connections (default: false).
  
  
• -debug_check_long_callbacks_ms
  If greater than zero, we check (in debug mode only !) that processing functions, callbacks and alarm functions take less then this amount of time, in miliseconds) (default: 500).
  
  
• -rtmp_debug_dump_buffers
  For really deep debugging, dumps to log the buffers before decoding (default: 0).
  
  
• -rtmp_debug_dump_sent_eventsFor debugging, dump all sent RTMP events (default false).
  
  
• -rtmp_debug_dump_only_control
  For debugging, dump just the control events (no audio/video) *if* -rtmp_debug_dump_sent_events is turned on (default: false).
  
  
• -stream_log_detailed_media_tags
  Log each media tag that we send over HTTP -- really detailed :) (default: false).
  
  

Help on Command Line Flags:

We use google flags library for our command line flags. These are flags offered by that library:

• -help
  Show help on all flags [tip: all flags can have two dashes].
  
  
• -helpfull
  Show help on all flags -- same as -help).
  
  
• -helpmatch
  Show help on modules whose name contains the specified substr.
  
  
• -helpon
  show help on the modules named by this flag value.
  
  
• -helppackage
  Show help on all modules in the main package.
  
  
• -helpshort
  Show help on only the main module for this program.
  
  
• -helpxml
  Produce an xml version of help.
  
  
• -version
  Show version and build info and exit.
  
  

Flags related to Flags:

We use google flags library for our command line flags. These are flags offered by that library:

• -flagfile
  Load flags from file.
  
  
• -fromenv
  Set flags from the environment [use 'export FLAGS_flag1=value'].
  
  
• -tryfromenv
  Set flags from the environment if present.
  
  
• -undefok
  Comma-separated list of flag names that it is okay to specify on the command line even if the program does not define a flag with that name. IMPORTANT: flags in this list that have arguments MUST use the flag=value format).