whisperLib

When we started whisperCast, we looked for a general purpose (networking) library to allow us to implement a high performance (TCP/IP) server in C++ under Linux.

Unfortunately we didn't find any, so we started to write one from the scratch.

Having performance and usability as our main goals, we chose an asynchronous (epoll based) model, relaying heavily on asynchronously invoked functors to process data or events.

The model of a server application is one with a small number of networking threads (normally just one), in which processing callbacks are called. These callbacks can serve requests immediately (in the same thread), or use a thread from a thread pool if longer or if some synchronous waiting is involved. All communication is non blocking in the networking threads, and for all networking events we detect we call registered callbacks which in turn detect evens of a higher level and call the corresponding registered callbacks, and so on, the only constraint being that these callbacks should return pretty quickly, so the networking is not block. All processing should be done asynchronously, or in a separate thread.

For example we can imagine detecting some read event to be processed on a socket (file descriptor), which involves calling a HandleReadEvent in net::TcpConnection, which may result in reading some data from the wire in a buffer. For precessing the data may be we have a registered function from an http::Server, which detects a POST request on path /rpc/MyService. That path may be registered by an rpc::HttpServer, which has its processing callback invoked when the request is detected. That in turn may call the invoked function for a hypothetical service MyService in order to actually serve the request.

* Download - get the goodies.

Components

• whisperlib/common/base - general components and utilities including logging, timers, callback definition, string manipulation and a fully imported instance of google-flags library for command line flags parsing and manipulation
• whisperlib/common/sync - wrappers over several pthread utilities
• whisperlib/common/io - general I/O utilities:
  • whisperlib/common/io/file - utilities for file manipulation, and libaio wrappers for easy use
  • whisperlib/common/io/zlib - easy to use wrappers for zlib
  • whisperlib/common/io/buffer - buffers manipulation
  • whisperlib/common/io/logio - record-based logs - reading and writing
  • whisperlib/common/io/checkpointing - utilities for maintaining a persistent state

• whisperlib/net/base - the basics of networking - including an epoll wrapping Selector class, connection objects and so on.
• whisperlib/net/http - HTTP protocol implementation (client & server)
• whisperlib/net/rpc/lib - an RPC library (client & server), so far supports JSON & internal binary marshaling with specifications written in an easy to comprehend internal laguage.
• whisperlib/net/rpc/parser - parses and compiles RPC specifications into *.h/cc or .js files.
• whisperlib/net/url - url parsing and manipulation (basically a full inclusion of google-url library, with some aditions)
• whisperlib/net/util - various network utilities

More documentation is needed to be written, and it will, if you want you can take a look at the source code, including use cases in our tests.