NAME
coroipc_overview - Overview of coroipc libraries
OVERVIEW
The coroipcs and coroipcc libraries provide a generically reusable very
high performance shared memory IPC sytem for client and service
applications. It supports many features including:
65536 user services and 65536 command ids per service.
Shared memory implementation for very high performance.
A synchronous request/response channel and asynchronous response
channel per ipc connection.
User defined private data per IPC connection.
Ability to call a function per service on ipc connection and
disconnection.
Authenticated IPC connection with ability for developer to define which
UIDs and GIDs are valid at connection time.
Fully abstracted poll system so that any poll library may be used.
User defined selector for determining the proper function to call per
service and id.
Description of the libraries
There are two shared libraries available for developing IPC client
applications.
The first library is coroipcs.so which is responsible for the server
implementation. This library should be linked with the server and then
initialized with coroipcs_init(3).
Once the library is initialized, it will provide service to coroipcc.so
library users.
The second library is coroipcc.so which is responsible for the client
implementation. This library should be linked with the client and
requires no initialization. This library provides communication
functions for sending and receiving synchronous requests, and also
reading asynchronous message requests from the server.
Initializing the coroipcs library
To use the coroipcs library, the developer creates a
coroipcs_init_state structure and populates it with function names.
The functions do various operations described in coroipcs_init(3) man
page. Not all operations must be specified. If some are missing, the
corosync ipcs library will automatically populate the structure with
internal versions which provide basic functionality.
Communicating with the coroipcc clients
Every ipc connection is represented by a void * pointer which uniquely
identifies the data set for the IPC connection. Each IPC connection
also contains user defined private data. To obtain this private data
pointer, the function coroipcs_private_data_get(3) function can be
called.
There are two channels for communication. The primary channel is the
synchronous request/response channel. Another channel is available for
out of band asynchronous responses from the server.
To send a response on the syncronous channel,
coroipcs_response_send(3) or coroipcs_response_iov_send(3) should be
used. To send a message on the asynchronous channel,
coroipcs_dispatch_send(3) or coroipc_dispatch_iov_send(3) should be
used.
The abstracted poll system
There are many different poll systems in use in applications. They are
usually intricately tied up in the implementation of the application
and each provide different APIs and functionality. To manage this, the
coroipcs library provides callbacks in coroipcs_init(3) which should be
called when a new connection should be added for accept system calls or
to dispatch messages.
These callbacks add the relevant fd to the application’s poll system.
When the application poll system triggers the callback registered by
the user defined poll adding functions, they then call either
coroipc_handler_accept(3) or coroipc_handler_dispatch(3).
Initializing the coroipcc library
No initialization is required in the coroipcc library.
Lifecycle of an IPC connection.
An IPC connection is made to the server with
coroipcc_service_connect(3). This function connects to the server and
requests channels be created for communication. To disconnect, the
client either exits or executes the function
coroipcc_service_disconnect(3).
Synchronous communication
There are two functions for sending a request and receiving a response.
The first function coroipcc_msg_send_reply_receive(3) sends an iovector
request and receives a response. This function copies the response
into the response buffer. the second function
coroipcc_msg_end_reply_receive_in_buf(3) does not copy the response
buffer and allows for zero-copy reading of the response when the
lifetime of the response buffer is known.
Asynchronous communication
The coroipcc_dispatch_recv(3) function receives an out-of-band
asyncronous message. Unlike the synchronous communication channel, the
asynchronous messages are queued and can provide very high out-of-band
performance.
To determine when to call coroipcc_dispatch_recv(3) the
corosync_fd_get(3) call is used to obtain a file descriptor used in the
poll(2) or select(2) system calls.
Finally the current dispatch flow control state can be obtained with
coroipcc_flow_control_get(3).
Performance
The ipc system is tuned for very high performance while also being
comletely abstracted from the underlying poll mechanism and any
internalisms required by the server. The ipc system achieves such high
performance by using shared memory as oppossed to slower techniques
such as UNIX_PF sockets.
We intend to do further development to allow syncronous requests to
return messages in an asyncronous way to avoid blocking involved in the
syncronous request/response model used today for higher throughput in
some use cases.
Security
The ipc system uses default operating system security mechanics to
ensure ipc connections are validated. A callback used with
coroipcs_init(3) is called for every new ipc connection with the
parameters of UID and GID. The callback then determines if the UID and
GID are authenticated for communication. More about this topic can be
viewed in the coroipcs_init(3) man page.
SEE ALSO
coroipcs_ipc_init(3), coroipcs_ipc_exit(3),
coroipcs_private_data_get(3), coroipcs_respone_send(3),
coroipcs_response_iov_send(3), coroipcs_dispatch_send(3),
coroipcs_dispatch_iov_send(3), coroipcs_refcount_inc(3),
coroipcs_refcount_dec(3), coroipcs_handler_accept(3),
coroipcs_handler_dispatch(3),
cooripcc_service_connect(3), coroipcc_service_disconnect(3),
coroipcc_msg_send_reply_receive(3),
coroipcc_msg_send_reply_receive_in_buf(3), coroipcc_dispatch_recv(3),
coroipcc_fd_get(3), coroipcc_dispatch_flow_control_get(3)