socktools.base_sock - base socket implementation

This module provides the base classes used in the other modules, you should probably ignore it unless you plan to implement a new socket type.

class socktools.base_sock.BaseSock(bind=None, connect=None, handlers={}, timeout=10, tick_interval=0.25, sock=None, meta_sock=None, logger=None, no_async=False)

Bases: object

The base class from which specific socket classes inherit

This class is the base socket class which handles queuing of messages and the main event loop. It should not be used directly but rather subclassed or one of the existing child classes used.

Keyword Arguments:  

• bind (tuple) – if not None, specifies a TCP/IP endpoint as (ip,port) to bind to and listen on
• connect (tuple) – if not None, specifies a TCP/IP endpoint as (ip,port) to connect to
• handlers (dict) – maps message types to list of message handler functions
• timeout (int) – time in seconds before a peer is considered to have timed out
• tick_interval (int) – time in seconds to wait between ticks
• sock (socket.socket) – if not None, specifies a socket object to be used - should be used only for testing
• meta_sock (meta_sock.MetaSock) – if not None, specifies the meta socket this socket belongs to, see meta_sock.py
• logger (logging.Logger) – a logger object
• no_async (bool) – if set True, no attempts will be made to go async except for sending messages

known_peers

dict

all currently connected peers have an entry in this dict, the contents of the dict are another dict with metadata

pool

eventlet.GreenPool

all greenlets spawned by the socket belong to this pool, with a default concurrency of 1000

parse_q

eventlet.queue.LightQueue

raw packet data is added to this queue before being parsed, each item is a tuple of (data,addr)

in_q

eventlet.queue.LightQueue

parsed messages are added to this queue after being parsed

timeout

int

the timeout interval in seconds - peers must talk to us on a regular basis to avoid being timed out

tick_interval

int

the tick interval - what exactly a tick does is up to the application

handlers

dict

maps message types to list of message handler functions

sock

arbitrary object representing the physical socket, defaults to an instance of DummySocket

meta_sock

see meta_sock.py

logger

logging.logger

the logger to use for this socket

active

bool

indicates whether this socket is active and working

add_handler(msg_type, handler, exclusive=False)

Add a handler for the specified message type

This method adds message handlers to the socket after it has been setup, allowing dynamic handlers.

It is preferable to use static handlers whenever possible

Parameters:

• msg_type – the message type the handler is for, this depends on application but is usually an int
• handler – the handler function to add, must accept params (addr,msg_type,msg_data)

Keyword Arguments:  

exclusive (bool) – if True, deletes all previously set handlers for this message type

child_setup()

Convenient setup hook for child classes

To avoid the need to rewrite __init__ when inheriting from this class, __init__ calls this method before spawning threads and going active.

Without this hook, any child class that adds extra params to __init__ would need to have a copy of the implementation as it is not possible to modify certain things after threads are spawned.

See msgtype_mixin.py for a good example of how to use this.

In the default implementation it does nothing.

close_peer(peer)

Removes a peer from the known_peers list and does any required cleanup

Parameters:peer (tuple) – the peer to close as a TCP/IP endpoint tuple, i.e (ip,port)

Notes

The default implementation simply removes the peer from the known_peers list, any child class should do the same when overriding this method

connect_to(endpoint)

Connect to a specified endpoint

This method should be overridden and used to implement any authentication etc before adding the specified endpoint to the known peers list In the default implementation nothing is done here except adding the peer to the known peers list

Arg:

endpoint (tuple): the TCP/IP endpoint as (ip,port)

create_socket()

Create the physical socket object

To be of any practical use, this should be overridden

Returns:The physical socket object, in the default implementation this is a DummySocket instance Return type:socket.socket

decode_msg(data)

Used internally - decodes raw messages

If encode_msg() is implemented, this should do the inverse

Parameters:data (str) – the message serialised as a string and still encoded Returns:the decoded but still serialised message Return type:str

encode_msg(data)

Encode a serialised message

This method is where you should implement any form of encoding required by the physical socket including things such as prefixing a length for TCP sockets.

You may also handle compression,encryption etc here.

In the default implementation this is an identity function.

Parameters:data (str) – the message serialised as a string Returns:the encoded message Return type:str

get_default_handlers()

Get a default handlers dict

When inheriting from this class you should setup default handlers needed by the protocol by overriding this method. Additionally, children should always call the parent for this method.

Returns:a dict mapping message types to handlers, can be 0-length but must be a dict Return type:dict

good_peer(peer)

Check if the specified peer is one we want to talk to

If this method returns False for a peer as it sends us a message, that message will be dropped and not parsed. The default implementation returns True for every peer

Parameters:peer (tuple) – the TCP/IP endpoint of the peer as an (ip,port) tuple Returns:True if the peer is good, otherwise False Return type:bool

handle_all(from_addr, msg_type, msg_data)

Called before doing anything with a decoded/parsed packet before all other handlers

This can be used as a hook to implement whatever you want with decoded packets - stuff like encryption and compression though belongs in parse_msg().

After being parsed all messages pass through this method and any transformations required can be applied. Most applications will not need this and the default implementation (which is a simple identity function) will work fine.

One application where this may be of use is to send “unknown peer” type messages if the address is not in the known peers dict.

Parameters:

• from_addr (tuple) – The remote peer’s TCP/IP endpoint as an (ip,port) tuple
• msg_type – The message type, what this is depends on the application but is usually an int
• msg_data – The message data, what this is depends on the application

Returns:

(from_addr,msg_type,msg_data) - the message, modified or not, by default this is identical to the params unless overridden

Return type:

tuple

handler_thread()

Used internally - reads from in_q and passes to the appropriate handler

Note

This method should only be run from inside the class and inside a greenthread

handler_wrapper(handler, addr, msg_type, msg_data)

Invokes the specified handler while catching exceptions

This method invokes a handler function while catching and logging any exceptions. If an exception is raised by the handler, BaseSock.log_error() is called to notify the end user

Parameters:

• handler (function) – a function accepting params (addr,msg_type,msg_data)
• addr (tuple) – TCP/IP endpoint for the peer that originated the message
• msg_type – the message type - this depends on the application but usually an int
• msg_data – the message data - this depends on the application but usually a tuple or dict

log_debug(msg)

Logs debug info - if debug mode is off, this method should do nothing

The default implementation prints the message to stdout

Parameters:msg (str) – message to log

log_error(msg, exc=None)

Log an error with exception data

This method logs errors and should be overridden to use whatever logging mechanism is appropriate in the end user application.

The default implementation simply prints the message and exception data to stdout

Parameters:msg (str) – error message

Keyword args

exc (Exception): a python exception related to the error

parse_msg(data)

Parse a raw message into a format usable by the application

This method should be overridden by the application as appropriate and handles message parsing from raw packets.

Parameters:data (str) – the raw packet to be parsed, this should be one whole message Returns:a tuple of (msg_type,msg_data) - the specific types of msg_type and msg_data depend on the application Return type:tuple

Note

If not overridden, by default this method returns a message of type 0 and the raw data as msg_data. Please also note that this method should NOT implement anything beyond parsing, see good_peer() and handle_all() for higher level logic.

parser_thread()

Used internally - reads from parse_q and puts parsed messages into in_q

Note

This method should only be run from inside the class and inside a greenthread

read_raw()

Read a single raw packet from the socket or sockets

If the underlying physical socket is UDP, this method should simply read from it and return as quickly as possible.

If the underlying physical socket is a TCP socket connected to one single peer, this method should read from it and return as quickly as possible.

If the underlying physical socket is a TCP server, this method should return a raw packet from the next available client.

In all cases, this method should return a raw packet without parsing of any kind beyond size checking.

If no data is available, this method should block using eventlet.greenthread.sleep(0) or equivalent until data is available.

Note

The default implementation together with DummySocket always returns a null-length string from localhost:1337 Using the default implementation with another physical socket type probably won’t work

Warning

this method must NOT block except via eventlet, failure to heed this warning will result in crippled performance

Returns:(data,from_addr) - data should be a string or byte buffer, from_addr should be a TCP/IP endpoint tuple in the form (ip,port) Return type:tuple

recv_thread()

Used internally - reads from the socket and puts the raw packets into parse_q

Warning

This method absolutely must not be called from anywhere except inside the class or bad things will happen

run_handler(addr=None, msg_type=None, msg_data=None)

send_msg(msg_type, msg_data, to_peer=None)

Encode and then send a single message to the specified peer

This method encodes and then sends a single message to the specified peer, or optionally to all connected peers.

Parameters:

• msg_type (int) – the type of message to send
• msg_data – format depends on the application, usually a tuple, list or dict

Note

This method relies upon the serialise_msg() method and the encode_msg() method - please ensure these methods are defined appropriately

Keyword Arguments:  to_peer (tuple) – the TCP/IP endpoint as a (host,ip) tuple to transmit to, if this is set to None then all peers will get the message

send_raw(data, to_peer=None)

Send a single raw packet from the socket to the specified peer

This method sends a single raw packet (already encoded as appropriate) to the specified peer, or optionally to all connected peers.

The whole packet must be transmitted before this method returns and it must block if required via eventlet.greenthread.sleep(0) or equivalent

Parameters:data (str) – the raw data to send, this must already be encoded and ready for transmission on the physical socket - including length if required Keyword Arguments:  to_peer (tuple) – the TCP/IP endpoint as a (host,ip) tuple to transmit to, if this is set to None then all peers will be sent the packet

Warning

This method must NOT block except via eventlet, failure to heed this warning will result in crippled performance. Further, it is vitally important to wrap the actual transmission in an appropriate try/catch block when broadcasting so that a failure to transmit to one client does not result in failing to transmit to others

serialise_msg(msg_type, msg_data)

Serialise a single message

This method is where you should implement your message serialisation code for custom protocols.

In the default implementation this function just returns str(msg_data)

Parameters:

• msg_type (int) – the message type
• msg_data – a python object to be serialised into a message

Returns:

the serialised message

Return type:

str

setup_logger(logger)

Setup a logger object to use instead of stdout

Often it makes sense to log stuff to a logger object instead of dumping it to stdout using print. This is especially useful for daemon processes where you may otherwise not see the output at all.

Parameters:logger (logging.logger) – The logger object to use

tick()

Perform application-specific tick

The default implementation does absolutely nothing, you should override this if you want tick functionality While debugging it may be helpful to insert a sleep in here to see what happens when you miss ticks

Parameters:diff (int) – the recorded time in seconds the last tick took to run, 0 on the first iteration

tick_thread()

Used internally to call the tick() method in a loop with accurate timing

Many network protocols, 3D gaming ones in particular, require a fixed frequency “tick”, that’s what is implemented here. This thread runs tick() in a loop and measures the time it took, adjusting the delay to compensate

Note

This code makes no guarantees at all that the specified tick interval is actually achievable - if you get missed ticks, optimise your tick() to be more async

timeout_thread()

Used internally - removes peers from the known_peers dict if they haven’t sent us a packet within the last timeout interval

Warning

This method must not be called from anywhere except inside the class, and only one instance should run at a time

class socktools.base_sock.DummySocket(sock_family=2, sock_type=2, proto=0)

Bases: eventlet.greenio.base.GreenSocket

A socket object that does nothing

recvfrom(buflen=8192, flags=0)

Dummy implementation of socket.recvfrom()

This method simply returns a null-length string coming from address (‘127.0.0.1’,1337)