Protocol walkthroughs

Firstly, as seems mandatory when introducing new programming languages, let's look at the traditional ''Hello World'' example written as a cnet protocol. Although cnet protocols are written in ANSI-C, this example looks quite different:

#include <cnet.h> void reboot_node(CnetEvent ev, CnetTimer timer, CnetData data) { printf("hello world\n"); }

Things to first note are that the source code includes the standard cnet header file, as must all cnet protocol source files, and that there is no traditional main() function, nor a call to exit(), nor a return value. However, this ''protocol'' does run - each node implementing this protocol is rebooted when cnet invokes its event-handler for the EV_REBOOT event. The handler is passed the parameters of ev=EV_REBOOT, timer=NULLTIMER, and data=0, none of which are used by the handler. The handler prints out its message "hello world\n" using the ANSI-C function printf(), the message appears on the standard output window of each node (if running under Tcl/Tk), and the handler finally returns to whoever called it (in this case, the cnet simulator's scheduler).

In fact, the protocol is so simple that it doesn't even need the usual cnet topology file. We can commence its execution just using cnet's command-line options:

We can even replace the 2, above, with 101 to run the protocol on a much larger, random, network. Once every node in the network has rebooted and printed "hello world\n" the whole simulation keeps running, but doing nothing else because there is nothing left for it to do (strictly speaking, cnet could detect that there are no more significant events to schedule, and just terminate). All noteworthy but pointless.

Introducing timer events

#include <cnet.h> void reboot_node(CnetEvent ev, CnetTimer timer, CnetData data) { CnetInt64 when; /* Indicate that we are interested in the EV_TIMER1 event */ CHECK(CNET_set_handler(EV_TIMER1, timeouts, 0)); /* Request that EV_TIMER1 occur in 1sec, ignoring the return value */ int64_I2L(when, 1000000); CNET_start_timer(EV_TIMER1, when, 0); }

The first cnet function called is CNET_set_handler(). This instructs cnet to remember that for this node that any time the EV_TIMER1 event occurs, that cnet should call the function timeouts() to handle the event. The third parameter to CNET_set_handler() is actually ignored for this call, but must always be provided. The call to CNET_set_handler() is also ''wrapped'' in the function CHECK(), described in the Frequently Asked Questions, to ''automatically'' detect any error occuring from the call to CNET_set_handler().

Finally, in reboot_node(), the function CNET_start_timer() is called to request that the EV_TIMER1 event be raised (once) in 1 second's time. All times in cnet are measured in microseconds, and so we use 64-bit integers to hold these values (which can grow quite quickly). The CnetInt64 datatype is provided by cnet to simplify the use of these values. When the EV_TIMER1 event occurs, the third parameter to CNET_start_timer(), here 0, will be faithfully passed to the event handler for EV_TIMER1, which we know to be the timeouts() function. Next, let's look at the timeouts() function, which we know will be called 1 second after the reboot_node() handler is finished:

void timeouts(CnetEvent ev, CnetTimer timer, CnetData data) { static int which = 0; CnetInt64 when; printf("%3d.\t%s\n", which, (which%2) == 0 ? "tick" : "\ttock"); ++which; /* Reschedule EV_TIMER1 to occur again in 1 second */ int64_I2L(when, 1000000); CNET_start_timer(EV_TIMER1, when, 0); }

The timeouts() function will be called with three parameters: ev=EV_TIMER1, timer=somevalue, and data=0. The value of the second parameter, timer will actually be the return value of the call to CNET_start_timer() in reboot_node(). The return value is of type CnetTimer, but we are not concerned with its value in this simple protocol. The data=0 value is the third parameter given to the CNET_start_timer() call in reboot_node().

The first time that timeouts() is called, it will simply print the string 0. tick. It then calls CNET_start_timer() to again request that the handler for EV_TIMER1 be called in a further 1 second. Although timeouts() is, itself, the handler for EV_TIMER1, this is not a recursive call as no handler will be called until this current handler has returned. The timeouts() function will be called, again, in another 1 seconds and will, this time, simply print 1. tock. That's all it does - printing 0. tick, 1. tock, 2. tick, 3. tock ... every second. It's still not really a protocol.

Introducing the Physical Layer and debug events

#include <cnet.h> #include <ctype.h> #include <string.h> static int count = 0; static char *fruit[] = { "apple", "pineapple", "banana", "tomato", "plum" }; #define NFRUITS (sizeof(fruit) / sizeof(fruit[0]))

The first function called is, of course, reboot_node() to handle the EV_REBOOT event. In this handler we simply register this node's interest in a number of events, passing to CNET_set_handler() the addresses of the functions to be called when each event is raised. We also call CNET_set_debug_string() to place an indicated string/label on two of the five debug buttons available under Tcl/Tk. Out of habit, we wrap each call with CHECK to ensure that there have been no errors.

void reboot_node(CnetEvent ev, CnetTimer timer, CnetData data) { /* Indicate our interest in EV_DEBUG1, EV_DEBUG2 and EV_PHYSICALREADY */ CHECK(CNET_set_handler(EV_PHYSICALREADY, frame_arrived, 0)); CHECK(CNET_set_handler(EV_DEBUG1, send_reliable, 0)); CHECK(CNET_set_debug_string( EV_DEBUG1, "Reliable")); CHECK(CNET_set_handler(EV_DEBUG2, send_unreliable, 0)); CHECK(CNET_set_debug_string( EV_DEBUG2, "Unreliable")); }

We next look at two very similar functions. Each chooses a random character string (a fruit), uses it to format a frame for transmission, and determines the length of the frame (adding 1 for its NULL-byte terminator). The length of the frame and a checksum of the frame (using the checksum_internet() function provided by cnet) are next reported. The frame is finally transmitted across the Physical Layer. In handler send_reliable() we call CNET_write_physical_reliable() to bypass cnet's Error Layer. Handler send_unreliable() calls CNET_write_physical(), and because CLICK.top introduces frame corruption with a probability of one in four, some of the frames will be corrupt during transmission.

The three parameters to the CNET_write_physical* functions provide the required link number (1), the address of the frame to be transmitted, and the length (in bytes) of the frame. The length is passed by reference because, on return, the functions indicate how many bytes they accepted by possibly modifying this reference parameter. We will assume that all went well, and that all bytes were actually transmitted. There is no real need to have two such similar functions - we could, instead, have a single function which handles both of the EV_DEBUG1 and EV_DEBUG2 events and calls the appropriate transmission function after inspecting the value of ev.

static void send_reliable(CnetEvent ev, CnetTimer timer, CnetData data) { char frame[256]; int length; sprintf(frame, "message %d is %s", ++count, fruit[random()%NFRUITS]); length = strlen(frame) + 1; printf("sending %d bytes, checksum=%6d\n\n", length, checksum_internet((unsigned short *)frame,length)); CHECK(CNET_write_physical_reliable(1, frame, &length)); } static void send_unreliable(CnetEvent ev, CnetTimer timer, CnetData data) { char frame[256]; int length; sprintf(frame, "message %d is %s", ++count, fruit[random()%NFRUITS]); length = strlen(frame) + 1; printf("sending %d bytes, checksum=%6d\n\n", length, checksum_internet((unsigned short *)frame,length)); CHECK(CNET_write_physical(1, frame, &length)); }

Finally we handle the arrival of a frame with the handler for the EV_PHYSICALREADY event, frame_arrived(). We first determine the maximum length of the frame that we are able to receive. If this length is sufficient to receive the incoming frame, the next call to CNET_read_physical() will inform us of the link number on which the frame arrived (here, it will be 1), copy the frame into the address/space that we have provided, and inform us of the frame's actual length (i.e. length will be modified).

static void frame_arrived(CnetEvent ev, CnetTimer timer, CnetData data) { char frame[256]; int link, length, i, ch; length = sizeof(frame); CHECK(CNET_read_physical(&link, frame, &length)); printf(" %d bytes arrived on link %d, checksum=%6d : \"", length, link, checksum_internet((unsigned short *)frame,length)); for(i=0 ; i<length-1 ; ++i) { ch = frame[i]; if(!(isalnum(ch) || ch == ' ')) ch = '?'; putchar(ch); } printf("\"\n\n"); }

We next report the length and link number of the new frame, and again print its checksum (using the same checksum function as used by the sender). If the frame was corrupted in transmit, we will observe that the ''before and after'' checksum values will be different. Identical checksum values indicate that the frame arrived intact. We finally print out the actual bytes of the message - if the frame arrives intact, we will see the expected message carrying tha name of one of the fruits. If the frame was corrupted, we will see a '?' character for any unexpected byte.

A complete stopandwait Data-Link Layer protocol

This implementation is based on Tanenbaum's `protocol 4', 2nd edition, p227 (or his 3rd edition, p205). This protocol employs only data and acknowledgement frames - piggybacking and negative acknowledgements are not supported.

We first define some global types, data structures, and variables for this protocol. It is important to understand that each of these is unique to each of the nodes in the simulation. Although each of the nodes will typically use the same source code file, each node has its own local copy of its variables. It is not possible for one node to modify the variables in another node. The only way for the nodes to communicate is via the Physical Layer.

#include <cnet.h> #include <stdlib.h> #include <string.h> typedef enum { DATA, ACK } FRAMEKIND; typedef struct { char data[MAX_MESSAGE_SIZE]; } MSG; typedef struct { FRAMEKIND kind; /* only ever DATA or ACK */ int len; /* the length of the msg field only */ int checksum; /* checksum of the whole frame */ int seq; /* only ever 0 or 1 */ MSG msg; } FRAME; #define FRAME_HEADER_SIZE (sizeof(FRAME) - sizeof(MSG)) #define FRAME_SIZE(f) (FRAME_HEADER_SIZE + f.len) static MSG lastmsg; static int lastlength = 0; static CnetTimer lasttimer = NULLTIMER; static int ackexpected = 0; static int nextframetosend = 0; static int frameexpected = 0;

We first declare a type, MSG, to receive a message from the node's Application Layer. We do not know, nor care, what will be in these messages, and so it is reasonable to declare them as an array of bytes of some maximum length, MAX_MESSAGE_SIZE. We also declare a type, FRAME, to carry the messages across the Physical Layer. Each instance of FRAME consists of two parts - firstly, the frame header consisting of four fields, and then the frame body or payload which is actually a message. We define two txtual constants in C, FRAME_HEADER_SIZE and FRAME_SIZE(f) to simplify the coding of our protocol. We finally define six global variables - three to keep a copy of, and remember atttributes of the last message received from the Application Layer, and three integers to track the sequence numbers of the stopandwait protocol. Note that the variables lastmsg, lastlength, lasttimer, nextframetosend, and ackexpected are all required by the sender of the protocol, and that only frameexpected is required by the receiver. However, because each node executes its own copy of the compiled code, using its own variables, and at any time takes on the role of either sender or receiver, the approach of defining all variables together is considered reasonable.

We next look at the mandatory reboot_node() function, and the simple handler of EV_DEBUG1 which simply prints the runtime state of the stopandwait protocol.

void reboot_node(CnetEvent ev, CnetTimer timer, CnetData data) { if(nodeinfo.nodenumber > 1) { fprintf(stderr,"This is not a 2-node network!\n"); exit(1); } CHECK(CNET_set_handler( EV_APPLICATIONREADY, application_ready, 0)); CHECK(CNET_set_handler( EV_PHYSICALREADY, physical_ready, 0)); CHECK(CNET_set_handler( EV_DRAWFRAME, draw_frame, 0)); CHECK(CNET_set_handler( EV_TIMER1, timeouts, 0)); CHECK(CNET_set_handler( EV_DEBUG1, showstate, 0)); CHECK(CNET_set_debug_string( EV_DEBUG1, "State")); if(nodeinfo.nodenumber == 1) CNET_enable_application(ALLNODES); } static void showstate(CnetEvent ev, CnetTimer timer, CnetData data) { printf( "\tackexpected\t= %d\n\tnextframetosend\t= %d\n\tframeexpected\t= %d\n", ackexpected, nextframetosend, frameexpected); }

As this file only provides a Data-Link layer protocol, we first ensure that this is only a network of 2 nodes. Note that if any node calls the C function exit(), that the whole simulation will terminate. There is little else new here other than handlers for the EV_APPLICATIONREADY and EV_DRAWFRAME events.

cnet provides the ability for our interior protocols to control the rate of new messages coming ''down'' from the Application Layer. We do this by enabling and disabling the Application Layer or, more particularly, by enabling and disabling the generation of messages for certain remote hosts. We need to call CNET_enable_application() in at least one node for anything further to happen. This protocol is written so that only one node (number 0) will generate and transmit messages and the other (number 1) will receive them. This self-imposed restriction makes it easier to understand early protocols. The restriction can easily be removed by removing the line

    if(nodeinfo.nodenumber == 1)

in reboot_node(). Both nodes will then transmit and receive (why?). The meaning and handling of the EV_DRAWFRAME event is fully described elsewhere - see Drawing data frames in cnet.

The first thing of interest that will occur after each node has rebooted is that one node's Application Layer will generate and announce a new message for delivery. We handle the EV_APPLICATIONREADY event with our application_ready() function:

static void application_ready(CnetEvent ev, CnetTimer timer, CnetData data) { CnetAddr destaddr; lastlength = sizeof(MSG); CHECK(CNET_read_application(&destaddr, (char *)&lastmsg, &lastlength)); CNET_disable_application(ALLNODES); printf("down from application, seq=%d\n",nextframetosend); transmit_frame(&lastmsg, DATA, lastlength, nextframetosend); nextframetosend = 1-nextframetosend; }

We first determine the maximum sized message that our protocol can handle and pass that, by reference, to the CNET_read_application() function. Asssuming that we have provided enough buffer space, on return CNET_read_application() informs our interior protocol of the intended network destination of the new message's destination, copies the actual message into our variable lastmsg, and modifies lastlength to tell us how long the message is. We next call CNET_disable_application() to tell our node's Application Layer to stop generating messages (for any node). We finally pass the new message to our function, transmit_frame() (shown shortly), along with parameters in support of our stopandwait protocol.

Our transmit_frame() function performs the final actions before something is transmitted across the Physical Layer. Parameters provide the message to be transmitted, an indication as to whether it is data or an acknowledgment, its length, and its sequence number as part of the stopandwait protocol.

static void transmit_frame(MSG *msg, FRAMEKIND kind, int msglen, int seqno) { FRAME f; f.kind = kind; f.seq = seqno; f.checksum = 0; f.len = msglen; if(kind == ACK) printf("ACK transmitted, seq=%d\n",seqno); else if(kind == DATA) { CnetInt64 timeout; float f1; memcpy(&f.msg, (char *)msg, msglen); printf(" DATA transmitted, seq=%d\n",seqno); int64_L2F(f1, linkinfo[1].propagationdelay); int64_F2L(timeout, 3.0*(f1 + 1000000*(FRAME_SIZE(f) * 8.0)/linkinfo[1].bandwidth)); lasttimer = CNET_start_timer(EV_TIMER1, timeout, 0); } msglen = FRAME_SIZE(f); f.checksum = checksum_ccitt((unsigned char *)&f, msglen); CHECK(CNET_write_physical(1, (char *)&f, &msglen)); }

We initialize the header fields of a frame, of type FRAME, and, if data, embed the message into the frame, by copying the bytes of the message into the corresponding field of the frame. Again, if the message is data, we need to estimate the amount of time that it will take for the message to travel to its destination, be processed by the remote node, and for its acknowledgment frame to return. It is important that we keep the units of our calculation correct - a link's propagation delay is measured in microseconds, the frame's size in bytes, and a link's bandwidth in bits per second. We multiply the whole calculation by 3 for a reasonable estimate of the upper bound for the complete round-trip time. We call CNET_start_timer() to have the EV_TIMER1 event raised sometime after we expect the acknowledgment to return.

We finally calculate a checksum of the frame to be transmitted, embed the value of the checksum in the frame itself(!), and call CNET_write_physical() to transmit the frame on link 1.

Next, we provide the most complex handler of the EV_PHYSICALREADY event, which is invoked when a frame arrives, via a link, at the Physical layer. We first call CNET_read_physical() to read in the frame, first telling it how much buffer space we are providing to receive the frame. On return, the function tells us on which link the frame arrived, copies the frame to our provided buffer space, and tells us how long (in bytes) the frame is. We again use CHECK() to automatically detect any errors.

We next use one of the provided checksum functions to determine if the frame has been corrupted in its travel. Although not demanded by cnet, is is necessary to use the same checksum function on both sender and receiver. We extract the expected checksum, as calculated by the sender, from the frame itself(!) and compare it with the locally calculated value. It is unwise to attempt to make any sense of any of the contents of a corrupted frame. If we detect corruption, we simply ignore the newly arrived frame, confident that it will be retransmitted in the future.

If the frame is an expected acknowledgment, we know that its corresponding data frame has arrived safely, and so we stop the retransmission timer. Gloating with pride, we call CNET_enable_application() so that the Application Layer may generate the next message for delivery.

If the frame is expected data, we write a copy of the frame's embedded message (only) to our Application Layer with CNET_write_application(). We again use CHECK() to automatically detect if our protocol has let through any errors. Finally, if the frame was a data frame, the expected one or not, we reply with an acknowledgment frame using transmit_frame(), described earlier.

static void physical_ready(CnetEvent ev, CnetTimer timer, CnetData data) { FRAME f; int link, len, checksum; len = sizeof(FRAME); CHECK(CNET_read_physical(&link, (char *)&f, &len)); checksum = f.checksum; f.checksum = 0; if(checksum_ccitt((unsigned char *)&f, len) != checksum) { printf("\t\t\t\tBAD checksum - frame ignored\n"); return; /* bad checksum, ignore frame */ } if(f.kind == ACK) { if(f.seq == ackexpected) { printf("\t\t\t\tACK received, seq=%d\n", f.seq); CNET_stop_timer(lasttimer); ackexpected = 1-ackexpected; CNET_enable_application(ALLNODES); } } else if(f.kind == DATA) { printf("\t\t\t\tDATA received, seq=%d, ", f.seq); if(f.seq == frameexpected) { printf("up to application\n"); len = f.len; CHECK(CNET_write_application((char *)&f.msg, &len)); frameexpected = 1-frameexpected; } else printf("ignored\n"); transmit_frame((MSG *)NULL, ACK, 0, f.seq); } }

If the topology file sets either of the probframecorrupt or probframeloss link attributes, and the data frame is corrupt or lost, then the standard stopandwait protocol will not send an acknowledgment from the receiver back to the sender. Moreover, even if the data frame arrives safely, the acknowledgment frame itself may be corrupt or lost on its return. We thus require a ''timeout'' function to cause the retransmission of a frame if the sender has not seen a valid acknowledgment frame after some period of time. Our timeouts() function handles the EV_TIMER1 event for the timer that was initially started in transmit_frame(). Note that we do not need to explicitly stop a timer if its handler is invoked - it is stopped implicitly by cnet.

static void timeouts(CnetEvent ev, CnetTimer timer, CnetData data) { if(timer == lasttimer) { printf("timeout, seq=%d\n", ackexpected); transmit_frame(&lastmsg, DATA, lastlength, ackexpected); } }

Done. A real Data-Link Layer protocol providing reliable delivery in the presence of frame corruption and loss.

Lessons learnt

• Each protocol source file must include the <cnet.h> header file.
• All nodes actually execute the same (read-only) copy of the compiled protocol code, but each node has its own copy of any variables in the code - after all, each node is a simulating a distinct computer, and they don't share their RAM. Nodes may only communicate using the Physical Layer.
• Each node's protocol code must have one function to receive the initial EV_REBOOT event. By default, this is named reboot_node() but may be renamed with the -R command-line option.
• All event handlers are invoked with three parameters providing the event (the reason for being invoked), a timer (often NULLTIMER if there is no meaningful timer), and a user provided data value.
• The reboot_node() handler should indicate which future events are of interest with CNET_set_handler() and place labels on the debug buttons with CNET_set_debug_string().
• Event handlers should simply perform their responsibilities as quickly as possible, and then return to enable other handlers to be invoked. They should not loop indefinitely, nor attempt to sleep, wait or poll for other events.
• Calls to some functions, such as CNET_read_application() and CNET_read_physical(), must first indicate, using a parameter passed by reference, how much buffer space they are providing to receive some data. On return, these functions modify the same parameter to report how much buffer space was actually required and used.
• If a frame appears to have been corrupted in its travel (as determined by one of the provided checksum functions), it is unwise to attempt to make any sense of any of the contents of the frame.
• Most functions provided by cnet return 0 on success and 1 on failure (with the obvious exception of the CNET_start_timer() function). Most function calls may be ''wrapped'' with the CHECK() function to automatically detect and report most errors. It indicates a serious error in your protocols if one of the cnet functions reports an error and your protocol ignores it.
• We do not need to explicitly stop a timer if its handler is invoked - it is stopped by cnet. However, it is not really a fatal error to stop a timer that has already expired.

Good luck.