GSM Gateway
Building GSM Gateway from scratch
Matej Tomčík

DOWNLOAD

GSM Gateway modem, service, interface and emulator

Objective

Since I own Netduino Plus and SIM900 GPRS Shield, I decided to find a use for these two components and build a GSM gateway. The aim of this project was to provide basic functionality for remote clients to send SMS messages through a web interface along with the ability of reception of incoming SMS messages and phone calls. To put this all together I had to use C#, .Net Micro Framework, C++, Boost, MySQL, PHP and JavaScript.

Netduino Plus SIM900 GPRS Shield

Functional layout

Because this was not my first attempt to create a GSM gateway (two previous attempts failed because of complexity), I decided to analyze the problem before I started doing any programming. One can say that sending an SMS message through a GSM modem takes a few lines of code to do it. It is a simple task unless you want your gateway to do more than send SMS messages. In my case, I needed to receive SMS and phone calls as well and then save these into a database. To accomplish this, you need to divide the solution into multiple individual parts.

Netduino

As some of you may know, GSM modems use serial communication to talk to a terminal. In my project, Netduino represents the terminal. For example, to send an SMS message, Netduino issues AT+CMGS command, then sends the message body and then waits for acknowledgement as seen below:

AT+CMGS="00421944176952"
> Hello world
+CMGS: <NUM>
OK

The main issue with such a terminal is that you want to be notified of incoming calls or SMS messages while being able to issue client requests to the modem. But you cannot have two threads receiving input from the modem, because it gets intercepted by one of them, not all of them. And you cannot have one thread doing all the work, requests must be processed asynchronously at the time of reception, the same goes for notifications issued by the modem which have highest priority.

I managed to resolve this issue by dividing the terminal firmware into four threads:

Lets examine two scenarios.

First, a remote client wants to send an SMS messages

He issues a request which gets intercepted by the server thread. Server ensures there is no request being processed. Then it creates a new command to be issued to the modem (see image below for all commands supported by the terminal), sets it as the only one active command and executes first sequence, which results in sending modem this line:

AT+CMGS="<RECIPIENT>"

Server thread then resumes its execution. Once the modem is ready to accept the message body, it issues:

> 

As soon as this sequence is received, main thread unblocks and starts processing. First of all it checks whether there is an active command being processed. If yes then it forwards this response to the command ProcessResponse handler which decides whether the command has been completed, failed or needs more processing. Once the command completes, it generates a response and schedules it for completion. Requests scheduler dequeues this response and sends it back to the remote client.

// Receive next line from modem
while ((line = Receive(modem)) != null)
{
  // Main thread must lock IssuedCommand, because server may have issued
  // command to the modem and did not set the active command yet
  // but is holding mutex
  lock (IssuedCommand.SyncRoot)
  {
    // Here, the issued command is always the last executed command
    issuedCommand = IssuedCommand.ActiveCommand;
 
    // If there is an issued command awaiting to be completed
    if (issuedCommand != null)
    {
      try
      {
        // Process response
        if (issuedCommand.ProcessResponse(modem, line))
        {
          // Complete issued command
          issuedCommand.Complete(outputChannel, requestsScheduler);
 
          // Create next command in the chain and set it as active command
          IssuedCommand.SetActiveCommand(issuedCommand.Next(modem));
        }
        continue;
      }
      catch (UnrecognizedCommandException)
      {
        // Complete issued command
        issuedCommand.Complete(outputChannel, requestsScheduler);
 
        // Reset command
        IssuedCommand.SetActiveCommand(null);
      }
    }
    // ... default response handlers, see example #2

Second, an SMS message has been received by the modem

Modem sends a notification message to the terminal:

+CMTI: "SM",<NUM>

Main thread checks whether there is an active command being processed. If not, the message is caught by the default response handlers. One of them, dedicated to the +CMTI, simply creates a new command to read all unread SMS messages and sets it as the active command.

AT+CMGL="REC UNREAD"
+CMGL: 1,"REC UNREAD","<FROM>","<TO>","<DATE>"
<MESSAGE>
OK

Next cycle, once modem sends response to the AT+CMGL command, this response will be processed by ProcessResponse handler, which reads message header and content. Once there is no message to be read, ProcessResponse returns true, main thread will then call Next method which simply creates another command to delete all read messages. After that, a response is generated and sent through the output channel to the service which intercepts it and saves into the database.

if (line.StartsWith("+CRING")) // Incoming call
{
  // Request call number
  modem.Send("AT+CLCC", LineTermination.CarriageReturn);
  // Parse incoming call type
  line = line.Substring(8);
  IncomingCallType incomingCalltype = IncomingCallType.Voice;
  if (line != "VOICE")
  {
    if (line == "FAX")
      incomingCalltype = IncomingCallType.Fax;
    else if (line == "ASYNC")
      incomingCalltype = IncomingCallType.AsyncTransparent;
    else if (line == "SYNC")
      incomingCalltype = IncomingCallType.SyncTransparent;
    else if (line == "REL ASYNC")
      incomingCalltype = IncomingCallType.AsyncRelay;
    else if (line == "REL SYNC")
      incomingCalltype = IncomingCallType.SyncRelay;
  }
  // Initialize new incoming call
  IncomingCall incomingCall = new IncomingCall(incomingCalltype);
  // Await for +CLCC response
  IssuedCommand.SetActiveCommand(new IssuedClccCommand(incomingCall));
}
else if (line.StartsWith("+CMTI")) // New SMS received
{
  // List all unread received SMS
  modem.Send(IssuedCmglCommand.Command, LineTermination.CarriageReturn);
  // Await for +CMGL response
  IssuedCommand.SetActiveCommand(new IssuedCmglCommand());
}

Issued commands

PC service

PC service (or server service), is the most complex piece of the puzzle. It consists of multiple worker threads, asynchronous queues and fault protections, just to make life easier for the Netduino firmware and the web service.

PC Service

When the service starts, it first connects to the database. Without a valid connection, no request can be made or on message received. After the connection is established, program creates modem repository, an auto-updating list of active modems. This initialization triggers the first discovery request, which sends a broadcast message requesting all modems to send back their product informations. If no modem within a given time interval responds, the request will be resent until there is at least one active modem in the repository. This operation runs in a separate thread, so the main thread continues initialization by creating input channel (reception of incoming messages), requests pool (all requests that need an active modem are processed here), requests scheduler (keeps track of active requests), server (HTTP server to control the service) and finally it creates a snapshot.

// ... includes and static variables
 
// Program entry point
int main(int argc, char* argv[])
{
  // Register Ctrl+C handler
  SetConsoleCtrlHandler(ConsoleHandlerRoutine, TRUE);
 
  // Initialize service uptime
  Terminal::ServiceSnapshot::InitializeUptime();
 
  // Repeat until the service should stop
  while (!closeService)
  {
    try
    {
      // First initialize connection with the database
      Terminal::Database database;
      // Initialize modem repository in prior to the input channel, because input
      // channel may receive a Hello message and tries to discover new devices in the
      // network. The modem repository is responsible for device discovery
      Terminal::ModemRepository modemRepository(database);
      // At this point a "ResetActiveModemSessions" was called resetting all active
      // modem sessions in the database so no conflict can happen due to invalid entries
      modemRepository.Discover();
 
      // Initialize input channel completion queue
      Terminal::InputChannelQueue inputChannelQueue(database);
      boost::thread inputChannelQueueThread(boost::ref(inputChannelQueue));
 
      // Initialize input channel so we start receiving events as soon as possible
      Terminal::InputChannel inputChannel(database, modemRepository, inputChannelQueue);
      boost::thread inputChannelThread(boost::ref(inputChannel));
 
      // Initialize requests pool
      Terminal::ServerRequestsPool requestsPool;
 
      // Initialize notifications list
      Terminal::NotificationsList notifications;
 
      // Initialize requests scheduler
      Terminal::RequestsScheduler scheduler(database, modemRepository, requestsPool,
        notifications);
      // At this point all requests in the database with the status of 'Processing' are
      // reset to default state
      boost::thread schedulerThread(boost::ref(scheduler));
 
      // Initialize server
      Terminal::Server server(requestsPool, modemRepository, notifications, scheduler);
      boost::thread serverThread(boost::ref(server));
 
      // Initialize snapshot
      Terminal::ServiceSnapshot snapshot(inputChannelQueue, modemRepository, requestsPool,
        database, scheduler, notifications);
 
      // Wait for runtime requesting termination
      terminationRequest.Wait();
 
      // Close modem repository
      modemRepository.Close();
      // Stop scheduler
      scheduler.Close();
      schedulerThread.join();
      // Stop server
      server.Close();
      serverThread.join();
      // Purge notifications
      notifications.Purge();
      // Stop requests pool
      requestsPool.Close();
      requestsPool.WaitForCompletion();
      // Stop input channel
      inputChannel.Close();
      inputChannelThread.join();
      // Stop input channel completion queue and wait for it to complete
      inputChannelQueue.Close();
      inputChannelQueueThread.join();
 
      // Reset termination request if the runtime should only restart
      if (!closeService)
        terminationRequest.Reset();
    }
    catch (const std::exception & ex)
    {
      Terminal::ServiceSnapshot::IncreaseFaults();
      // In case of an exception
      VERBOSE_EXCEPTION("Main") << ex.what() << std::endl;
    }
    catch (...)
    {
      Terminal::ServiceSnapshot::IncreaseFaults();
      // In case of an exception
      VERBOSE_EXCEPTION("Main") << std::endl;
    }
  }
}
 
// Console handler to handle Ctrl+C
BOOL WINAPI ConsoleHandlerRoutine(DWORD dwCtrlType)
{
  // Catch Ctrl+C only
  if (dwCtrlType == CTRL_C_EVENT)
  {
#ifdef VERBOSE_INFO
    { VERBOSE_INFORMATION << "Termination request" << std::endl; }
#endif
    // Issue termination request
    Terminal::IssueTerminationRequest(true);
 
    // Return TRUE to indicate that we have taken care of the program termination
    return TRUE;
  }
  else
    // Let the default function handle the control code
    return FALSE;
}
 
namespace Terminal
{
 
// Issues termination request
inline void IssueTerminationRequest(const bool stop)
{
  closeService = stop;
  // Signals termination request
  terminationRequest.Set();
}
 
}

If any error occurs at any stage, already initialized resources are freed and the whole initialization process starts again. Once the service is up and running, only a remote client can stop it. To demonstrate how does the service work, we have to use an example.

Sending an SMS message through the service

When a client wants to send an SMS message, it has to first insert the message into the database. Then it optionally sends a PUSH message to the service, requesting the scheduler to select all not processed messages from database. If no PUSH message is issued, the scheduler will process the request in the next cycle.

Once the message is about to be executed, a new worker thread is created for the request, taking the message as a parameter. The thread is started and request begins its execution by requesting the modem repository to create a modem request. Unlike server request, modem request has its own queue in which it gets executed. Simply described as a packet sent to the terminal awaiting to be completed. Once the modem request is assigned, the message can be delivered through it to the terminal awaiting an acknowledgement whether or not the execution completed successfuly.

When the request completes (either successfuly or with an error), it updates message state in the database to "Completed" and optionally notifies the remote client about the completion. The notification is sent only when the client sends a PUSH command together with a parameter indicating that a notification is required. In that case a new entry is created in the notifications list and keeps the connection alive until completion or cancellation. If the request execution takes too long, the notification can be cancelled with a "Timeout" error, however this does not specify whether the request completed successfuly or not.

Web interface

In order to enable remote clients to send SMS messages, I had to make a web service and a web interface. Web service simply accepts requests, stores them in the database and notifies the service. The web interface (control panel) allows me to see sent/received SMS messages and phone calls, service status messages, modems being present on the network, manage remote accounts and most important, to control the service. Both web service and control panel are built using PHP and JavaScript.

Control panel Control panel - snapshot

Testing

The most difficult part was the testing. Take for example Netduino. I would have to rebuild the firmware and deploy it every time I made the smallest change in the code. Also I could not just send SMS messages until I was absolutely sure that the code behind is working properly. So I had to come up with a solution, otherwise it would be a waste of time and money.

Modem emulator Terminal GUI terminal Lua modem

So I made these projects, just to make the debugging: