SIOS — the serial protocol syntax I made

When I first started making AVR modules, there was no Raspberry Pi or ESP8266. Arduino was introduced in 2005, the same year I started with the AVR microcontroller. But I didn’t know of it until many years later.

There weren’t any Wi-Fi enabled microcontrollers at that time. Both Ethernet and Bluetooth interfaces were more expensive, and harder to interface. So I ended up using the good old serial port.

I made my first module with a serial interface in 2007. RS-232 was cheap, and easy to implement.

With RS-232 I got a way of getting data between the microcontrollers and the computer, but I still needed a defined syntax. So I set out to make one, and called it SIOS — Serial Input/Output System.

Motivation

My main project at the time, The Rack box, was getting big. Everything was controlled by microcontroller modules, acting on input signals. I wanted to bring all I/Os into the computer, which would allow me to create more advanced automations — quicker and easier.

The status panel for the rack box, consisting of LEDs and switches, was also getting full. Any changes or additions were time consuming, and difficult. Instead I wanted a software status panel that I could easily update and expand 😃

Syntax

I made some big changes to the syntax in 2013 — this became the 2nd revision. With rev. 2 the formatting and encoding was changes, but both revisions had the same basic parts:

• Module address
• Request type
• Request ID
• Request value
• Checksum

Revision 1

• : was used as a delimiter
• numeric values in base-10
• addressing started on 001 (hard coded)
• baud rate was 9600

001:o:3:1:132

The command above tells module 001 to set its output o number 3 to true 1.

Revision 2

• multiple delimiters used, to make it easier to identify different segments of the command
  • , for identifier (address, type and ID)
  • : for value
  • # for checksum
• base-16 (hex) used for numeric values
• addressing made programmable; 32 to 125 (0x20 to 0x7D)
  • with 126 (0x7E) used for broadcast
• digital IOs addressed as byte, not individually
• module status and status bit was introduced
• baud rate bumped to 38400

20,o,0:4#84

The command above tells module 0x20 (32) to set its digital output byte to 4, which means turn on output 3 (you know; 1 2 4 8…)

Module address

There really was no need for a module address; as RS-232 can only be between two devices. But with addressing it was possible to have multiple modules connected to a single host, without the client knowing what serial port they were connected to.

My plan was to switch to RS-485, which can be multipoint and would require addressing. I only made a single RS-485 module, before switching to Arduino, Raspberry Pi and ESP8266.

Request types

• Output: o
• Input: i
• Status: s
• Unit setup: u

Request ID

For revision 1 the base-10 request ID was simply the digital IO pin. But for revision 2 this was changed to byte for digital IOs:

• 0: Digital input or output byte
• 1-15 (0x1-0xF): Analog input or PWM output

Module status

Module status and status bit was introduced with revision 2, and allowed the module to inform the host of its capabilities.

Status bits

If any module status bit was set, meaning the value was above 0, the status LED on the module was red instead of green.

• 0: BootFlag
  • set to true on startup, could be manually reset
• 2: DefAddressFlag
  • set to true on startup if module was using the default address, could be manually reset
• 4: ManFailFlag
  • default false, manually toggled

Module setup

Also introduced with revision 2. A software option to reboot the module was useful when programming over the serial port. The modules would first be rebooted, then put in programming mode.

1. reboot required if changed

Modules

In total I made six modules. Two IO modules were rebuilt to be stand-alone sensors.

• Online monitoring unit (OMU)
  • Later rebuilt to Online monitoring unit 2 (OMU2)
    • Later rebuilt to Temperature and humidity sensor
• Online serial interface device (OSID)
  • Later rebuilt to Temperature and light sensor with signalling LEDs
• Online environmental and signalling unit (OESU)
• Digital emergency interface module (DEIM)
• Advanced serial interface module (ASIM)
• Online monitoring unit 3 (OMU3)
  • aka. Temperature and auxiliary sensor

Serial ports

Initially I used a serial expansion card from Comtrol with an external interface. One PCI slot gave me 8 serial ports. I made adapters to turn the DB-25 female into RS-232 DB-9 male.

This worked fine as long as all the modules were in the same place.

When I got a house and started spreading the modules out, I instead used a couple of serial servers and some USB to serial adapters.

The serial servers made a single serial port available on the network through a TCP port. They were quite expensive, and made the whole setup bulky.

Firmware

I wrote all my AVR firmware in BASCOM-AVR. I programmed all modules with a bootloader, which allowed me to update the firmware using the serial port. So much easier than opening the modules and taking the microcontroller out.

The firmware for the different modules is available in git repositories on the respective project pages.

Software

Python

This is a simple, proof-of-concept Python script to send and receive data from a serial module. It sends commands to the broadcast address (126), splits, decodes and verifies the received data.

import serial #pyserial
import re, sys
import functools, operator

def hex2dec(s):
  try:
    return int(s, 16)
  except (ValueError, TypeError) as ex:
    print('Type/Value: %s', str(ex))

ser = serial.Serial('/dev/ttyUSB0', 38400, timeout=1)

adr = 126 # broadcast
s = str(sys.argv[1])

send_str = chr(adr)+','+s+'\r\n'

ser.write(bytes(send_str, 'utf-8'))
response = ser.readline()
line = response.decode("utf-8")

if not line:
    raise ValueError("Empty response")

print(line.rstrip())
pos_adr = line.find(':')
pos_crc = line.find('#')
str_adr = line[pos_adr-6:pos_adr]
str_val = line[pos_adr+1:pos_crc]
str_crc = line[pos_crc+1:len(line)-1] #-1

checksum = functools.reduce(operator.add, map(ord, str_adr + ":" + str_val))
while checksum > 255:
  checksum -= 256

if int(str_crc) == checksum:
    print("Checksum OK")

if re.match('u,1',s):
  print(hex2dec(str_val), str_val)
elif re.match('s,[1-6]',s):
  print(str_val)
elif re.match('s,[7-9]',s):
  print(hex2dec(str_val[0:2]), hex2dec(str_val[2:4]))
else:
  print(hex2dec(str_val))

In the example below, we send the command o,0,F to the module. Serial port and address is set in the Python script.

$ python3 sios.py o,0,F
20,o,0:000F#105
Checksum OK
15

We receive 20,o,0:000F#105, which means:

• from module 0x20 (32)
• type is o (output)
• ID is 0x0 (digital output byte)
• value is 0x000F
• checksum is 105

The checksum is verified to match the received string, and the value converted to base-10 is 15. We have turned on digital output 1, 2, 3 and 4.

Demonstration

What happens here:

• Resetting status bit 1 and 2
  • Status LED turns green when all status bits are off
• Turn on digital outputs 1 to 4
• Turn off digital outputs
• Set PWM output 1 value to 16, 80, 255 and 0
• Read digital inputs, while activating toggle switch
• Read analog inputs 1 and 2
• Toggle status bit 3
• Read status data values 1 to 9

Serial server

Serial server was written in Visual Basic, not the best language, but it was what I knew at the time. It handled communication with all the serial modules.

• acted as a gateway between the serial modules and a connected serial client
• forwarded single TCP commands to the correct serial module
• converted received analog sensor values to physical units
• handled timers and triggers
• received a heartbeat signal from the serial modules every 20 seconds
• cached the last received states for all IOs for all modules, this was forwarded to the serial client on login

Configuration screenshots

Bad programming

The application worked, but was badly written. All code related to the serial ports was duplicated for each port, making it a lot of work to add support for additional serial ports.

When there was a lot of traffic to multiple modules; the serial server had problems keeping up, and occasionally lost some strings.

It started small, and grow out of proportions. It remained the weakest link in the system for its entire lifetime.

TCP commands

Serial server could receive commands on a TCP socket, the socket was automatically terminated after the command was forwarded.

I made a Windows application called SerialTCPcommand.exe:

 SerialTCPcommand.exe password;002:o:08:1

In Linux netcat could be used:

 echo "password;002:o:08:1" | netcat ip-address port

Module config files

Each serial module had a config file, e.g. 001.ini, which informed the serial server of the capabilities of the module:

• Baud rate, data bit, start-char
• Module name
• Number of inputs
• Number of outputs
• Can receive requests (0/1)

Serial server logs

Serial server wrote three kinds of logs:

Historical

Every action stored in one file. Periodically parsed to a MySQL database, and used for historical data.

Value

The converted value of all analog sensors was written in individual files. Used for sharing data with other applications, like mrtg and Serial Client 4.x.

State

Current state of all serial modules and IO ports was written in individual files. Used for sharing data with Serial Client 4.x.

Serial client

The client communicated with the serial modules through the server — like a user interface. It had very limited logic, and relied on the serial server for converted analog sensor values, triggers, etc.

The server and client shared data using a TCP connection and log files.

Version 1

Version 1 of the client was written in Visual Basic, and was terrible. It had a number of major bugs and was unable to handle any latency in the connection to the server, causing it to constantly loose information.

Version 2

Version 2 was written in Flash, which was really hot at the time, and could be run in a browser — which was a major advantage. Flash also made it possible to make a decent UI, and handled latency in the server connection well.

The CCTV photo was captured by a TV input card, which saved it to the web server every 3 seconds.

I even think I experimented with animations and text-to-voice 😛

Version 3

Version 3 was written in ActionScript (still Flash). Basically version 2 with an updated UI and added controls.

Version 4

Version 4 was written in PHP and HTML, with CSS styling. Native web, and leaving that Flash crap behind. Unlike the previous versions; it didn’t need a TCP connection to the serial server, only HTTP(s) to the web server. This meant that it also worked behind corporate firewalls 😃

At this point the limitations of the serial server was really starting to show… It saved state and value files on a Samba share, that the PHP client read. A shell_exec was used to send commands to the serial modules:

shell_exec("echo $command | netcat serialserver_ip port");

Dirty 👎

I was starting to use a MySQL database more and more; looking up the sensor names, historical data, etc. And shortly after I ditched the serial server and started using Python instead.

I also made a simple, frameless, version of this client — for WAP. Remember WAP? Wireless Application Protocol. Early day mobile phone web browsing 😄