SIOS — the serial protocol syntax I made
When I first started making AVR modules, there was no Raspberry Pi or ESP8266. Arduino was introduced in 2005, the same year I started with the AVR microcontroller. But I didn’t know of it until many years later.
There weren’t any Wi-Fi enabled microcontrollers at that time. Both Ethernet and Bluetooth interfaces were more expensive, and harder to interface. So I ended up using the good old serial port.
With RS-232 I got a way of getting data between the microcontrollers and the computer, but I still needed a defined syntax. So I set out to make one, and called it SIOS — Serial Input/Output System.
Table of contents
My main project at the time, The Rack box, was getting big. Everything was controlled by microcontroller modules, acting on input signals. I wanted to bring all I/Os into the computer, which would allow me to create more advanced automations — quicker and easier.
The status panel for the rack box, consisting of LEDs and switches, was also getting full. Any changes or additions were time consuming, and difficult. Instead I wanted a software status panel that I could easily update and expand 😃
I made some big changes to the syntax in 2013 — this became the 2nd revision. With rev. 2 the formatting and encoding was changes, but both revisions had the same basic parts:
- Module address
- Request type
- Request ID
- Request value
:was used as a delimiter
- numeric values in base-10
- addressing started on 001 (hard coded)
- baud rate was 9600
The command above tells module
001 to set its output
3 to true
- multiple delimiters used, to make it easier to identify different segments of the command
,for identifier (address, type and ID)
- base-16 (hex) used for numeric values
- addressing made programmable; 32 to 125 (
- with 126 (
0x7E) used for broadcast
- with 126 (
- digital IOs addressed as byte, not individually
- module status and status bit was introduced
- baud rate bumped to 38400
The command above tells module
0x20 (32) to set its digital output byte to
4, which means turn on output 3 (you know; 1 2 4 8…)
There really was no need for a module address; as RS-232 can only be between two devices. But with addressing it was possible to have multiple modules connected to a single host, without the client knowing what serial port they were connected to.
My plan was to switch to RS-485, which can be multipoint and would require addressing. I only made a single RS-485 module, before switching to Arduino, Raspberry Pi and ESP8266.
- Unit setup:
For revision 1 the base-10 request ID was simply the digital IO pin. But for revision 2 this was changed to byte for digital IOs:
0: Digital input or output byte
1-15 (0x1-0xF): Analog input or PWM output
Module status and status bit was introduced with revision 2, and allowed the module to inform the host of its capabilities.
|#Adr||Status bit||Status Info||Type|
|7||DIO||Digital in, out||#Double|
|8||AI||Analog in, bit||#Double|
|9||AO||Analog out, bit||#Double|
If any module status bit was set, meaning the value was above 0, the status LED on the module was red instead of green.
- set to true on startup, could be manually reset
- set to true on startup if module was using the default address, could be manually reset
- default false, manually toggled
Also introduced with revision 2. A software option to reboot the module was useful when programming over the serial port. The modules would first be rebooted, then put in programming mode.
|1||ADR||Module address ¹||Byte|
- reboot required if changed
In total I made six modules. Two IO modules were rebuilt to be stand-alone sensors.
- Online monitoring unit (OMU)
- Online serial interface device (OSID)
- Later rebuilt to Temperature and light sensor with signalling LEDs
- Online environmental and signalling unit (OESU)
- Digital emergency interface module (DEIM)
- Advanced serial interface module (ASIM)
- Online monitoring unit 3 (OMU3)
Initially I used a serial expansion card from Comtrol with an external interface. One PCI slot gave me 8 serial ports. I made adapters to turn the DB-25 female into RS-232 DB-9 male.
This worked fine as long as all the modules were in the same place.
When I got a house and started spreading the modules out, I instead used a couple of serial servers and some USB to serial adapters.
The serial servers made a single serial port available on the network through a TCP port. They were quite expensive, and made the whole setup bulky.
I wrote all my AVR firmware in BASCOM-AVR. I programmed all modules with a bootloader, which allowed me to update the firmware using the serial port. So much easier than opening the modules and taking the microcontroller out.
The firmware for the different modules is available in git repositories on the respective project pages.
This is a simple, proof-of-concept Python script to send and receive data from a serial module. It sends commands to the broadcast address (126), splits, decodes and verifies the received data.
import serial #pyserial import re, sys import functools, operator def hex2dec(s): try: return int(s, 16) except (ValueError, TypeError) as ex: print('Type/Value: %s', str(ex)) ser = serial.Serial('/dev/ttyUSB0', 38400, timeout=1) adr = 126 # broadcast s = str(sys.argv) send_str = chr(adr)+','+s+'\r\n' ser.write(bytes(send_str, 'utf-8')) response = ser.readline() line = response.decode("utf-8") if not line: raise ValueError("Empty response") print(line.rstrip()) pos_adr = line.find(':') pos_crc = line.find('#') str_adr = line[pos_adr-6:pos_adr] str_val = line[pos_adr+1:pos_crc] str_crc = line[pos_crc+1:len(line)-1] #-1 checksum = functools.reduce(operator.add, map(ord, str_adr + ":" + str_val)) while checksum > 255: checksum -= 256 if int(str_crc) == checksum: print("Checksum OK") if re.match('u,1',s): print(hex2dec(str_val), str_val) elif re.match('s,[1-6]',s): print(str_val) elif re.match('s,[7-9]',s): print(hex2dec(str_val[0:2]), hex2dec(str_val[2:4])) else: print(hex2dec(str_val))
In the example below, we send the command
o,0,F to the module. Serial port and address is set in the Python script.
$ python3 sios.py o,0,F 20,o,0:000F#105 Checksum OK 15
20,o,0:000F#105, which means:
- from module 0x20 (32)
- type is
- ID is
0x0(digital output byte)
- value is
- checksum is 105
The checksum is verified to match the received string, and the value converted to base-10 is
15. We have turned on digital output 1, 2, 3 and 4.
What happens here:
- Resetting status bit 1 and 2
- Status LED turns green when all status bits are off
- Turn on digital outputs 1 to 4
- Turn off digital outputs
- Set PWM output 1 value to 16, 80, 255 and 0
- Read digital inputs, while activating toggle switch
- Read analog inputs 1 and 2
- Toggle status bit 3
- Read status data values 1 to 9
Serial server was written in Visual Basic, not the best language, but it was what I knew at the time. It handled communication with all the serial modules.
- acted as a gateway between the serial modules and a connected serial client
- forwarded single TCP commands to the correct serial module
- converted received analog sensor values to physical units
- handled timers and triggers
- received a heartbeat signal from the serial modules every 20 seconds
- cached the last received states for all IOs for all modules, this was forwarded to the serial client on login
The application worked, but was badly written. All code related to the serial ports was duplicated for each port, making it a lot of work to add support for additional serial ports.
When there was a lot of traffic to multiple modules; the serial server had problems keeping up, and occasionally lost some strings.
It started small, and grow out of proportions. It remained the weakest link in the system for its entire lifetime.
Serial server could receive commands on a TCP socket, the socket was automatically terminated after the command was forwarded.
I made a Windows application called SerialTCPcommand.exe:
netcat could be used:
echo "password;002:o:08:1" | netcat ip-address port
Each serial module had a config file, e.g.
001.ini, which informed the serial server of the capabilities of the module:
- Baud rate, data bit, start-char
- Module name
- Number of inputs
- Number of outputs
- Can receive requests (0/1)
Serial server wrote three kinds of logs:
- Every action stored in one file. Periodically parsed to a MySQL database, and used for historical data.
- The converted value of all analog sensors was written in individual files. Used for sharing data with other applications, like mrtg and Serial Client 4.x.
- Current state of all serial modules and IO ports was written in individual files. Used for sharing data with Serial Client 4.x.
The client communicated with the serial modules through the server — like a user interface. It had very limited logic, and relied on the serial server for converted analog sensor values, triggers, etc.
The server and client shared data using a TCP connection and log files.
Version 1 of the client was written in Visual Basic, and was terrible. It had a number of major bugs and was unable to handle any latency in the connection to the server, causing it to constantly loose information.
Version 2 was written in Flash, which was really hot at the time, and could be run in a browser — which was a major advantage. Flash also made it possible to make a decent UI, and handled latency in the server connection well.
The CCTV photo was captured by a TV input card, which saved it to the web server every 3 seconds.
I even think I experimented with animations and text-to-voice 😛
Version 3 was written in ActionScript (still Flash). Basically version 2 with an updated UI and added controls.
Version 4 was written in PHP and HTML, with CSS styling. Native web, and leaving that Flash crap behind. Unlike the previous versions; it didn’t need a TCP connection to the serial server, only HTTP(s) to the web server. This meant that it also worked behind corporate firewalls 😃
At this point the limitations of the serial server was really starting to show… It saved state and value files on a Samba share, that the PHP client read. A
shell_exec was used to send commands to the serial modules:
shell_exec("echo $command | netcat serialserver_ip port");
I was starting to use a MySQL database more and more; looking up the sensor names, historical data, etc. And shortly after I ditched the serial server and started using Python instead.
I also made a simple, frameless, version of this client — for WAP. Remember WAP? Wireless Application Protocol. Early day mobile phone web browsing 😄