Communication Protocol

Packet Types: Requests and Answers

Devices communicate by sending data packets to each other. There are several types of such packets.

Requests are used to get the current status from another device ("Get") or to change it ("Set"), with an optional response of the updated status ("SetGet"). A "Deliver" message is similar to a "Set", but the data which is sent within is the current status of the sender, and not a requested modification of the receiver status. A "Deliver" request is usually sent from a client to the base station. Every request is queued in the request queue and repeated when no answer is received in time.

The receiving device answers by sending an acknowledge ("Ack") with optional updated status ("AckStatus"). As answer to a "Deliver" request, the answer is always an "Ack", because the status in the message was from the requester, and can't be changed by the receiver. A device may also broadcast its current status without being asked for it ("Status").

Basic Packet Layout

The packet layout is defined to support the before mentioned types. A packet is a complete sequence of bytes that is transmitted in one step. It consists of a common header, a header extension and (in some cases) the message data itself. All SHC devices know the header and header extension format to support the communication protocol and react accordingly. In contrast, the message data will only be understood by devices that support the specific messages. This also means that additional messages can and will be defined later on for new SHC devices and all older devices in use can work correctly without change (even if they won't understand the message data). For explanation it is assumed that the packet is not encrypted until otherwise mentioned.

The basic layout of SHC packets

Header

The header contains information necessary for every packet:

• The CRC32 value is the checksum of all bytes after the checksum itself until the end of the packet, which is at a 16 Bit boundary because of the AES encryption. This means if you have a packet using 17 bytes, you need two AES blocks (32 bytes). The CRC32 is then the checksum of byte 4 to 31.
• The SenderID is the DeviceID of the device sending the packet.
• PacketCounter: To ensure that a possible attacker cannot control a device by simply repeating an encrypted packet, every packet is valid only once. The internal packet counter is used per device and incremented for every packet sent. For the packet counter, a 24 bit integer value is used, allowing 16.777.216 packets. This is enough to cover 32 years of operation if a device sends one packet every minute.
• The MessageType defines what the rest of the packet means and how to decode it.

Header Extension

The header extension contains additional parameters depending on the MessageType in the header:

• The ReceiverID is the DeviceID of the device that should react on a request ("Get", "Set" or "GetSet").
• MessageGroupID and MessageID tell which data property of a device should be changed or returned (requests) or which data is returned (answers "AckStatus" or "Status"). The same definition may be used for different MessageTypes (e.g. "SetGet" and "AckStatus"), and a device usually answers using the same MessageID as in the request. But this is not mandatory. Please see the specification of an SHC device for the detailed sequences.
• AckSenderID and AckPacketCounter are the reference to the request that is acknowledged ("Ack", "AckStatus").
• The Error field tells if a request was successful (0) or an error occurred (1). This is an extra field to allow easy detection of errors without the need to parse and interpret the message data for that.

Message Data

The message data depends on the MessageGroup and MessageID in the header extension.

• It contains the message specific data.
• The format is defined in the packet_layout.xml file. Look into the generated Message Catalog for details.

Device Interfaces, defined by MessageGroupID and MessageID

The complete set of messages is sorted by message groups. The messages (MessageIDs) of one message group (MessageGroupID) define the interface of a device type and describe how it can be controlled. Think of a dimmer for example. This type of device may support a request to set the brightness, another request for configuration and a third one for a brightness animation. These three messages would be the dimmer interface and could be defined in a message group "dimmer". Of course, a specific dimmer implementation may support only a part of the messages in the message group.

The use of MessageGroupIDs and MessageIDs for identification of messages is done to strengthen thinking about device types and their interfaces rather than thinking about single messages. If new devices are developed and messages are defined, think about the whole interface and decide which one to change wisely.

One SHC device (e.g. a weather station) may support more than one of these interfaces, thus supporting messages with different MessageGroupIDs. This is especially true for the "Generic" message group, which is designed to be used by many devices.

Packet Format Definition and Generation

The packet format, specifically the message groups and messages, are defined in the packet_layout.xml file. The packet layout metamodel (packet_metamodel.xsd) describes the possible entries. In principle, the metamodel should not change over time, whereas the packet layout changes every time when new device types are invented or messages are added or redefined. A change in the metamodel also makes it necessary to change tools, a change in the layout should not.

Packet Generation Concept

As you can see in the picture, the layout is used for source code generation as well as generation of the Message Catalog. The source code generator is included in the E2P editor (Java program) and can create C header files. When using FHEM, the SHC parser is used to decode the data strings received by the base station and to create strings to send messages.

Packet Layout Metamodel

The metamodel contains definitions according to the format already described in the previous chapters. The block "DataValue" represents a series of values with one of the supported data types.

Data Types

Several primitive data types are defined and can be used for the e2p layout definition as well as the packet layout. The UIntValue is an unsigned integer within a given range using a defined amount of bits. The IntValue is a signed integer. The BoolValue can be true or false and is one bit in size in packets. EnumValue is used for settings represended as integer values in the firmware, but have an additional name describing the meaning of the value. The ByteArray consists of a specific amount of bytes. The Reserved element is used to block a number of bits which are not used currently.

Some types have an optional default value. This is used especially in the e2p editor to have a working preset for new devices.

To store a series of values, it is possible to define an Array or one of the other data types. E.g. values SwitchState[i] with i from 0 to 7 could be defined and used instead of 8 values SwitchState0, SwitchState1 and so on. This allows to access the elements in a loop programatically.

There's also the possibility to have structured arrays (which are similar to arrays of struct elements in C). The difference between a structured array with sub-elements A[i] and B[i] and two arrays with values A[i] and B[i] is the order of the data. In the structured array, the order is A[0], B[0], A[1], B[1],..., whereas two normal arrays would result in ordering the data elements like A[0], A[1],... B[0], B[1],... The difference may not seem important at first glance, but the structured array makes it possible to truncate a message when only a part of array elements are needed, and that's exactly where they are used for.

The byte and bit order for all values is big-endian, so the most significant byte / bit comes first. This is primarily to simplify the understanding of values which don't start or end at byte borders.

Firmware Packet API

Header files with access functions are generated from the packet layout file and stored under the src_common directory of the firmware source code. If you write or change a device firmware, use solely these functions to create and interpret the packet data. Read more about it in the Packet API How-to.

Communication Principles

Basic communication principles are described in the following paragraph. Please consider them when defining new interfaces and sequences.

One Message per Packet

It would be possible to send several messages in one packet. But this would also increase the packet size additionally, because each message part would have its own MessageGroupID and MessageID. An additional length field would be necessary because a receiver (e.g. base station) of a packet could not know a new MessageID (and thus the length of the message data). Therefore, each packet only contains one message. The length of the message is then implicitly defined as the amount of bytes between the start of message data and the end of the packet.

Truncated Messages

To save space, a sender is allowed to truncate a message at a point from which on only bytes with value 0 follow. A receiver has to assume that all missing bytes which are defined for the MessageID are zero.

Use SetGet if Status needed

An acknowledge ("Ack") is used to tell the sender of a request that the request was received and processed. It does not include the updated status itself.

If a requestor (e.g. BaseStation) also wants to know the current status of the receiver, it should use a "SetGet" rather than using a "Set" and an additional "Get" to ask for the updated status. This way traffic can be reduced. On the other hand, the receiver should assume that the requestor does not need the updated status when he only sends a "Set" and therefore should not send an updated "Status" immediately after the "Ack".

Immediate Answers (no Pending Tasks)

For simplicity, it is assumed that every request can be processed in "no time" and the result is sent back as acknowledge immediately. There may be a delay of some hundred milliseconds, but if a device takes longer to change a (physical) state, the result is sent immediately, too (think of a garage door that takes several seconds to open). To reflect the current status correctly despite the immediate acknowledge, the messages have to be defined appropriately. For example, the garage door device could have a status "door is opening" and send another status "door is opened" several seconds later when the door has finished opening.

This concept is chosen to keep the protocol and implementation in the SHC devices simple. The expected behaviour at the requestor is also use case specific. The user should have the possibility to configure it easily without changing the devices firmware or protocol. Therefore, the concept of pending tasks (with retries, aborts etc.) has to be defined in the server software if needed.

Error Handling

An SHC device should not have a generic error state it gets into when "something" bad happens. Also no device type independent generic error state is defined. Instead, all possible states a device can get into should be foreseen as specific and intended states which tell clearly what the error condition means.

In contrast, the possibility to acknowledge a request with an error is provided universally. It shows that the request can't be handled and the receiver did not do what the requestor asked for. This error flag in the acknowledge is meant to detect and report errors easily. However, it's up to the receiver to react accordingly and retry the same or another request later if needed. An "Ack" with active Error flag should also be sent when a MessageID is not supported by a device. Otherwise, the requestor would produce more traffic by repeating his request.

Broadcast Requests

A broadcast request is a request with receiver ID 4095 (which is the maximum ID). It is meant to be processed by any receiver that can process the request. In contrast to a normal request, the sender does not wait for a specific acknowledge and the request queue is not used.

It should be considered carefully which request for which device type is useful as a broadcast request. A device could reject reacting on a broadcast and only react on a request specifically for its device ID.

If a broadcast request changes the state of a device, the device should answer with a Status or AckStatus, but use a delay to reduce the possibility that status updates from different devices are sent at the same time. Be aware that these status updates are not guaranteed to be received by the requestor, because the retry mechanism (with acknowledges) is not used.

Answer with different MessageID

In most cases, a "Set" or "SetGet" changes a property of a device which can be returned as a "Status" or "AckStatus" in the same message format (MessageID). This means you change a value in the device and it acts accordingly. Sometimes though, an answer containing the same type of data (same MessageID) is not reasonable. Here are some examples:

• Request an action: Example: Request a water pump so pump 100ml of water. (It would not make sense to answer that 100ml are being pumped, because the device would have been started already and this is no longer lasting status to work with.)
• Change only a part of internal data structures: Example: Request an alarm clock that supports 10 alarms to add an alarm at 7.00 o' clock. (Returning this one alarm in a Status would not be useful, but returning all alarms that are programmed might be.)
• Relative change: Example: Ask a dimmer to increase the brightness by 10%. (The relative value would not be useful for any other device, but the resulting absolute brightness might be.)

In other protocols, there might be separate message types for such requests. This can lead to an inconsistent use of message types because there are corner cases where it is not easy to decide what to use. In smarthomatic, "Set" messages are also used for such requests. In this case though, the device can usually only answer with a "AckStatus" or "Status" with another MessageID (and therefore with a message containing a different form of data). The MessageID for the request is then typically restricted to the message types "Set", "SetGet" and "Ack" (the device reports that the command was understood / executed). The MessageID used for the answer is typically defined for the MessageTypes "Get", "Status" and "AckStatus" only.

To give you an example, the alarm clock from above may receive an AlarmClock_Alarm_SetGet("7.00") request and answers with an "AckStatus" of a different MessageID AlarmClock_AllAlarms_AckStatus, which contains a set of up to 10 alarms which are currently programmed.

Typical Communication Sequences

(Cyclic) Status

A device sends a status without being asked to. This is typical for simple sensors that broadcast their measured values at a fixed time cycle. Especially when such a device is not programmed to receive packets, it's the only way it communicates. A transmission error can't be detected and the according packet is therefore not repeated.

Request (with automatic retries)

A device requests to change a setting / value / state "x" by sending a "Set" packet. The packet is acknowledged by the receiver. The return value is "ok". The sender can assume that the state change was done as requested. The second time the packet is lost and automatically repeated using the request queue.

Second Request after improper Answer

A device requests to change a setting / value / state "x" by sending a "SetGet" packet. The receiver does not change the state accordingly and answers with another state and active "Error" bit in the AckStatus packet. Depending on the current state "2", the sender now requests state "3" and gets a positive answer.

Request depending on another value

A device first requests a setting / value / state "x" by sending a "Get" packet. Depending on the current state "4", the sender now requests state "7" for another setting / value / state "y".

Request a change that takes longer

Assume a garage door drive which is requested to open the (previously closed) door. After the request is sent, the engine to open the door is powered and this is acknowledged with state "opening" immediately. Opening the door takes some time. Because the sender is impatient, he requests the current state. But the door is not finished opening. After the door is finally open, the status is sent by the garage door device.

This approach means the requestor has to poll the receiver sooner or later if he's interested if the long lasting action is finished. If he doesn't and no status update is received, he can't tell if the action is not finished yet or the status update was lost because of a transmission error.

Broadcast Request

A device requests to change a setting / value / state "x" by sending a "SetGet" broadcast request packet with ReceiverID 4095. The receivers 1 and 2 react by sending an AckStatus packet after a different timeout they choose. The acknowledge of receiver 2 shows an error.