Paul Tagliamonte: designing arf, an sdr iq encoding format đ¶
Interested in future updates? Follow me on mastodon at @paul@soylent.green. Posts about `hz.tools` will be tagged #hztools.
đ¶ Want to jump right to the draft? I'll be maintaining ARF going forward at /draft-tagliamonte-arf-00.txt.
Itâs true â processing data from software defined radios can be a bit complex đđđ â which tends to keep all but the most grizzled experts and bravest souls from playing with it. While I wouldnât describe myself as either, I will say that Iâve stuck with it for longer than most would have expected of me. One of the biggest takeaways I have from my adventures with software defined radio is that thereâs a lot of cool crossover opportunity between RF and nearly every other field of engineering.
Fairly early on, I decided on a very light metadata scheme to track SDR captures, called rfcap. rfcap has withstood my test of time, and I can go back to even my earliest captures and still make sense of what they are â IQ format, capture frequencies, sample rates, etc. A huge part of this was the simplicity of the scheme (fixed-lengh header, byte-aligned to supported capture formats), which made it roughly as easy to work with as a raw file of IQ samples.
However, rfcap has a number of downsides. Itâs only a single, fixed-length header. If the frequency of operation changed during the capture, that change is not represented in the capture information. Itâs not possible to easily represent mulit-channel coherent IQ streams, and additional metadata is condemned to adjacent text files.
# ARF (Archive of RF)
A few years ago, I needed to finally solve some of these shortcomings and tried to see if a new format would stick. I sat down and wrote out my design goals before I started figuring out what it looked like.
First, whatever I come up with must be capable of being streamed and processed while being streamed. This includes streaming across the network or merely written to disk as itâs being created. No post-processing required. This is mostly an artifact of how Iâve built all my tools and how I intereact with my SDRs. I use them extensively over the network (both locally, as well as remotely by friends across my wider lan). This decision sometimes even prompts me to do some crazy things from time to time.
I need actual, real support for multiple IQ channels from my multi-channel SDRs (Ettus, Kerberos/Kracken SDR, etc) for playing with things like beamforming. My new format must be capable of storing multiple streams in a single capture file, rather than a pile of files in a directory (and hope theyâre aligned).
Finally, metadata must be capable of being stored in-band. The initial set of metadata I needed to formalize in-stream were `Frequency Changes` and `Discontinuities`. Since then, ARF has grown a few more.
After getting all that down, I opted to start at what I thought the simplest container would look like, TLV (tag-length-value) encoded packets. This is a fairly well trodden path, and used by a bunch of existing protocols we all know and love. Each ARF file (or stream) was a set of encoded âpacketsâ (sometimes called data units in other specs). This means that unknown packet types may be skipped (since the length is included) and additional data can be added after the existing fields without breaking existing decoders.
tag
flags
length
value
**Heads up!** Once this is posted, I'm not super likely to update this page. Once this goes out, the latest stable copy of the ARF spec is maintained at draft-tagliamonte-arf-00.txt. This page may quickly become out of date, so if you're actually interested in implementing this, I've put a lot of effort into making the draft comprehensive, and I plan to maintain it as I edit the format.
Unlike a âtraditionalâ TLV structure, I opted to add âflagsâ to the top-level packet. This gives me a bit of wiggle room down the line, and gives me a feature that I like from ASN.1 â a âcriticalâ bit. The critical bit indicates that the packet must be understood fully by implementers, which allows future backward incompatible changes by marking a new packet type as critical. This would only really be done if something meaningfully changed the interpretation of the backwards compatible data to follow.
Flag | Description
---|---
0x01| Critical (tag must be understood)
Within each Packet is a `tag` field. This tag indicates how the contents of the `value` field should be interpreted.
Tag ID | Description
---|---
0x01| Header
0x02| Stream Header
0x03| Samples
0x04| Frequency Change
0x05| Timing
0x06| Discontinuity
0x07| Location
0xFE| Vendor Extension
In order to help with checking the basic parsing and encoding of this format, the following is an example packet which should parse without error.
00, // tag (0; no subpacket is 0 yet)
00, // flags (0; no flags)
00, 00 // length (0; no data)
// data would go here, but there is none
Additionally, throughout the rest of the subpackets, there are a few unique and shared datatypes. I document them all more clearly in the draft, but to quickly run through them here too:
### UUID
This field represents a globally unique idenfifer, as defined by RFC 9562, as 16 raw bytes.
### Frequency
Data encoded in a Frequency field is stored as microhz (1 Hz is stored as 1000000, 2 Hz is stored as 2000000) as an unsigned 64 bit integer. This has a minimum value of 0 Hz, and a maximum value of 18446744073709551615 uHz, or just above 18.4 THz. This is a bit of a tradeoff, but itâs a set of issues that I would gladly contend with rather than deal with the related issues with storing frequency data as a floating point value downstream. Not a huge factor, but as an aside, this is also how my current generation SDR processing code (`sparky`) stores Frequency data internally, which makes conversion between the two natural.
### IQ samples
ARF supports IQ samples in a number of different formats. Part of the idea here is I want it to be easy for capturing programs to encode ARF for a specific radio without mandating a single iq format representation. For IQ types with a scalar value which takes more than a single byte, this is always paired with a Byte Order field, to indicate if the IQ scalar values are little or big endian.
ID | Name | Description
---|---|---
0x01| f32| interleaved 32 bit floating point scalar values
0x02| i8 | interleaved 8 bit signed integer scalar values
0x03| i16| interleaved 16 bit signed integer scalar values
0x04| u8 | interleaved 8 bit unsigned integer scalar values
0x05| f64| interleaved 64 bit floating point scalar values
0x06| f16| interleaved 16 bit floating point scalar values
## Header
Each ARF file must start with a specific Header packet. The header contains information about the ARF stream writ large to follow. Header packets are always marked as âcriticalâ.
magic
flags
start
guid
site guid
#st
In order to help with checking the basic parsing and encoding of this format, the following is an example header subpacket (when encoded or decoded this will be found inside an ARF packet as described above) which should parse without error, with known values.
00, 00, 00, fa, de, dc, ab, 1e, // magic
00, 00, 00, 00, 00, 00, 00, 00, // flags
18, 27, a6, c0, b5, 3b, 06, 07, // start time (1740543127)
// guid (fb47f2f0-957f-4545-94b3-75bc4018dd4b)
fb, 47, f2, f0, 95, 7f, 45, 45,
94, b3, 75, bc, 40, 18, dd, 4b,
// site_id (ba07c5ce-352b-4b20-a8ac-782628e805ca)
ba, 07, c5, ce, 35, 2b, 4b, 20,
a8, ac, 78, 26, 28, e8, 05, ca
## Stream Header
Immediately after the arf Header, some number of Stream Headers follow. There must be exactly the same number of Stream Header packets as are indicated by the `num streams` field of the Header. This has the nice effect of enabling clients to read all the stream headers without requiring buffering of âunreadâ packets from the stream.
id
flags
fmt
bo
rate
freq
guid
site
In order to help with checking the basic parsing and encoding of this format, the following is an example stream header subpacket (when encoded or decoded this will be found inside an ARF packet as described above) which should parse without error, with known values.
00, 01, // id (1)
00, 00, 00, 00, 00, 00, 00, 00, // flags
01, // format (float32)
01, // byte order (Little Endian)
00, 00, 01, d1, a9, 4a, 20, 00, // rate (2 MHz)
00, 00, 5a, f3, 10, 7a, 40, 00, // frequency (100 MHz)
// guid (7b98019d-694e-417a-8f18-167e2052be4d)
7b, 98, 01, 9d, 69, 4e, 41, 7a,
8f, 18, 16, 7e, 20, 52, be, 4d,
// site_id (98c98dc7-c3c6-47fe-bc05-05fb37b2e0db)
98, c9, 8d, c7, c3, c6, 47, fe,
bc, 05, 05, fb, 37, b2, e0, db,
## Samples
Block of IQ samples in the format indicated by this streamâs `format` and `byte_order` field sent in the related Stream Header.
id
iq samples
In order to help with checking the basic parsing and encoding of this format, the following is an samples subpacket (when encoded or decoded this will be found inside an ARF packet as described above). The IQ values here are notional (and are either 2 8 bit samples, or 1 16 bit sample, depending on what the related Stream Header was).
01, // id
ab, cd, ab, cd, // iq samples
## Frequency Change
The center frequency of the IQ stream has changed since the Stream Header or last Frequency Change has been sent. This is useful to capture IQ streams that are jumping around in frequency during the duration of the capture, rather than starting and stopping them.
id
frequency
In order to help with checking the basic parsing and encoding of this format, the following is a frequency change subpacket (when encoded or decoded this will be found inside an ARF packet as described above).
01, // id
00, 00, b5, e6, 20, f4, 80, 00 // frequency (200 MHz)
## Discontinuity
Since the last Samples packet for this stream, samples have been dropped or not encoded to this stream. This can be used for a stream that has dropped samples for some reason, a large gap (radio was needed for something else), or communicating âiq snippitsâ.
id
In order to help with checking the basic parsing and encoding of this format, the following is a discontinuity subpacket (when encoded or decoded this will be found inside an ARF packet as described above).
01, // id
## Location
Up-to-date location as of this moment of the IQ stream, usually from a GPS. This allows for in-band geospatial information to be marked in the IQ stream. This can be used for all sorts of things (detected IQ packet snippits aligned with a time and location or a survey of rf noise in an area)
flags
sys
lat
long
el
accuracy
The `sys` field indicates the Geodetic system to be used for the provided `latitude`, `longitude` and `elevation` fields. The full list of supported geodetic systems is currently just WGS84, but in case something meaningfully changes in the future, itâd be nice to migrate forward.
Unfortunately, being a bit of a coward here, the accuracy field is a bit of a cop-out. Iâd really rather it be what we see out of kinematic state estimation tools like a kalman filter, or at minimum, some sort of ellipsoid. This is neither of those - itâs a perfect sphere of error where we pick the largest error in any direction and use that. Truthfully, I canât be bothered to model this accurately, and I donât want to contort myself into half-assing something I know I will half-ass just because I know better.
System | Description
---|---
0x01| WGS84 - World Geodetic System 1984
In order to help with checking the basic parsing and encoding of this format, the following is a location subpacket (when encoded or decoded this will be found inside an ARF packet as described above).
00, 00, 00, 00, 00, 00, 00, 00, // flags
01, // system (wgs84)
3f, f3, be, 76, c8, b4, 39, 58, // latitude (1.234)
40, 02, c2, 8f, 5c, 28, f5, c3, // longitude (2.345)
40, 59, 00, 00, 00, 00, 00, 00, // elevation (100)
40, 24, 00, 00, 00, 00, 00, 00 // accuracy (10)
## Vendor Extension
In addition to the fields I put in the spec, I expect that I may need custom packet types I canât think of now. Thereâs all sorts of useful data that could be encoded into the stream, so Iâd rather there be an officially sanctioned mechanism that allows future work on the spec without constraining myself.
Just an example, Iâve used a custom subpacket to create test vectors, the data is encoded into a Vendor Extension, followed by the IQ for the modulated packet. If the demodulated data and in-band original data donât match, weâve regressed. You could imagine in-band speech-to-text, antenna rotator azimuth information, or demodulated digital sideband data (like FM HDR data) too. Or even things I canât even think of!
id
data
In order to help with checking the basic parsing and encoding of this format, the following is a vendor extension subpacket (when encoded or decoded this will be found inside an ARF packet as described above).
// extension id (b24305f6-ff73-4b7a-ae99-7a6b37a5d5cd)
b2, 43, 05, f6, ff, 73, 4b, 7a,
ae, 99, 7a, 6b, 37, a5, d5, cd,
// data (0x01, 0x02, 0x03, 0x04, 0x05)
01, 02, 03, 04, 05
# Tradeoffs
The biggest tradeoff that Iâm not _entirely_ happy with is limiting the length of a packet to `u16` â 65535 bytes. Given the u8 sample header, this limits us to 8191 32 bit sample pairs at a time. I wound up believing that the overhead in terms of additional packet framing is worth it â because always encoding 4 byte lengths felt like overkill, and a dynamic length scheme ballooned codepaths in the decoder that I was trying to keep as easy to change as possible as I worked with the format.
