EasyDRF

[Easy Digital Radio Files]

* Some standard Windows library functions and compiler generated code *may* access the Windows Registry.
 

** Latest EasyDRF updates **
 

Back to Tech Menu


EasyDRF is on GitHub:

Main:
https://github.com/DazDSP/EasyDRF

Executable (updated at times*) Click file then use "Download RAW" to save.
https://github.com/DazDSP/EasyDRF/tree/master/Release

** INSTALLATION **

NOTE: Bare program executable file, no installer. To install, make a new folder called "EasyDRF" on C drive, and move the downloaded file into it. Make a shortcut to EasyDRF.exe and move the shortcut to somewhere convenient for launching the app. That's all!

Important: Do not install EasyDRF in or under a system folder such as Downloads, or it can cause security problems - such as failure to create folders or save files.


* EasyDRF is being updated less often now, as the performance is largely adequate and each update requires the executable to be submitted to various antivirus vendors for analysis.

NOTE: EasyDRF has been cleared by Microsoft as "Not malware". If it is detected as malware in Windows, please update your antivirus definitions.
 

If you still have concerns:

You can get a free second opinion from HouseCall:
https://www.trendmicro.com/en_au/forHome/products/housecall.html

EasyDRF can be exempted from scans:
https://mspoweruser.com/getting-false-positives-heres-how-to-add-exemptions-to-windows-defenders-scan/


Regarding security:

Shortwave Radiogram and Radio Northern Europe International


EasyDRF is routinely used to encode playlist data for Radio Northern Europe International (RNEI), and was also used for the first Shortwave Radiogram HamDRM test broadcast on 21st, 25th and 28th April 2021.

The Shortwave Radiogram test showed how the simple carousel repeat method of error correction was inefficient, and could easily fail due to repeated errors in the same place.

Since that time, Reed-Solomon coding has been added to EasyDRF for vastly improved error performance. Some other protocol weaknesses have also been addressed.

These protocol improvements mean that EasyDRF is now the only program that can decode this content. The RS data is not EasyPal compatible.

Another test broadcast using these advances was produced, but was not broadcast due to the already busy schedule of the Shortwave Radiogram creator.

Instead, there was an RNEI Radiogram broadcast on February 3rd, 2022 (UTC). DETAILS HERE. RECORDING HERE.


There was another Shortwave Radiogram EasyDRF special broadcast from May 11-16, 2023.

Details are here: EasyDRF Shortwave Radiogram - May 2023.

Some videos of the decoding are here:
Part 1.
Part 2.

There was another Shortwave Radiogram EasyDRF special broadcast from November 17-20, 2023.

Details are here: EasyDRF Shortwave Radiogram - Nov 2023

Some videos of the decoding are here:
Part 1.
Part 2.


** Quick Setup for EasyDRF Radiogram Decoding from AM Broadcasts **


EasyDRF adds some new features and removes others.

EasyDRF was based on an older version of the WinDRM source, merged with the HamDRM DLL source. The old WinDRM source lacked certain features, and contained bugs.

EasyDRF Main Features:

New features/bugfixes:

The EasyDRF RS data size protocol was updated on Nov 10th, 2021. This is not compatible with older versions. To decode recordings from older versions requires the old software dated Oct 31st. The new software has some degree of backwards compatibility, but this requires the old header to be received - and this can't be guaranteed in variable shortwave conditions.


Removed features:


Future experimental features or improvements:

Some possible broadcasting uses of EasyDRF

The encoded signals can be sent over standard 3kHz SSB HF channels, or via standard AM broadcast transmitters on any suitable radio band.

The great efficiency in transmitting compressed data, plus the heavy error correction and robust QAM4 mode makes the system able to cope with poor radio conditions.

For automated transmissions, the software would ideally need to be made into a multi-platform command line program.

Broadcast Optimization

For best data decoding performance, some special steps are needed before transmission:

Update - PAPR processing added

PAPR = Peak to Average Power Ratio.

Automatic PAPR transmit processing has been added to EasyDRF. PAPR clips and filters the data signal for higher average power. This is done by converting the OFDM audio signal to a Weaver-mode SSB 0Hz IF IQ signal, processing, then converting back to audio again.

CESSB overshoot compensation methods are used to prevent filter overshoots. The peak output level is very well controlled.

Hilbert clipping and filtering is performed at a reduced samplerate of 12kHz to make the filter performance better, and to reduce CPU load. Using multirate processing makes the output very clean, because the 12kHz rate allows sharper filters. All filters are FIR types for linear phase.

The peak audio output level is now set higher at -3dBFS (previously was -12dBFS).

Output spectrum in QAM4 mode with 6dB clipping at 48kHz samplerate:


Above: A decode from WRMI with low SNR, showing a
good decode even with serious segment loss.
Via a Kuwait KiwiSDR - 11,680km on 7780kHz.

EasyDRF State:

IO: Audio interface.

Freq: Frequency sync.

Time: Time/Phase sync.

Frame: Frame sync.

FAC: Fast Access Channel CRC status.

MSC: Main Service Channel CRC status.
 

EasyDRF Displays:

Call: Callsign setting of transmitting station.

SNR: Decoder Signal-to-Noise level (signal quality).

Info: Total data segments / Good data segments / Current data segment.

Codec: RS mode of data (if used) and "Data" in Data mode, Speech codec type in Voice mode.

DC: DC (zero) frequency of COFDM signal (default 350Hz).

Mode: HamDRM mode letter / Interleaving / QAM level / MSC protection level / Bandwidth.
 

EasyDRF Stats:

[filename]: Filename of file being sent. A .gz or .lz extension is added if the data was compressed for transmission. This is removed after decompression and before saving.

E: The last RS error code for debugging purposes.

Bytes: The data size of the (uncompressed/compressed and/or RS encoded) file being sent. In RS modes, this will be a multiple of 255, as the RS block size is 255 bytes.

ID: The transport ID of the data object.

Bfr: The RS and Erasure buffer number being used (0 or 1).
 

EasyDRF Filestate:

-blank- No signal. Decoder is not locked.

WAIT nn%: (Blue) Waiting for adequate data - now with percent completed display. This is the percent of total file or RS data that has arrived. RS data will decode before 100%.

Try... nnn: (Yellow) RS decoding was attempted, and has failed with errors. More data is needed. Number shows decode attempt count.

SAVED: (Green) Good decode. File has been saved. The green colour persists for four seconds minimum.

FAILED: (Red) Failed decode. In a transmitted file sequence this text may not be displayed, due to the next file starting immediately after. The red colour persists for four seconds.
 

EasyDRF buttons:

RESET: Reset the decoder and re-acquire the signal. Only use this if the decoder is taking too long to automatically reset at the start of a new transmission.

BSR: Bad Segment Request - (For amateur two-way communication only) - Send a request to the other station to re-transmit the file segments the decoder missed. (The BSR function only works for files sent WITHOUT RS coding. With RS coding, there really is no need for BSR requests if the RS level is adequate for the radio conditions.) * BSR now DOES work with on-the-fly LZMA compressed files. Nov 23, 2021.

SPA: Show Picture Anyway - Save an incomplete file, even though it will be corrupt. This won't work if the header is missed. (For uncompressed files and NON-RS modes only.)

Rx Files: Open the folder where the received files are saved.

These buttons are for sending waterfall text. The buttons no longer need pre-recorded WAV files, and can't play them anymore.
G: "GOOD COPY". This is for showing the other station that you had a good decode. Loads "good.txt" if present, for multilanguage support.
B: "BAD COPY". This is for showing the other station that you had a bad decode. Loads "bad.txt" if present, for multilanguage support.
ID: Sends the callsign. This is to ID using your callsign.
TUNE: Send HamDRM tuning tones. This is for playing a tuning signal to help the other station tune accurately to you.

Transmit control buttons:
TX File: Display the Transmit File selection dialog.
TX Voice: Begin transmitting in digital voice mode. Digital voice mode is now fixed - Oct 24th 2022.
 

EasyDRF instructions:

 
Receive:
Start the application and set the audio input to the source. (To feed audio in from another application, you may need a "virtual audio cable".)

When a file is received, it's filename will be displayed on the left side of the status bar. The bargraph shows the data progress. SNR shows the signal-to-noise ratio of the incoming signal. If the SNR is high enough for the QAM mode used, the data will be good (green). If the SNR falls too low, the data segments will get CRC errors (red).

In RS modes, the file will save before it reaches 100%, if there is enough good data recovered. When the file is saved, "SAVED" will appear on the right side of the status bar.

Click "Rx Files" to open the Rx Files folder where the file was saved. Compressible files like text or html are compressed on-the-fly using LZMA, and are then automatically decompressed before saving.

Received files can be displayed in various ways:

Web browsers are good general purpose viewers for text, html, jpg, webp, gif and png files.

If an html file is sent before some image files, it can include a javascript loader to automatically load the images as they are received. This is done on the Shortwave Radiogram HamDRM test broadcasts.

Important notes:

Advanced signal recovery methods for AM data broadcasts:
 
The multi-pass decoding method shown below is now largely superseded by the AM phase correction method:
  • Demodulate the AM signal using AM-Sync as independent LSB and USB channels, or as I and Q channels.
  • Automatically phase align the audio of each channel by splitting the audio into multiple bands, measuring the phase of each corresponding channel, and rotating them both to match. This needs to be done using a DSP plugin, or an advanced SDR.
  • Phase correct the audio to compensate for all filter and phase rotator delays in the transmit and receive processing. This typically requires the higher frequencies to be delayed more than the lower frequencies.
An AM-Sync demodulator with these advanced methods is being developed.


Sometimes a weak or noisy AM signal is just below the RS decoding threshold. In this case, it is possible to get more good data out of the signal by special processing. This requires an IQ (IF) or ISB (demodulated) recording of the signal.

The signal must be demodulated into LSB and USB using a carrier phase-locked ISB or ECSSB demodulator, and the LSB and USB audio is combined in the following ways in a sound editor (such as Audacity):

The file audio is added by playing them simultaneously in a multitrack editor.

Play each audio combination into the decoder, and the good segment data will slowly accumulate. It's also worth making more than one pass, as this can recover additional data.

* NOTE: EasyDRF only uses the LEFT audio input. Sound file output should be made into mono by splitting stereo files into separate mono channels, and reducing the gain on each if needed to avoid clipping.

In really difficult cases, a 0.5 to 1 mS delay can be added to the combined sidebands to move the selective fading notches and recover even more data:

One final trick is to use audio compression/limiting on the audio to reduce the fading severity and recover even more data. The recovery time of the compressor/limiter should be around 100mS.

These methods can be used to recover signals that are a long way below the decoding threshold, but they are tedious.


 
Transmit:
First, set your callsign under Setup -> Callsign.

Next, select transmit settings in "Settings" menu.

MODES

Mode A: Fastest. For groundwave or satellite use, where there is no multipath.
Mode B: Fast. For HF, where there is some multipath.
Mode E: Slower. For HF, where there is multipath and severe Doppler shift.

QAM4: Robust. Most resistant to noise and distortion.
QAM16: Faster. Will tolerate some noise and distortion.
QAM64: Fastest. Needs good SNR and low distortion.

MSC Protection: Normal. Slower. Maximum OFDM error correction. (Not functional in QAM4 mode.)
MSC Protection: Low. Faster. Less OFDM error correction.

Interleave Short 400mS: Lower latency.
Interleave Long 2s: Good for slower fading.

DC Offset: 350Hz is the standard lower limit of the OFDM signal.

Bandwidth 2.5kHz: Faster and slightly wider. Choose this for AM use.
Bandwidth 2.3kHz: Slightly slower but able to fit through narrower 2.4kHz SSB filters.

When transmit configuration is complete, click OK.

Click Tx File button.

Select number of repeats (old) or RS error correction mode (new, and far better).

Select Long Leadin to allow the decoder more time to lock.

Select Sequence Add to automatically add a numbered file sequence to the transmit list.

Click Add File to open the file select dialog. Only one file at a time can be added by this method.

Click Return to cancel, or TX to transmit the files in the transmit list.

Files are automatically compressed using LZMA if they have any of the following extensions:
html, htm, css, js, xml, json, ico, svg, mid, txt.


New features:

Work continues to improve the reliability of EasyDRF in difficult shortwave conditions.

Previously, EasyDRF used a repeat method to help ensure all file segments are decoded. This was wasteful of bandwidth, and each pass could still lose the same segment - making it fail.

Another common HamDRM application called "EasyPal" uses interleaved Reed-Solomon (RS) encoding to improve this situation. Unfortunately, the protocol extension it uses is unknown, and it still has weaknesses.

The error correction scheme used on the HamDRM version of Shortwave Radiogram program 200 was to send the HTML 3 times, and the images 2 times each. This reduced the throughput for the HTML to 33%, and the images to 50%. Some files can still be lost due to data being lost in the same segments on each pass.

Interleaved Reed-Solomon coding (with erasure data processing) can correct for certain amounts of data loss at any position in the data stream, with less overhead than repeats:
 

 RS
Level 		 RS
Code		 Overhead  Data Loss
Tolerance 		 Transfer
Speed 
1 255,224 +14% 11% 88%
2 255,192 +33% 24% 75%
3 255,160 +59% 36% 63%
4 255,128 +99% 49% 50%

255 = Block size
224,192,160,128 = Data size
Parity size = Block size - Data size

The EasyDRF RS modes are not compatible with EasyPal,

even though they are similar.


Data loss during reception is mostly caused by interference or signal fades taking the SNR below a usable level for some file segments. Even if the signal is at a usable level for most of the time, the fading can make decoding impossible due to these missing file segments. RS coding can fully correct this, up to a certain percentage of missing segments.

RS coding of the transmitted file data greatly improves the reliability of file transmission, but the file header information also needs to be made robust, as the header contains the filename and file size information. If this is lost, decoding is impossible.

A new scheme has been developed for EasyDRF, where a new header is appended to the file before RS coding and transmission. This ensures the new header always decodes if the file decodes successfully.

After decoding, EasyDRF removes the new header from the incoming file before saving it.

New file header format:

The old header is still sent, but the new header is more robust and is guaranteed to decode if the file decodes. This avoids the problem of having no filename or size information to save the file with.

The RS data size in bytes or total RS block count information is necessary to decide when to attempt the decode of the RS encoded data. The exact RS data size is required for the deinterleaving to work correctly.

But due to the interleaved RS coding, if this header is appended to the file it isn't possible to read it until after RS decoding. Since the exact RS data size is needed before deinterleaving, the new header can't be used to send file size information in a timely way.

To get around this, the number of 255 byte RS data blocks is now sent in serial data form, using four of the unused segment header bits in the "Repetition index" field. This allows the data to arrive in only 4 segments time, and it continuously repeats all through the file transmission in case any segments are lost. This RS data block count is multiplied by 255 in the decoder to get the total RS data size in bytes, which is then used to compute the total segment count and to compute the proportion of correctly received segments. This allows a good decision to be made on when to attempt the RS decode, and the RS data size allows correct deinterleaving of the RS data.

Also, the RS mode being used is signalled using the "Reserved for future addition" segment header bits in the "User access field", allowing the RS mode to be easily and reliably identified.

Modified signalling:

  1. "CRC used" decoder field is locked in the always ON state, since it is always used. This eliminates a possible source of errors. Added Sep 17th, 2021.
  2. (Previously unused) "Repetition index" field = 4 bits of file size data sent in serial form, timing locked to the segment number in Modulo 7 (LOW nibble first). Old method, pre Nov 10, 2021.
  3. (Previously unused) "Repetition index" field = 4 bits of RS block count data sent in serial form, timing locked to the segment number in Modulo 4 (LOW nibble first). Updated Nov 10, 2021.
  4. (Previously unused) "Reserved for future addition" field = 3 bits to indicate the RS mode being sent.
#2 updated to #3:
Since the RS block size is 255 bytes, the RS data size is always a multiple of 255. This means it is wasteful to send the exact RS data size in bytes via the 4-bit "Repetition index" field, as it takes longer to send this 28 bit value (7 segments time).

Instead, the number of RS data blocks is now sent, and this number is multiplied by 255 after decoding to compute the RS data size in bytes. This only takes 4 segments time to send. The particular nibble being sent is computed by the segment number Modulo 4 (LOW nibble first).

Using less segments to send the data increases the chance of this information being correctly received. 12 bits (3 segments x 4 bits) would only allow a data size of 1M, which is slightly inadequate. 16 bits (4 segments x 4 bits) allows an RS data size of 16.7M, which is more than adequate - and still an improvement over sending the byte total (4 segments instead of 7).

Unfortunately, this change breaks compatibility with older versions (Oct 31, 2021 and prior). The Oct 31, 2021 version is still available using the GitHub file history in case it is needed to decode old broadcast recordings.

4-bit "Repetition Index" field reassignment (now sends RS block count):

Bit Mask Segment # Sequence (MOD 4)
0000000000001111 0
0000000011110000 1
0000111100000000 2
1111000000000000 3

3-bit "Reserved for Future Addition" field reassignment (now sends RS mode):

RFA#
Bits
RS Mode
0
000
None
1
001
RS1
2
010
RS2
3
011
RS3
4
100
RS4
5
101
future
6
110
future
7
111
future

The Dream/WinDRM limitation:

EasyDRF is based on WinDRM which is based on Dream, using the MOT data function to send files. The Dream MOT function divides files into segments for transmission, and requires all segments to be received before the file can be saved. It also requires the file header information in segment 0 to know the filename and filesize. The Dream MOT code saves the incoming DRM modem serial data into a buffer array for each segment, and when the header is received it copies all these buffers into a new buffer in byte form.

Even if the filename and size information is sent by alternate means, and the file segment data is sent using RS coding to make it resistant to data loss, the byte data is not normally made available if the header data fails to decode.

This required a work-around to allow the serial segment data to be directly converted into byte data and sent out to the RS decoder routine.

The work-around was to add new code to read out the existing serial buffers, decode the serial data to byte format, and save into a new buffer.
 

Summary of changes:


SNR Stats logging:

Added Sep 26th, 2021.
Received signal data is saved to a separate Javascript file if filenames have an "SWRG", "RNEI" or "RCAR" prefix. The stats filename is in the form "SWRG*.js", "RNEI*.js" or "RCAR*.js". This is included to allow easy collection of stats for shortwave broadcasting applications. The .js file is a set of Javascript objects, with variables for each file denoting the SNR and segment counts.

The following data stats are saved for each file:

The filenames must be in this form:

SWRG - Shortwave Radiogram program
HTML file: SWRG-nnn-00.html Main text file including Javascript image loader and SNR loader.
Image files: SWRG-nnn-nn.ext Image files - WEBP, SVG and AVIF have been used. JPG and GIF are also possible.

RNEI - Radio Northern Europe International - Music Program
HTML file: RNEI*.html Main text file including SNR loader. RNEI typically sends only one HTML file for the program playlist.

RCAR - Radio Carpathia - Music Program
HTML file: RCAR*.html Main text file including SNR loader. RCAR typically sends only one HTML file for the program playlist.

The file SNR and segment data is only saved if the file is saved. If the file doesn't decode, there is no data saved for the file.

Updated Nov 27th, 2021:
Stats logging is now done for each incoming good file segment. This allows the webpage display to show real-time SNR and file progress, and to save stats even if a file fails to decode. To display stats, the first file must be HTML and it must contain Javascript code to display the stats on the page. This also means the HTML file must successfully save before stats can be shown.

NOTE: If the old header (segment 0) is missed, the filename will be unknown until either the last segment is received, or until RS decoding takes place. No stats can be saved for unknown files unless the file successfully decodes so the filename is known. This is because the file number is derived from the filename.
 

Internal tone and waterfall text generation:

The G, B, and ID buttons now generate waterfall text without requiring external audio files. The TUNE button also no longer requires an external audio file.

When these buttons are clicked, the corresponding text message is sent (or tones in the case of the TUNE button).

* Please note: On Linux systems using Wine, the default location for the "good.txt" and "bad.txt" files could be the user home directory, and not the directory EasyDRF is installed in.

Digital Voice Audio Scope:

An Audio Scope has been added for Digital Voice modes. The scope displays incoming and outgoing voice waveforms, and in transmit mode the trace turns red when the level is nearing overload. The scope only displays voice input and output, and not the OFDM signal.


Experimental update to LPC-10 Digital Voice mode:

The LPC-10 Digital voice mode has been updated to work on all HamDRM data modes. This allows Digital Voice to use the robust QAM4 mode, but at lower quality.

Unfortunately, the LPC-10 CODEC is designed for a fixed samplerate of 8kHz, so running it at other rates causes problems, such as aliasing and poor speech quality.

The aliasing has been fixed by modifying the code to adjust the LPC-10 sample blocksize over the largest range it will run at. This has made significant improvements to quality, but caused problems with CODEC stability and accuracy. Some of the LPC-10 buffers have been made larger, and various filters and parameters have been tuned to improve stability and sound quality.

Most of these errors and glitches at low rates have now been fixed. At higher rates (QAM64) the LPC-10 CODEC performs poorly, as the samplerate is too high and the speech parameters scale incorrectly. At lower rates, performance is better, but varies at each bitrate.

Performance has been optimized at these two mode combinations:
Normal: Mode B, QAM16, Normal Protection, 2.5kHz.
Robust: Mode B, QAM4, 2.3kHz.

An advanced quality audio compressor and noise gate have also been added to the Digital Voice mode audio input. This amplifies the audio input for a consistent speech level, while suppressing background noise which causes severe CODEC artifacts. The audio compressor and noisegate feature delayed adaptive recovery time, and lookahead limiting for extremely low distortion levels.

The audio compressor can be enabled and disabled in the Settings menu:

Work is ongoing to improve performance...
 

Known bugs and limitations:

Ideas for improvements:


Any thoughts or suggestions on these topics can be made to: