Intel HEX File Parsing using C++

INTRODUCTION

---

I am trying to make an ESP8266 based Arduino Serial Uploader using its SPI File System.
(Project ongoing || I will post the project link as soon as I finish it. But, you can always follow the GitHub repo)

Since I plan to upload the Arduino IDE .hex compile output directly to the file system, I need to parse it and give byte* output before uploading it to the system flash.

You must be wondering why hex and not binary straightaway?
Well, if you want to do some editing on the output file, .bin displays garbage in the text editor.

HEX files (basically ASCII text files) are larger and more complicated than binary files because they consist of the binary data plus various information like memory addresses, where the data needs to be written.
Example File:

Scared much?

There are many HEX parser libraries available out there, but I needed a bare minimum one.
So, let’s get started.

EXPLANATION

---

You can go through the nitty-gritty of the Intel HEX format here and understand the nomenclature/characteristics.
After you are able to read a HEX file, the C++ implementation should be a walk in the park.

I made a HEXparser class to use it later in my Web Server code.
Here’s the class declaration:

class HEXparser
{
	public:
		HEXparser();
		void ParseRecord(byte* data); // Import data from spiffs.open() in web server.
		byte* FetchRaw();             // will be called from main server code
		byte* FetchAddress();	      // for flashing

	private:
		int recType = 0;              // Here only 0/1 is needed-data present/end of file.
		int length = 0;  
		int index = 0;                // to keep track of the no. of bytes in each chunk
		byte RawPage[128];            // sending 128 bytes of 'chunk' each time
		byte address[2];              // storage array
		bool firstTime = true;        // data address is to be updated only after the first chunk transfer

		/* Intel HEX file format info */
		int RecordLength(byte* record);
		void RecordAddress(byte* record);
		int RecordType(byte* record);
                //                             //
		void extractData(byte* record, int len);
		void FileEnd();
};

THE GITHUB REPOSITORY

CODE BREAKDOWN

---

We basically loop over a given file, going through each line/HEX record and parsing them.

1. How each record will be extracted from the file (I have not written the server code yet. This is just for reference):

void Server::proto(WiFiClient* client, String filen) 
{
  SPIFFS.begin();
  File file = SPIFFS.open(filen, "r");
  
  if(file) 
  {
    HEXparser hexParse;
    while(file.available()) 
    {
      byte holder[50];
      String data = file.readStringUntil('\n');
      data.getBytes(holder, data.length());
      hexParse.ParseRecord(holder);
      
      if(hexParse.IsPageReady()) 
      {
        byte* page = hexParse.FetchRaw();
        byte* address = hexParse.FetchAddress();
      }
    }
  }
  file.close();
  SPIFFS.end();
}

2. Now, let’s parse the record.

This is the main function. All the function calls are explained after this.

void HEXparser::ParseRecord(byte* record)
{ 
   recType = RecordType(record);
   if(recType == 0)                             //  data present
	{
	 length = RecordLength(record);
	 extractData(record, length);
         if(index == 128)                      // chunk full
		{
		 if(!firstTime)
			{
			 address[1] += 0x40;   //update data address 
                         if(address[1] == 0)
				{
				   address[0] += 0x1;
				}
			}
                        index = 0;
			firstTime = false;
		}
	}
        if(recType == 1)                        // end of file
	{
	    FileEnd();
	}
}

3. Analyzing the record

Extracting useful information about the record, following the Intel format:

// length = number of data bytes
int HEXparser::RecordLength(byte* record)
{
	char holder[3];

	holder[0] = record[1];
	holder[1] = record[2];
	holder[2] = '\0';

	return strtol(holder, 0, 16);
}

// the address where the corresponding record data are to be allocated in memory.
void HEXparser::RecordAddress(byte* record)
{
	char holder[3];
	
	holder[2] = '\0';

	holder[0] = record[3];
	holder[1] = record[4];
	address[0] = strtol(buff, 0, 16);

	holder[0] = record[5];
	holder[1] = record[6];
	address[1] = strtol(buff, 0, 16); 
}

// data or end of file
int HEXparser::RecordType(byte* record)
{
	char holder[3];

	holder[0] = record[7];
	holder[1] = record[8];
	holder[2] = '\0';

	return strtol(holder, 0, 16);	
}

4. Extracting and preparing the program data to be flash-ready.

First, we need to convert the whole data to bytes.
For this, a flash page is created, which holds the data bytes. Now, data is not sent continuously through the STK500v1 protocol (which I’ll be using to program).
It needs to be chunks of data, 128 bytes of them ( source: avrdude.conf from arduino IDE files).

// each record is passed
void HEXparser::extractData(byte* record, int len)
{
	int begin = 9;             // start position of data in record
	int end = (len*2) + begin; //data is in sets of 2
	char holder[3];
	holder[2] = '\0';

	for(int i=begin;begin<end;i=i+2)
	{
		holder[0] = record[i];
		holder[1] = record[i+1];

		RawPage[index] = strtol(holder, 0, 16);
		index++;                  // keeping track of chunk size
	}
}

Configuring Chunks

• Addresses are updated once index exceeds 127 to form a new page/chunk.

•  I came across another case where we need to control the page index. When we are done traversing the whole file, but index is less than 127, we need to fill the unused areas with the specified constant value. A fill value of 0xFF is often used with this option because it corresponds to erased flash.

// check record type||chunk filled or not
void HEXparser::FileEnd()
{
	address[1] += 0x40;
	if(address[1] == 0)
	{
		address[0] += 0x1;
	}

	while(index<128)
	{
		RawPage[index] = 0xFF;
		index++ ;
	}
}

5. Finally, sending the RawPage and address to the main server code, used while flashing the page.

/* Return calls to spiff/web server */
byte* HEXparser::FetchRaw()
{
	return RawPage;
}

byte* HEXparser::FetchAddress()
{
	return address;
}