Data format on X6 packet streaming boards

[jhenderson]

Innovative's X6 -series XMC products and the PEX6-COP and VPX6-COP utilize industry-standard VITA49 packets to communicate data between devices during data streaming. The VITA49 protocol is described in attachment X6_subpacket.pdf and ([url]http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=6&ved=0CEkQFjAF&url=http%3A%2F%2Fwdv.com%2FElectronics%2FReference%2FSDROoverWEB-VITA.pdf&ei=zqodT7_eFoWbiALT0OnICA&usg=AFQjCNGv4EQiJxtsCcecEAjh3AURJ5tJmw&sig2=xmI-qSYa-c9r3V2RyJwmKw)[/url].

Data exchanged directly between FPGA devices over rocket I/O channels is communicated using this protocol natively. That is, the VITA49 packet is the minimum atom of data exchange for inter-FPGA-device connections. However, data exchanged with a host processor employ Velocia packets as a container for multiple VITA sub-packets in order to mitigate per-packet interrupt processing overhead. This approach is detailed below.

Familiarize yourself with the contents of Chapter 13. in the Malibu Library User's Guide: Using the X6 Family Baseboards in Malibu. The information below assumes a working understanding of that information.

X6 data streams contain a variable number of Velocia packets each consisting of a header followed by data. Each header defines the size of the subsequent data field, in units of 32-bit words.

Velocia Packet Format:
Header: 4, 32-bit words (versus X3/X5 products whose header size is 2 32-bit words)
Data: Variable length number of 32-bit words as specified by header.

Header Format
Header word0, bits 0..23: Size of Data field, in 32-bit words
Header word0, bits 24..31: ID # of peripheral which sourced the data
Header word 1..3, bits 0..31: Zero (application-specific)

The ID field in header word 0, bits 28..31 identifies the source of the data. For instance on the X6-400M, ID #0 corresponds to time series data sent from the card to the Host while ID #2 corresponds to time-series data which is generated by the Host software and sent to the card. Words 1..3 are reserved for application-specific purposes, but typically are initialized to zero in stock factory logic.

Data Format
The data payload within each Velocia packet contains one or more VITA49 sub-packets. The data payload of each VITA49 packet is sourced from a single device whose ID is contained in the header for that packet. Typically, 0x1000 samples of data are contained in the payload of each VITA49 packet.

For instance, the ADCs on the X6-400M, produce samples which are 14-bits signed extended into a 16-bit field (two bytes/sample). That is, the X6-400M produces C-style short, 16-bit data within the data field of its VITA49 sub-packets. Sample data produced by ADC 0 is tagged with source ID (SID) 100, whereas sample data produced by ADC 1 is tagged with SID 101. Time series data for each channel is packed into the data payload of separate VITA49 packets. If only channel 0 is enabled, VITA49 packets with SID 101 would not appear within the Velocia packets sent to the host.

If ADC0 only were enabled, the format of the payload of each of it's VITA49 packets would be:

DATA:
16-bit A/D0 sample 0
16-bit A/D0 sample 1
...
16-bit A/D0 sample N

If ADC1 only, the format is identical to above, except all data in packet is sourced from A/D #1.

The Malibu VitaPacketDataFile object can be used to parse files containing Velocia packets containing VITA49 sub-packets. BinView uses this object to allow real-time viewing of X6 data sets.

Channel Extraction
Due to the complexity in dealing with VITA49 subpackets embedded within Velocia container packets is significant and potentially time-consuming, a means of efficiently extracting the samples corresponding to each channel is required. This can most easily be accomplished using the VITA packet parser object (Innovative::VeloMergeParser) provided within the Malibu libraries.

VeloMergeParser parses all of the VITA49 sub-packets contained within Velocia packets received from the X6 data stream. Data is automatically sorted according to source device ID. Data from each device ID is accumulated internally until a user-specified threshold is exceeded, upon which the data is packaged into a standard Malibu Buffer object and passed to the application software for processing via a dedicated event handler.

Assume that an X6 -series card is delivering payload data containing 16-bit samples acquired from two active channels and that the extracted data is to be enqueued into three separate STL-based channel FIFOs. One possible implementation of the received data event handler is:

[mattijsdegroot]

I still have a question on how to use the VMPDataAvailable function exactly.

You claim that your code illustrates how to enqueue the extracted data into three separate STL-based channel FIFOs, but I see only a VMPLogger.LogWithHeader call and not an enqueue of the data.

I would like to copy the sample data (without headers) into a separate buffer for each channel. And perhaps it would be nice to also copy all alert packages that occur in the stream into a separate alert buffer. Can you illustrate how that should be done?

I will do some further reading to see If I can solve it myself, but if you could point me in the right direction it would be very much appreciated.

Best regards,
Mattijs

[csmith]

The Merge Parser will divide the packets from the N data sources into separate buffers, calling VMPDataAvaiable() when each fills up. In this case, we want to log them all with headers to one file, because our analysis program aligns the data and displays it all together.

To do what you want, you need to change VMPDataAvailable() to inspect the header of the packet and call Log() to put the packet in one of N previously set up DataLogger objects.

Alerts are different - for one thing they are by default screened out of the data set before HandleDataAvailable(). If you include them, then you need to inspect the PeripheralID in order to separate alert packets from data packets. Once you do that, you just log the alert packets in its own DataLogger.

I don't recall if the normal alert callback gives you access to the raw alert event information.

There is a post in the X6-1000 section called "Packet Streaming and Parsing" that has some code examples of reading the header of Velo Packets. This might help you understand.

[mattijsdegroot]

Thanks for your answer,

Instead of calling Log() I plan to take the data from the packet after inspection of the header and copy it into a buffer that I can read from Labview. Are there any problems with that? Is it guaranteed that all VeloBuffers (except the last one in an acquisition perhaps) all have the size requested? Is just getting the data pointer with Event.Data.Data() and copying SizeInBytes() bytes
to my own buffer the way to do it?

Best,
Mattijs

[csmith]

Technically, there is no guarantee that an incoming Velo buffer is of any particular size. In practice, they all are the size you request. In any event, if you read the "Size" from the packet when you get it, you will know the correct size to be used and avoid overrunning.

You can also look at the AccessDatagram and other Datagram classes that are provided to automatically simplify the viewing of a buffer as a float, double, int, char, or even a structure. It converts the sizes into the correct units to hide all those '*sizeof(int)'

As long as you do a copy, you should be OK. The data in a buffer will probably be released at the end of the event handler function, so passing a pointer to be used later would be a problem unless you saved the buffer too.

[mattijsdegroot]

It looks like the VeloMergeParser cannot keep up with the full speed of the board on our system (as expected). I would like to test the maximum speed at which I can reliably obtain all samples.

What happens when parsing cannot keep up? Are the packets in the VMP buffer overwritten? Is there a way to tell how many packets were lost because of slow parsing?

Mattijs

[csmith]

The Merge Parser never overwrites or loses anything internally - every buffer you insert is fully parsed and the merge data given to you for processing.

If it is too slow, Velo packets build up inside of Malibu, and credit is denied to the target card, and its internal queues fill and eventually data is lost at the analog end of the system. So the result of not keeping up are gaps in the analog data, not in the packets.

The best way to test the rate of the Merge Parser is to cram packets into it in a tight loop and measure how long it takes to finish the loop. This doesn't require hardware. The rates you can get depend on the size of the Velo and Vita packets in, and the output Merge Packet size as well. Bigger packets are better.

The Merge Parser requires that the Vita Packets in it be aligned with the start of the Velo Packets. This gives it some additional speed, and also means that if you can just not submit a packet to the parser at intervals to increase the effective rate while losing total coverage. For instance, once a second you could drop a 100 packets or so in and discard the rest to give you a snapshot while not affecting the overall throughput very much.

[collingwood]

Time series data for each channel is packed into the data payload of separate VITA49 packets............./

[csmith]

In general, the format of the data in each VITA packet is up to the logic and thus can vary. But the normal case on input is to have a channel in its own VITA packet.

[ARNAUD]

On the first post, there is a picture of alert format.
This picture is not available.

Can you add it?

[jhenderson]

Image added