Solution for session freezing / overruns @ 2kHz sample rate (with SD only)
Hello,
Freezing during long sessions, low 250Hz sample rate, 150mS (and more) overruns, all that bothered me.
Then I started to play with a version of the soft proposed by @biomurph : https://github.com/OpenBCI/OpenBCI_32-write-to-DS-only
To only write on the SD Card, not using the RfArduino, suppressed the freezing problem...
But overruns were always there.
Since these overruns seemed to be due to a "wait SD not busy" function, I modified it to continue the pooling of the A/D converter.during the waiting time...
Of coarse we need a circular buffer and so on... Details are in the few lines of code below.
Pooling during the waiting time was not enough , I had to add pooling at some critical point in the main loop...
To verify the result I "time stamped" each sample using the "input capture" of a timer value on the occurence of the Data Ready falling edge. I have written this value after the 8 channels on the SD card file. I have also written a "tag" to know when or where from the code the pooling pushed datas in the buffer. To understand what was going on I added the values of the writing and reading indexes when pushing and pulling. And finally I wrote a short matlab script to look at the OBCI_nn.txt file.
I did this with the 32bit card and a low cost SD card : SD 16Gb card from Transcend ( TS16GUSDHC10 ) Class 10 , 30MB/s, 200x.
The results probably depend on the type of SD Card...
I launch the programme with the serial terminal of the Arduino IDE, as Biomurph recommended to me.
I write : "+0++A++N+" (with out the "+" it does not work...)
At the end we still receive the message :
Max write time: 51716 uS
Min write time: 404 uS
Overruns: 5
fileBlock,micros
1,8771
608,12767
6720,12811
7744,7766
8800,51716
In fact it is when the SD waiting time occures and how long time... It does not mean that some data were lost. To know if, look at the "time stamp" in the file.
With my hardware I tested up to 2kHz and found no data lost. At 2kHz the game of read and write indexes shows that the swallowing of the buffered data becomes difficult but it goes.
@biomurph : How to send you the full code? (if you want it).
There under some details of the code and a matlab script.
Have fun,
Yannick.
//==========================================================================
// In File OpenBCI_32_SD_Only.ino
void loop() // +0++a++N+ +0++a++N+
{
if(is_running )
{
while(!( IC4CON & 0x0008)){} ; // while(!(OBCI.isDataAvailable())){} // wait for DRDY pin2.D9..PGED3/VREF+/CVREF+/AN0/C3INC/RPA0/CTED1/PMD7/RA0
OBCI.ere |=0x100 ; OBCI.push(); // reads A/D and push data , time stamp, ere, accel, aux...
while ( OBCI.read_ix != OBCI.write_ix )
{
OBCI.updateChannelData(); // get from the FIFO ADS data , time stamp, ere, accel, aux...
if( IC4CON & 0x0008 ) { OBCI.ere |=0x200 ; OBCI.push(); } ; // reads A/D and push data , time stamp, ere, accel, aux...
if(SDfileOpen)
{
writeDataToSDcard ( SDsampleCounter++ );
//if(SDsampleCounter == 0 && writeToSDonly ) { Serial0.print("SD BLOCKS written: "); Serial0.println(blockCounter); } // verbosity
}
if( IC4CON & 0x0008 ) { OBCI.ere |=0x400 ; OBCI.push(); } ; // reads A/D and push data , time stamp, ere, accel, aux...
if(!writeToSDonly) OBCI.sendChannelData();
}
if ( SDsampleCounter == 0 ) { digitalWrite(LED, HIGH); } // yj **** digitalWrite(18, HIGH); Serial0.write(0x0a);
if ( SDsampleCounter == 3 ) { digitalWrite(LED, LOW); } // yj **** digitalWrite(18, LOW );
}
eventSerial();
}
//=================================================================================
// In File Definitions_32.h
// Sample Rate 16KHz 8kHz 4kHz 2kHz 1kHz 500Hz 250Hz Do not use
// 000 001 010 011 100 101 110 111
#define Sample_Rate 0b011
#define circ_buf_len 512
#define circ_buf_mask (circ_buf_len-1)
//==========================================================================================
// In File OpenBCI_32_Daisy.h *******************************
void push(void);
byte circ_buf[32*circ_buf_len] ; // The FIFO First In - First Out ; N samples * 32 bytes
uint32_t time_stamp ; // the value of TMR4 a 32 bit timer or TMR3 16bits red by an "Input capture event" )
uint16_t ere ; // just for test...
uint16_t read_ix ; // read index pointing samples
uint16_t write_ix ; // write index
uint16_t b_time_stamp ; // From circ_buf
uint16_t b_ere ; // From circ_buf to debug I wrote these 3 informations in the buffer
uint16_t b_read_ix ; // From circ_buf "
uint16_t b_write_ix ; // From circ_buf "
//====================================================================================
// In File OpenBCI_32_Daisy.cpp **** reads and push data , time stamp, ere, accel, aux... ******************
void OpenBCI_32_Daisy::push(void)
{
csHigh(SD_SS); // release spi ????????????????
byte * ptr_buf = &circ_buf[(circ_buf_mask & write_ix)*32] ;
// Timer 3 value is in Input Capture 4 Buffer ; ere++ to note fifo's depth (I never got 2)
while(IC4CON & 0x0008) { time_stamp = IC4BUF ; ere++ ; } ;
csLow(BOARD_ADS); // open SPI for AD 1
for(int i = 0; i< 3; i++){ xfer(0x00); } // unused daisy stat
for(int i = 0; i<24; i++){ *ptr_buf++ = xfer(0x00); } // read 24 bits of channel data in 8*3 byte chunks
csHigh(BOARD_ADS); // close SPI
uint16_t * u16 = (uint16_t *) ptr_buf ;
* u16++ = (uint16_t) time_stamp ; // ; for test
* u16++ = ere ; ere = 0 ; // ; for test
* u16++ = write_ix ; write_ix++ ; // ; for test
* u16 = read_ix ; // ; for test
// To add : read & push accelerometer data
if(useAccel && LIS3DH_DataAvailable()) { LIS3DH_updateAxisData(); } // addAccelToSD = true; fresh axis data goes into the X Y Z
// To add : read & push auxliary data
if(daisyPresent)
{ ptr_buf = &circ_buf[(circ_buf_mask & write_ix)*32] ; write_ix++;
csLow(DAISY_ADS); // open SPI for AD 2
for(int i = 0; i< 3; i++){ xfer(0x00); }
for(int i = 0; i<24; i++){ *ptr_buf++ = xfer(0x00); } // read 24 bits of channel data in 8 3 byte chunks
csHigh(DAISY_ADS); // close SPI
//
// To add : read something else... alternatively or not ...
}
csLow(SD_SS); // take spi ?????????????
}
//========================================================================
// In File OpenBCI_32_Daisy.cpp
// Retrieve the fresh ADS data , time stamp, ere, accel, aux... from the FIFO
void OpenBCI_32_Daisy::updateBoardData(){
if (daisyPresent) for(int i=0; i<8; i++) lastBoardChannelDataInt[i] = boardChannelDataInt[i];
byte * ptr_buf = &circ_buf[(circ_buf_mask & read_ix)*32] ;
for(int i = 0; i<8; i++) {
for(int j=0; j<3; j++) boardChannelDataInt[i] = (boardChannelDataInt[i]<<8) | *ptr_buf++ ;
if(bitRead(boardChannelDataInt[i],23) == 1) boardChannelDataInt[i] |= 0xFF000000;
else{ boardChannelDataInt[i] &= 0x00FFFFFF; } // Convert 24bit to 32bit
}
b_time_stamp = *((uint16_t*)ptr_buf) ; // for test
ptr_buf += 2 ; b_ere = *((uint16_t*)ptr_buf) ; // for test
ptr_buf += 2 ; b_write_ix = *((uint16_t*)ptr_buf) ; // for test
ptr_buf += 2 ; b_read_ix = *((uint16_t*)ptr_buf) ; read_ix++; // for test
if(daisyPresent ){ int byteCounter = 0 ; // if (daisyPresent && !firstDataPacket) ????
for(int i=0; i<8; i++){ meanBoardChannelDataInt[i] = (lastBoardChannelDataInt[i] + boardChannelDataInt[i])/2; } // take the average of this and the last sample
for(int i=0; i<8; i++){
for( int b=2; b>=0; b--){ meanBoardDataRaw[byteCounter] = (meanBoardChannelDataInt[i] >> (b*8)) & 0xFF; byteCounter++; } // place the average values in the meanRaw array
}
}
if(firstDataPacket == true) firstDataPacket = false; // is it usefull ? when first the false mean is sent...
}
// these data being retrieved, writeDataToSDcard format them using write to Hex which calls modified writeData( pCache ) .
//=====================================================================
// In File SD_Card_Stuff.ino
// Call to overloaded writedata function in order to call OBCI.push() during SD busy state
void writeDataToSDcard(byte sampleNumber){
// just for test and debug...
...
...
convertToHex ( OBCI.b_time_stamp , 3, true ); // to verify if data lost
convertToHex ( OBCI.b_ere , 3, true ); // to know where from push() was called
convertToHex ( circ_buf_mask & OBCI.b_write_ix , 3, true ); // indexes during writing in
convertToHex ( circ_buf_mask & OBCI.b_read_ix , 3, true );
convertToHex ( circ_buf_mask & OBCI. write_ix , 3, true ); // indexes during reading out
convertToHex ( circ_buf_mask & OBCI. read_ix , 3, false );
...
...
}
THE END IN THE NEXT MESSAGE
.
Freezing during long sessions, low 250Hz sample rate, 150mS (and more) overruns, all that bothered me.
Then I started to play with a version of the soft proposed by @biomurph : https://github.com/OpenBCI/OpenBCI_32-write-to-DS-only
To only write on the SD Card, not using the RfArduino, suppressed the freezing problem...
But overruns were always there.
Since these overruns seemed to be due to a "wait SD not busy" function, I modified it to continue the pooling of the A/D converter.during the waiting time...
Of coarse we need a circular buffer and so on... Details are in the few lines of code below.
Pooling during the waiting time was not enough , I had to add pooling at some critical point in the main loop...
To verify the result I "time stamped" each sample using the "input capture" of a timer value on the occurence of the Data Ready falling edge. I have written this value after the 8 channels on the SD card file. I have also written a "tag" to know when or where from the code the pooling pushed datas in the buffer. To understand what was going on I added the values of the writing and reading indexes when pushing and pulling. And finally I wrote a short matlab script to look at the OBCI_nn.txt file.
I did this with the 32bit card and a low cost SD card : SD 16Gb card from Transcend ( TS16GUSDHC10 ) Class 10 , 30MB/s, 200x.
The results probably depend on the type of SD Card...
I launch the programme with the serial terminal of the Arduino IDE, as Biomurph recommended to me.
I write : "+0++A++N+" (with out the "+" it does not work...)
At the end we still receive the message :
Max write time: 51716 uS
Min write time: 404 uS
Overruns: 5
fileBlock,micros
1,8771
608,12767
6720,12811
7744,7766
8800,51716
In fact it is when the SD waiting time occures and how long time... It does not mean that some data were lost. To know if, look at the "time stamp" in the file.
With my hardware I tested up to 2kHz and found no data lost. At 2kHz the game of read and write indexes shows that the swallowing of the buffered data becomes difficult but it goes.
@biomurph : How to send you the full code? (if you want it).
There under some details of the code and a matlab script.
Have fun,
Yannick.
//==========================================================================
// In File OpenBCI_32_SD_Only.ino
void loop() // +0++a++N+ +0++a++N+
{
if(is_running )
{
while(!( IC4CON & 0x0008)){} ; // while(!(OBCI.isDataAvailable())){} // wait for DRDY pin2.D9..PGED3/VREF+/CVREF+/AN0/C3INC/RPA0/CTED1/PMD7/RA0
OBCI.ere |=0x100 ; OBCI.push(); // reads A/D and push data , time stamp, ere, accel, aux...
while ( OBCI.read_ix != OBCI.write_ix )
{
OBCI.updateChannelData(); // get from the FIFO ADS data , time stamp, ere, accel, aux...
if( IC4CON & 0x0008 ) { OBCI.ere |=0x200 ; OBCI.push(); } ; // reads A/D and push data , time stamp, ere, accel, aux...
if(SDfileOpen)
{
writeDataToSDcard ( SDsampleCounter++ );
//if(SDsampleCounter == 0 && writeToSDonly ) { Serial0.print("SD BLOCKS written: "); Serial0.println(blockCounter); } // verbosity
}
if( IC4CON & 0x0008 ) { OBCI.ere |=0x400 ; OBCI.push(); } ; // reads A/D and push data , time stamp, ere, accel, aux...
if(!writeToSDonly) OBCI.sendChannelData();
}
if ( SDsampleCounter == 0 ) { digitalWrite(LED, HIGH); } // yj **** digitalWrite(18, HIGH); Serial0.write(0x0a);
if ( SDsampleCounter == 3 ) { digitalWrite(LED, LOW); } // yj **** digitalWrite(18, LOW );
}
eventSerial();
}
//=================================================================================
// In File Definitions_32.h
// Sample Rate 16KHz 8kHz 4kHz 2kHz 1kHz 500Hz 250Hz Do not use
// 000 001 010 011 100 101 110 111
#define Sample_Rate 0b011
#define circ_buf_len 512
#define circ_buf_mask (circ_buf_len-1)
//==========================================================================================
// In File OpenBCI_32_Daisy.h *******************************
void push(void);
byte circ_buf[32*circ_buf_len] ; // The FIFO First In - First Out ; N samples * 32 bytes
uint32_t time_stamp ; // the value of TMR4 a 32 bit timer or TMR3 16bits red by an "Input capture event" )
uint16_t ere ; // just for test...
uint16_t read_ix ; // read index pointing samples
uint16_t write_ix ; // write index
uint16_t b_time_stamp ; // From circ_buf
uint16_t b_ere ; // From circ_buf to debug I wrote these 3 informations in the buffer
uint16_t b_read_ix ; // From circ_buf "
uint16_t b_write_ix ; // From circ_buf "
//====================================================================================
// In File OpenBCI_32_Daisy.cpp **** reads and push data , time stamp, ere, accel, aux... ******************
void OpenBCI_32_Daisy::push(void)
{
csHigh(SD_SS); // release spi ????????????????
byte * ptr_buf = &circ_buf[(circ_buf_mask & write_ix)*32] ;
// Timer 3 value is in Input Capture 4 Buffer ; ere++ to note fifo's depth (I never got 2)
while(IC4CON & 0x0008) { time_stamp = IC4BUF ; ere++ ; } ;
csLow(BOARD_ADS); // open SPI for AD 1
for(int i = 0; i< 3; i++){ xfer(0x00); } // unused daisy stat
for(int i = 0; i<24; i++){ *ptr_buf++ = xfer(0x00); } // read 24 bits of channel data in 8*3 byte chunks
csHigh(BOARD_ADS); // close SPI
uint16_t * u16 = (uint16_t *) ptr_buf ;
* u16++ = (uint16_t) time_stamp ; // ; for test
* u16++ = ere ; ere = 0 ; // ; for test
* u16++ = write_ix ; write_ix++ ; // ; for test
* u16 = read_ix ; // ; for test
// To add : read & push accelerometer data
if(useAccel && LIS3DH_DataAvailable()) { LIS3DH_updateAxisData(); } // addAccelToSD = true; fresh axis data goes into the X Y Z
// To add : read & push auxliary data
if(daisyPresent)
{ ptr_buf = &circ_buf[(circ_buf_mask & write_ix)*32] ; write_ix++;
csLow(DAISY_ADS); // open SPI for AD 2
for(int i = 0; i< 3; i++){ xfer(0x00); }
for(int i = 0; i<24; i++){ *ptr_buf++ = xfer(0x00); } // read 24 bits of channel data in 8 3 byte chunks
csHigh(DAISY_ADS); // close SPI
//
// To add : read something else... alternatively or not ...
}
csLow(SD_SS); // take spi ?????????????
}
//========================================================================
// In File OpenBCI_32_Daisy.cpp
// Retrieve the fresh ADS data , time stamp, ere, accel, aux... from the FIFO
void OpenBCI_32_Daisy::updateBoardData(){
if (daisyPresent) for(int i=0; i<8; i++) lastBoardChannelDataInt[i] = boardChannelDataInt[i];
byte * ptr_buf = &circ_buf[(circ_buf_mask & read_ix)*32] ;
for(int i = 0; i<8; i++) {
for(int j=0; j<3; j++) boardChannelDataInt[i] = (boardChannelDataInt[i]<<8) | *ptr_buf++ ;
if(bitRead(boardChannelDataInt[i],23) == 1) boardChannelDataInt[i] |= 0xFF000000;
else{ boardChannelDataInt[i] &= 0x00FFFFFF; } // Convert 24bit to 32bit
}
b_time_stamp = *((uint16_t*)ptr_buf) ; // for test
ptr_buf += 2 ; b_ere = *((uint16_t*)ptr_buf) ; // for test
ptr_buf += 2 ; b_write_ix = *((uint16_t*)ptr_buf) ; // for test
ptr_buf += 2 ; b_read_ix = *((uint16_t*)ptr_buf) ; read_ix++; // for test
if(daisyPresent ){ int byteCounter = 0 ; // if (daisyPresent && !firstDataPacket) ????
for(int i=0; i<8; i++){ meanBoardChannelDataInt[i] = (lastBoardChannelDataInt[i] + boardChannelDataInt[i])/2; } // take the average of this and the last sample
for(int i=0; i<8; i++){
for( int b=2; b>=0; b--){ meanBoardDataRaw[byteCounter] = (meanBoardChannelDataInt[i] >> (b*8)) & 0xFF; byteCounter++; } // place the average values in the meanRaw array
}
}
if(firstDataPacket == true) firstDataPacket = false; // is it usefull ? when first the false mean is sent...
}
// these data being retrieved, writeDataToSDcard format them using write to Hex which calls modified writeData( pCache ) .
//=====================================================================
// In File SD_Card_Stuff.ino
// Call to overloaded writedata function in order to call OBCI.push() during SD busy state
void writeDataToSDcard(byte sampleNumber){
// just for test and debug...
...
...
convertToHex ( OBCI.b_time_stamp , 3, true ); // to verify if data lost
convertToHex ( OBCI.b_ere , 3, true ); // to know where from push() was called
convertToHex ( circ_buf_mask & OBCI.b_write_ix , 3, true ); // indexes during writing in
convertToHex ( circ_buf_mask & OBCI.b_read_ix , 3, true );
convertToHex ( circ_buf_mask & OBCI. write_ix , 3, true ); // indexes during reading out
convertToHex ( circ_buf_mask & OBCI. read_ix , 3, false );
...
...
}
THE END IN THE NEXT MESSAGE
.
Comments
// In File SD_Card_Stuff.ino ***** Call to overloaded writedata function in order to call OBCI.push() during SD busy state *******************
void writeCache(){
.....
.....
OBCI.csLow(SD_SS); // take spi
if(!card.writeData ( pCache, &OBCI )) {Serial0.println("block write fail");} // write the block // ********* yj ********
OBCI.csHigh(SD_SS); // release spi
if( IC4CON & 0x0008 ) { OBCI.ere |=0x20 ; OBCI.push(); } ; // reads A/D and push data , time stamp, ere, accel, aux...
.....
.....
}
//=========================================================================
// In File Sd2Card.h ************************
uint8_t writeData ( const uint8_t* src ); // old one
uint8_t writeData ( const uint8_t* src , OpenBCI_32_Daisy* OBCI ); // yj
uint8_t waitNotBusy ( uint16_t timeoutMillis ); // old one
uint8_t waitNotBusy ( uint16_t timeoutMillis , OpenBCI_32_Daisy* OBCI ); // yj
//==========================================================================
// In File Sd2Card.cpp
uint8_t Sd2Card::waitNotBusy ( uint16_t timeoutMillis, OpenBCI_32_Daisy* OBCI )
{
uint16_t t0 = millis();
do
{
while(!( IC4CON & 0x0008)){} ; // Wait ADS_DRDY to synchronize , if you simply test , it is worse
chipSelectHigh() ;
(*OBCI).ere |=0x10 ;
(*OBCI).push() ;
chipSelectLow() ;
// ---- Now we have a litle bit of time to write on the SD Card , if it is ready ...
if (spiRec() == 0XFF) return true;
}
while (((uint16_t)millis() - t0) < timeoutMillis);
return false;
}
//==========================================================================
// In File OpenBCI_32_SD_Only.ino
void setup(void)
{
...
...
T2CONbits.TON = 0 ; // Stop any 16/32-bit Timer2 operation
T3CONbits.TON = 0 ; // Stop any 16-bit Timer3 operation
T2CONbits.T32 = 0 ; // =1; Enable 32-bit Timer mode
T3CONbits.TCS = 0 ; // Select internal instruction cycle clock
T3CONbits.TGATE = 0 ; // Disable Gated Timer mode
T3CONbits.TCKPS = 7 ; // Select 1:256 Prescaler
TMR3 = 0x00; // Clear 32-bit Timer (lsw) 4 et (msw) 5
PR3 = 0xffff; // Load 32-bit period value (lsw) 4 et (msw) 5 ffff
IC4CON &= 0x7fff ; // Disable and reset module, disable clocks, disable interrupt generation and allow SFR modifications
IC4CONbits.ICM = 0 ; // 010 = capture falling edge
IC4CONbits.C32 = 0 ; // use 16 bit=0 ;32=1
IC4CONbits.ICTMR = 0 ; // use T3=0 ; t2=1
// IC1CONbits.FEDGE = 1; // capture rising edge first all others after ; inutile si ICM = 010 = On falling edge
IC4CONbits.ICM = 2 ; // 010 = capture falling edge
IC4CON |= 0x8000 ; // turn on IC4 // IC4CONbits.ON = 1; "do not compile" // bit 15 = ON bit
T3CONbits.TON = 1;
...
...
}
//===== matlab script ======
% ============== lecture du time stamp , flag et indexes ========
clear all
close all
fclose all
fichier = 'E:\OBCI_0e.txt' ;
fid = fopen(fichier,'r');
ligne = 0 ;
s = fgets(fid); s = fgets(fid); s = fgets(fid);
dd = dir(fichier) ; len = length(s);
d = zeros( round( dd.bytes / len ),6 ) ;
while 1
data_ligne = sscanf(s,'%2x,%6x,%6x,%6x,%6x,%6x,%6x,%6x,%6x,%4x,%4x,%4x,%4x,%4x,%4x');
ligne = ligne+1 ;
d(ligne,:) = data_ligne( 10:15)' ;
s = fgets(fid);
if(length(s)~=len) ; fprintf('ligne incomplete\n'); break; end ;
end;
fclose(fid);
d(1:end-1,1)=diff(d(:,1)); m = find(d(1:end-1,1)<0); d(m,1) = d(m,1)+2^16 ;
d=d(find(d(:,1)),:); d = d(1:end-1,:);
d(:,1) = round(d(:,1)*256*25e-9*1000/0.5);
plot(d(:,2)); % Value of "ere" ; shows where from push() was called
figure(2); plot(d(:,1)) % time between samples ; If no overruns, all values should be "1"
% look at the figure(1) the value of "x" when ere == 0x401 or 0x201 or 0x11
% ; in the main loop, ere == 0x101 ;
% x=35340 ; fprintf('%3d %4x %3x %3x %3x %3x\n',d(x:x+40,:)')
% find( d(1:end-1,2)==17 & diff(d(:,2)) == 0)
%f= find( d(1:end-1,2)==17 & diff(d(:,2)) == 0);
%plot(f)
I'm sure Joel @biomurph will want to upload your code some place on our Github. I think you can do that also and then do a push to our repository.
William
Thank's William, but "it is too early to crow victory"...
To do what I want this code works but since it depends on the context and the type of SD Card, it should be tested carefully...
For exemple I am not sure that 16 Channels / 2kHz would go... For that a binairy format on the SD should be used...
Yannick.
The SD only version of Biomurph uses the code "N" to start running.
In the file "OpenBCI_32_SD_Only.ino" you will find a case 'N' :
In this case I added the initialisation of write and read indexs.
case 'N': writeToSDonly = true;
...
OBCI.read_ix = 0 ;
OBCI.write_ix = 0 ;
break;
Sorry for this oversight.
Yannick.
I sent you a dropbox link.
Tell me if you can use it...
Yannick.