SD card writing speed.
Our maker space has a group working on brain waves. We will likely use OpenBCI eventually. At present we are using NeuroSky more or less just to get our analytic skills in order. I have interfaced one with an Arduino Uno and write the data to a SD card. Issue is I am using the Arduino SD library and it does not write fast enough. Data comes in every 2 ms but it takes the card 20 ms or more to process it. It is almost not a problem but we are rally close to Nyquest problems at 20 ms or 30 ms sampling.
Open BCI plans a SD recorder, how are you folks writing the data fast enough? Over the weekend I will prowl through the code both Arduino and OpenBCI. If anyone has any suggestions I would appreciate them. The cards are capable of read and write at amuch faster rate. I do not know them well it appears as thoughthat may have to change.
Open BCI plans a SD recorder, how are you folks writing the data fast enough? Over the weekend I will prowl through the code both Arduino and OpenBCI. If anyone has any suggestions I would appreciate them. The cards are capable of read and write at amuch faster rate. I do not know them well it appears as thoughthat may have to change.
Comments
I wrote some (much?) of the OpenBCI code on the Sourceforce/Github repository. We/I did not use an SD card in any of our work to date. We've always just streamed data to the PC or done our processing on the Arduino (without logging to SD).
It really takes 20ms to write to the SD on Arduino? Wow. That's unfortunate...though not too surprising given my own experience looking at the Adadruit SD library (which, if I understand correctly, is the basis for most Arduino SD libraries).
The problem that I see is that the file system checks are brutally painful on processing time. I believe that even that early Adafruit SD library recogized this penalty and offered an alternate "raw" writing mode where you can pre-allocate some filespace on the SD card and then write directly (with much lower overhead) to the pre-allocated space. if none of this sounds familiar, google around a bit...or post back here and maybe I can dig out my earlier references.
Either way -- pre-allocated or not -- it ALSO saves time if you buffer up your data so that you write it in something close to an SD packet size. In other words, don't write data in 32 byte units to the SD card....save it up until you can save something closer to a full SD packet/sector size. That'll save you a lot of time.
Thanks so much for being interested in this. I think OpenBCi is a great way to get access to one's own brain...and it excites me when I hear about people (besides myself) digging into this, too. Brain hacking, FTW!
Chip
It looks as though I have some study to do. I may get back but will delve in myself for a while, Data collection speed is an issue as the Nyquist limit starts creeping (galloping) up.
Do you know how fast the Adroids are at capturing data. They are another possible data logger but if their access time is similar nothing is gained. We have a meeting coming up and there are some pretty good Java folks there who do phone app development.
I will keep you posted particularly if we have success!
Tom
I've not yet used a phone to log data. I think that it would work great, though.
Again, in this context, I down think that the relative slowness of SD is due to the card itself...it's due to the Arduino's ability to execute all the software necessary to write to the card. If you used a faster microcontroller (like a Due? or a Raspberry Pi?), it would crank through the SD-servicing routines a lot faster and would probably work great for logging EEG data.
If you do data logging with your phone, be sure to post back here on the forum. I'd love to see your EEG hacking!
Chip
If you write in small chunks you will actually be wearing out your flash drive faster than normal because of all this extra allocation activity.
Thanks. I will spend next week getting more intimate with some sd libraries. I do need to go to the block writing and to a raw write. I found some alliterative libraries for the Arduino that report quite fast writes. I also have a Mega which will up the processor speed.
I have to get this off my desk before I go to the Android as that could be the ultimate time sink, It should be able to record the data on internal memory and then write to a micro sd as appropriate ( or dump via Bluetooth to processing or matlab).
I haven't done any serious coding in 20 over years so this process is not pretty to watch. (It might be funny to watch though).
More later hopefully successes.
Tom
Will get a DUE the Mega has the same clock speed.
Tom
You then need to ask yourself, what are the appropriate times to flush the buffers, say when your data collection is complete or at some clock interval, such as every couple seconds (if the data stream has stopped.)
Thanks. One of the libraries I am looking at essentially sets up an OS which does as you suggest. One thread services the Uart and the other services the sd card. This is new ground for me but the code is well written and I think I can get my way through it (famous last words). I will delve in next week. The OS sets a schedule for the two activities. I do not know if I can do that or if I need to set up some interrupt driven system.
The DUE may not be necessary but I am likely to use it interfacing the ADS1299. The group in South Carolina is using one of the new Teenseys (72MHz) to handle the 1299 data. So I will use there if nowhere else.
Have a good weekend.
Tom
I am starting on the SD card read with the RTOS system. I do not know witch of the variants I will use. There is a NillRTOS that is very small and may work.
At present I am on the hook to get an Arduino interface for a network TV visit related to some "brain" program that they are working on. Do you want me to also refer them to you or do you like quiet> I would be of the latter but it was not my call!
Meeting on Sunday and we will discuss the Android interface.
Tom
I'm coming to this discussion alittle late, but here you go.
There are issues writing to the SD card using the high level SDFat tools, for one.
Also, it make a big difference what kind of card you're using. Best to use one of the newer high-speed cards (ScanDisc is a good one. Look for class 10 or better).
There's an example in the SDFat library that performs BLOCK write for fastest logging in the west! In the example, arduino defines the size of the a continuous file, retrieves the block addresses, and uses the SD card cache as a 512 byte buffer. That last part is key, the data is written once to the cache on the card. Then, when the cache is full, arduino performs a block write of the cached data. The example code outputs to serial a pile of data about write time performance. When I ran the test code, I get an average block write time of ~700uS
Sometimes, there are overruns, but they are in the low mS range. Also, the card should be formatted on a Windows PC, and/or erased with at least security level 1 (which writes '0' to all the bytes).
I modified the example code for logging OpenBCI data format in HEX, and seeded an array of 8 long variables with data
<code>{0xABCDE1,0xABCDE2,0xABCDE3,0xABCDE4,0xABCDE5,0xABCDE6,0xABCDE7,0xABCDE8}</code>
Then, I repeatedly parsed the long array into HEX values and stored them in cache until I got to 512, then wrote the block. Here's the code
<pre><code>/*
*
* testing BLOCK WRITE using the OBCI data format
* first test logging OBCI data in HEX format using ',' separator and '\n' terminator
*
* This sketch simulates logging from a source that produces
* data at a constant rate of one block every MICROS_PER_BLOCK.
*
* This code is written to run on the OpenBCI V3 board.
* Adjust as needed if you are testing on different hardware.
*
* If a high quality SanDisk card is used with this sketch
* no overruns occur and the maximum block write time is
* under 1000 micros.
*
*
*/
#include <SdFat.h>
#include <SdFatUtil.h>
// SD chip select pin
const uint8_t chipSelect = 6;
// number of blocks in the contiguous file
const uint32_t BLOCK_COUNT = 1000UL; // times 512 = size of file to allocate
// time to produce a block of data
const uint32_t MICROS_PER_BLOCK = 2000; // minimum time that won't trigger overrun error
// file system
SdFat sd;
// test file
SdFile file;
// use cout to save memory use pstr to store strings in flash to save RAM
ArduinoOutStream cout(Serial);
// file extent
uint32_t bgnBlock, endBlock;
// init stats
uint16_t overruns = 0;
uint32_t maxWriteTime = 0;
uint16_t minWriteTime = 65000;
uint32_t t = micros();
uint32_t tNext = t;
uint32_t b = 0;
uint8_t* pCache;
const int numChannels = 8;
volatile int8_t currentChannel = 0; // used to remember Channel# on block overflow
volatile int8_t currentNibble = 5; // used to rememver place in HEX conversion on block overflow
volatile int16_t byteCounter = 0; // used to hold position in cache
long channelData; // incoming ADS values
boolean logging;
//
// store error strings in flash to save RAM
#define error(s) sd.errorHalt_P(PSTR(s))
//
// log of first overruns
#define OVER_DIM 20
struct {
uint32_t block; // holds block number that over-wrote
uint32_t micros; // holds the length of this of over-write
} over;
//
void setup(void) {
Serial.begin(9600);
pinMode(10,OUTPUT);
digitalWrite(10,HIGH); // de-select the ADS1299
pinMode(5,OUTPUT);
digitalWrite(5,HIGH); // de-select the LIS3DH
long seed = 0xABCDE1;
for(int c = 0; c < 9; c++){ // seed the channelData array
channelData = seed;
seed++;
}
}
//
void loop(void) {
while (Serial.read() >= 0) {} // clear out the serial buffer
cout << pstr("Type any character to start\n"); // prompt user to begin test
while (Serial.read() <= 0) {}
cout << pstr("Free RAM: ") << FreeRam() << endl;
// initialize the SD card at SPI_FULL_SPEED for best performance.
// try SPI_HALF_SPEED if bus errors occur.
if (!sd.begin(chipSelect, SPI_FULL_SPEED)) sd.initErrorHalt();
// delete possible existing file
sd.remove("SDtime.TXT"); // file name must be 8.3 format
// create a contiguous file
if (!file.createContiguous(sd.vwd(), "SDtime.TXT", 512UL*BLOCK_COUNT)) {
error("createContiguous failed");
}
// get the location of the file's blocks
if (!file.contiguousRange(&bgnBlock, &endBlock)) {
error("contiguousRange failed");
}
//*********************NOTE**************************************
// NO SdFile calls are allowed while cache is used for raw writes
//***************************************************************
// clear the cache and use it as a 512 byte buffer
pCache = (uint8_t*)sd.vol()->cacheClear();
cout << pstr("Start raw write of ") << file.fileSize() << pstr(" bytes\n");
cout << 512/56 << pstr(" OBCI data packets per block\n");
cout << pstr("Please wait ~") << ((BLOCK_COUNT*(512/56))*4000UL)/1000000UL;
cout << pstr(" seconds\n");
// tell card to setup for multiple block write with pre-erase
if (!sd.card()->erase(bgnBlock, endBlock)) error("card.erase failed");
if (!sd.card()->writeStart(bgnBlock, BLOCK_COUNT)) {
error("writeStart failed");
}
// initialize stats
overruns = 0; // number of overruns
maxWriteTime = 0; // longest block write time
minWriteTime = 65000; // shortest block write time
b = 0; // block counter
t = micros(); // note the time
logging = true;
while(logging){
// convert 24 bit number into HEX. 0000|0000 0000|0000 0000|0000
for (currentChannel = 0; currentChannel < 8; currentChannel++){
for (currentNibble = 5; currentNibble >= 0; currentNibble--){
byte nibble = (channelData >> currentNibble*4) & 0x0F;
if (nibble > 9){
nibble += 55; // convert to ASCII A-F
}else{
nibble += 48; // convert to ASCII 0-9
}
pCache[byteCounter] = nibble;
byteCounter++;
if(byteCounter == 512){
overRun();
}
}
if(currentChannel < 7){
pCache[byteCounter] = ',';
byteCounter++;
if(byteCounter == 512){
overRun();
}
}
}
// end of nested loops
pCache[byteCounter] = '\n';
byteCounter++;
if(byteCounter == 512){
overRun();
}
delayMicroseconds(4000); // delay to simulate 250SPS data rate
}
// total write time
t = micros() - t;
// end multiple block write mode
if (!sd.card()->writeStop()) error("writeStop failed");
cout << pstr("Done\n");
cout << pstr("Elapsed time: ") << setprecision(3)<< 1.e-6*t;
cout << pstr(" seconds\n");
cout << pstr("Max write time: ") << maxWriteTime << pstr(" micros\n");
cout << pstr("Min write time: ") << minWriteTime << pstr(" micros\n");
cout << pstr("Overruns: ") << overruns << endl;
if (overruns) {
uint8_t n = overruns > OVER_DIM ? OVER_DIM : overruns;
cout << pstr("fileBlock,micros") << endl;
for (uint8_t i = 0; i < n; i++) {
cout << over.block << ',' << over.micros << endl;
}
}
// close file for next pass of loop
file.close();
Serial.println();
}
void overRun(){
// write the 512 byte block
// Serial.println("writing BLOCK");
uint32_t tw = micros(); // time the write
if (!sd.card()->writeData(pCache)) error("writeData failed");
tw = micros() - tw;
if (tw > maxWriteTime) { // check for max write time
maxWriteTime = tw;
}
if (tw < minWriteTime){ // check for min write time
minWriteTime = tw;
}
// check for overrun
if (tw > MICROS_PER_BLOCK) {
if (overruns < OVER_DIM) {
over.block = b;
over.micros = tw;
}
overruns++;
}
byteCounter = 0; // reset 512 byte counter
b++; // increment BLOCK counter
if(b == BLOCK_COUNT){logging = false;} // we did it!
// write must be done by this time
}</code></pre>
Here is an example of the serial output
<pre><code>Type any character to start
Free RAM: 937
Start raw write of 512000 bytes
9 OBCI data packets per block
Please wait ~36 seconds
Done
Elapsed time: 41.730 seconds
Max write time: 716 micros
Min write time: 704 micros
Overruns: 0
Type any character to start
Free RAM: 937
Start raw write of 512000 bytes
9 OBCI data packets per block
Please wait ~36 seconds
Done
Elapsed time: 41.730 seconds
Max write time: 716 micros
Min write time: 704 micros
Overruns: 0
Type any character to start
</code></pre>
http://openbci.com/forum/index.php?p=/discussion/106/sd-card-writing-speed
Tom, Chip, and Joel all made interesting posts there. You may want to PM Tom and ask him where he is at with his project. Just click on his name there and you can use the Message button to PM him.
William