FAT in a fortnight
My GitHub account has been busy lately, thanks to not just one but two great open source–supporting clients!
Storage drivers for cutting-edge hardware
Top prize goes to Technical Machine, whose ambitious Tessel hardware platform is about to be released! Thanks to their sponsorship, the following modules have been published:
fatfs — you read the FAT spec, and it looks pretty simple, and it is. But turns out wrapping some linked lists, fussily packed bitfields, and some rules for mapping "most any Unicode" to "11 bytes of ASCII"— taking those primitives and building something that's maintainable and acts just like an asynchronous JavaScript interface to POSIX file APIs (and doesn't make mistakes!) is a bit of a project.
tessel-sdcard — this was also no small feat, the SD Card SPI protocol is pretty reasonable and also well documented, but working through all the nuances of card initialization and reliable data transfers wasn't trivial when all the other "working examples" were blocking C libraries that didn't have to worry much about concurrent SPI bus traffic.
fifolock — speaking of concurrency, at one point I implemented a small/quick "serial queue" function to the SD Card library to help keep card communications sane. Then that needed a sort of "automatic transactions" wrapper so the code using the queue would stay clean. There it sat, until I also needed the same thing in fatfs, too. Things got a little out of hand and I ended up putting in some of my own time to polish it up to my personal liking — while keeping it at exactly 42 lines of code [well, including the trailing newline. why am I even explaining/admitting this…?] — and the logic is now published as its own module.
struct-fu — strangely, this one might be my favorite? There were already a handful of ctypes/struct handling/buffer parsing libraries in the npmjs repository, but in a previous project I had started dabbling with a clean and promising "C-like struct declaration" interface design. More intentionally than "fifolock", I invested some of my own time into this too, but it grew alongside the needs of the other libraries and is serving them well. I'm happy with how it turned out; using "struct-fu" for buffer parsing gives me the same feeling I get when using d3 for DOM manipulation or fermata for HTTP — exposing plenty of low-level control, yet not burdening its user with many fussy responsibilities.
parsetition — on the other end of the spectrum, this is at the moment the weakest link, the least complete and mentioned with the least excitement. Basically, there are a variety of ways to partition a drive to make it host a FAT partition. Handling just a cursory subset needed to support the formatted "starter card" that Tessel ships with, right now "parsetition" is mostly an architectural placeholder written in the wee hours of the morning to bridge the gap between the "sdcard" and "fatfs" modules.
The Tessel platform is a big deal, and has an incredible team behind it. Due to my early work porting node-nrf to their platform, I've gotten to watch their hardware, software, tools and documentation get steadily more refined. I'm not sure people realize just how much the Technical Machine team has accomplished in the last year. This isn't a cheaper Arduino, or a smaller Arduino, or an Arduino-with-some-extra-shield-built-in. It's not an Arduino at all, but I don't think it will suffer from being "too different" or too vendor tied either — it builds on, participates in, and contributes to the most interesting and valuable things people are doing with node.js. I'm grateful I could help out with a relatively small (seriously!) piece of their platform…
Mobile demos for cutting-edge storage
A hearty honorable mention to Couchbase as well! I had the privilege to make a cameo appearance at HTML5 DevConf — remotely, using a demo app I wrote for both iOS and Android! CouchTalk shows how you might enable "local islands of connectivity" that still share data with each other in near-realtime through a remote server. It was a treat to take their team's "push-to-talk web chat room" demo and run with it, updating the browser logic to work with Couchbase Lite and filling out the mobile interfaces to make the Sync Gateway features a bit more visible.
We kept the demo simple, but it's clear that Couchbase has solved a lot of the problems I'd been having with actually deploying related databases in an offline-friendly multiuser architecture. [That's quite another blog post…] I hope the demo inspires others to try out Sync Gateway and Couchbase Lite (née TouchDB) for their mobile apps — I know it's gotten me interested in that space again. It was not only a fun project, but I'm glad they found a way to get me to finally try out their storage stack in earnest ;-)
One more thing…
In the midst of all this, I also deployed a major update to the eReader/workbook webapp that's been going through classroom trials (closed source, but lovingly handcrafted with HTML5/D3/CouchDB/node.js etc.) and somehow managed to sneak in a last-minute trip to the Bay Area so as not to miss Edward Tufte, Bret Victor, Mike Bostock and Jonathan Corum all in the same room! It's been kind of frantic, and as a result I've been about a week later getting to each of the projects' "wrap up" stage than I had hoped to on my side.
Because meanwhile, my wife has been working on an even more significant deliverable, which I might add, is also running about a week behind schedule…