Tips from the VFS files

The good thing about the VFS (Virtual File System) APIs in Palm OS are that they work just like a traditional desktop-style file system, so they are usually fast to learn and add to your application's code. The bad thing is that many developers don't realize that the best practices for using the VFS APIs differ both from Palm OS database approach, as well as from traditional file system approaches. You can certainly use them under the database or desktop models, but with a little reflection you might well improve your application's UI, usability and/or speed.

One of the most obvious differences is that reading from external storage is considerably slower than reading from internal RAM, which is what you do when using Palm OS Databases. If your application was first created under the assumption that reading stored data was effectively instantaneous, you'll probably have already noticed that random access through one or more large VFS-resident files results in a, shall we say, suboptimal user experience. Therefore, consider going directly against your well-honed memory conservation instincts and try caching data in RAM - perhaps even large quantities of data, for extended periods of time. Nearly all devices that have VFS slots also have a megabyte or more of dynamic heap, so unless you're already putting that space to good use, try caching it there. Alternatively investigate reading it in to the storage heap: most users have lots of space left there for you to use either Feature Pointers or else a temporary database. If your files are normal Palm OS databases and they're only on the card for convenience purposes, consider caching the entire database in RAM while you're running ? but make sure you delete the cache both when you quit as well as after a reset in case you crash (no offense implied!) or the user hits the reset button for some reason.

Another big difference between in-RAM databases and files is that simply finding your data takes more time. DmGetNextDatabaseByTypeCreator is really fast, but doing the simple equivalent with a file (assuming you haven't a unique file extension you're looking for) requires that you open each file and read the first 64 bytes. If you're looking within the /palm/launcher directory, your performance will drop the more applications the user installs on their card. This is a good time to remember that the user normally cares about perceived performance, not actual performance. Therefore if you really do need to scan every file in /palm/launcher to find your data, don't do that when the application launches but instead generate a list of files and then check them one by one during nilEvents. For example if you're showing thumbnails of pictures, get a directory listing and draw the first one's thumbnail immediately, but then return to your main event loop and have it call EvtGetEvent with a very short timeout. When you get a nilEvent, read the next file and draw its thumbnail, and repeat. (When you're done, make sure to change the timeout to be unlimited to save battery.) The total time spent will be about the same, but the user will not have to wait for it all to finish. Even better, if you don't expect the data to change often, cache the thumbnails along with their filenames and modification dates so if it looks like nothing has changed, then you can draw that first display before you open up a single file. Of course you'll need to adjust the rest of your application to know about these tricks, and ensure the current data has been read in at some point, but the speedups can be truly dramatic.

When designing a system to use VFS, you need to decide whether to keep your data in Palm OS database formats (i.e. PRC or PDB files) or in some other traditional file format (e.g. a .TXT file). Your users will probably appreciate being able to install large data files either on a card or internally, which is a vote for using databases in both places. There are utility VFS routines to read the contents of a record or resource from a card-resident PDB, but writing is problematic in such a condition because changing the size of a record or resource can mean that nearly all the file would have to be rewritten. (Consider reading the entire database into RAM and then saving the whole thing again with VFSExportDatabaseToFile, or else parsing the file yourself if performance requires it. The PDB/PRC format is well documented.) On the other hand, you might not be in control of the data files, if you're doing a photo viewer as described above you have to use the same format as the camera did. In that case your task is determining whether to also support saving the photos in RAM or not, in a different container.

If you're coming from a traditional desktop file system mindset, the biggest differences between that and VFS are in the user experience, but there are a few programming aspects too. The user experience when using VFS will ideally be indistinguishable from using internal RAM, and this is usually achievable with care and by using some of the aforementioned ideas. Entering filenames or using file browsers is natural to techie types like us, and might even be requested by users who don't realize there is a better way, but many superior interfaces recognize that files are a means to an end: the focus should be on working with the user's data, not on managing filenames and directories.

On a desktop system you also don't need to worry much about the user pulling out the hard drive that you're in the middle of using, or installing new ones while you're running. But on handheld devices, that needs to be easy and natural. While you're running, you should register for the sysNotifyVolumeMountedEvent and sysNotifyVolumeUnmountedEvent notifications and smoothly show the additional data or gracefully stop using any open files. Telling the user to put back the card is an option to be avoided if possible, since they might not notice your alert until much later, and then you've got to deal with the possibility of the card (or file) having been modified in some other system in the interim.

Also consider locality of reference when reading files, and the number of bytes you read at a time. Palm OS devices may or may not have an automatic low-level cache for the card, so reading a byte at a time might incur considerable overhead. It may well be better to read say 32k of data into your own cache: try some timing tests for your own scenarios.

Palm OS Databases are fast to search and read, and their characteristics are well suited for handheld use and UI. Files can be just as good if you take the time to provide a great user experience... as well as cache appropriately and work some idle-time sleight of hand.