filesystem as a database using only shell

[tallboy]

technosaurus, no matter how you succeed with your database, it is always a pleasure to read your code!

tallboy

[jpeps]

When you first started the thread I thought you had in mind to use no files at all -that entries would only consist of directories and/or links.

Using flat files to hold data becomes a space problem since every file has a minimum size even if it only contains one character. The block size of the filesystem determines the minimum size requirement -usually at least 4k.

That's just the beginning..think about complexity of memory allocation.

re: planning for ACID compliance. There's the small issue of database integrity. Databases use a complex series of locks, so that the master file maintains integrity while various users are simultaneously reading and writing to it. Issues like that are why they need millions of tests.

For a personal database, there are many ways to create a single file. I have a "MyLibrary" database that creates an XML file which is easily parsed for individual books and their related data.

[technosaurus]

That was the original plan, but then I read about a couple of filesystems that can store small files in the inode table, which most well planned databases should be. Using the files allows it to maintain a history, but if speed and storage is more important than keeping history it can rewrite the files once they grow large enough that they cannot fit in the inode (git also uses the filesystem and can do a similar operation IIRC)

[jpeps]

That was the original plan, but then I read about a couple of filesystems that can store small files in the inode table, which most well planned databases should be.

Wouldn't that involve a separate inode for every process of every user of every file?

[technosaurus]

Wouldn't that involve a separate inode for every process of every user of every file?

Sorry for the delayed reply, this kind of dropped off my radar, but to answer the question, each file gets an inode in the filesystem, but processes typically would only be opening 1 file descriptor (or so) at a time.

This would also be a good way to implement light weight, parse-free configuration.

the c psuedo code would look like

Code: Select all

int get_value(const char *path, void *buf,size_t len){
  int fd = open(path, O_RDONLY);
  if (fd<0) return fd;
  len=read(fd,buf,len);
  close(fd);
  return len; 
}
int set_value(const char *path, void *buf,size_t len){
  int fd = open(path, O_CREAT|O_WRONLY|O_TRUNC);
  if (fd<0) return fd;
  len=write(fd,buf,len);
  close(fd);
  return len; 
}

So rather than having a file with lots of VAR=value,
It would be a directory containing files named VAR with value as the contents.
This could be a simple RGB color for a background, a string for a tooltip or even a struct for a desktop icon.

Now all you have to do is run an inotify watch on the directory in case entries are added/changed/removed and the config program can be completely separated. The total size of config data may be a bit larger, but now you don't have to rewrite/parse an entire configuration file for every single change. You don't need to keep global variables around for holding the values (just read'em when you need'em) If you are really clever, with a few code changes you can put the type for each variable in the first byte so that it is simple to automatically build a config gui (for example the enum FILENAME would indicate a type 1 and the gui could read the data as a string for a file selector widget)

Edit: Here are some filesystems capable of storing small files with inode data rather than allocating a whole ??kb block
ext4, btrfs, zfs, reiserfs, gpfs