Simple key-value store

Sometimes you just need to store a lot of key-value data, e.g., for caching purposes. There are many solutions available that can serve this purpose, such as memcached, LevelDB, Redis, or just about any database.

If you want something simple that does not introduce new dependencies, Python has the shelve module, which is backed by the dbm Unix database module.

A number of years ago Seb Sauvage created dbdict which is like shelve but backed by SQLite. I have updated Seb's code to Python 3, and made the project available on Github.

With dbdict you can't directly store Python objects. Only numbers, strings and binary data. Objects need to be serialized first in order to be stored. Use e.g. pickle, json (or simplejson) or yaml for that purpose. For comparisson, the shelve module transparently uses pickle to serialise Python objects.

The advantages of dbdict are as quoted from the README:

Python dictionaries are very efficient objects for fast data access. But when data is too large to fit in memory, you want to keep data on disk but available for fast random access.

Here's a dictionary-like object which uses a SQLite database backend for random access to the dictionary's key-value pairs:

You can work on datasets which do not fit in memory. Size is not limited by memory, but by disk. Can hold up to several tera-bytes of data (thanks to SQLite).

Behaves like a dictionary (can be used in place of a dictionary object in many cases) for storing numbers, strings and binary data.

Data persists between program runs, and is written to disk immediately when inserting a key-value pair in the dictionary.

ACID (data integrity): Storage file integrity is assured. No half-written data. It's really hard to mess up data.

Efficient: You do not have to re-write a whole database file when changing only one item. Only the relevant parts of the file are changed.

You can mix several key types (you can do d['foo']=bar and d[7]=5468) as with a regular dict object.

You can share this dictionary with other languages and systems (SQLite databases are portable, and the SQlite library is available on a wide range of systems/languages, from mainframes to PDA/iPhone, from Python to Java, C/C++, C#, Perl etc.)

My implementation extends Seb's dbdict with the following bells and whistles:

Use of iterators to reduce memory usage.
A vacuum function, for freeing disk space on-demand.
Automatically free disk space when dictionary is cleared
Re-indexing function, for use after many inserts and deletes.

So, if you are in need of storing many/large key-value pairs, consider using my implementation of dbdict.

Install using pip:

python3 -m pip install https://github.com/nephics/dbdict/archive/main.zip

Use it like a standard dictionary, except that you give it a name (eg.'tempdict'):

from dbdict import dbdict
d = dbdict('tempdict')
d['foo'] = 'bar'
# At this point, the key value pair foo and bar is written to disk.
d['John'] = 'doh!'
d['pi'] = 3.999
d['pi'] = 3.14159  # replaces the previous version of pi
d['pi'] += 1
d.close()    # close the database file

Oct 30, 2020