I'm in the process of planning a NNTP download robot for KDE 4.x. The plan is for it to have feature parity with NewsLeecher, which I've been using for quite a while. Unfortunately, NewsLeecher has performance issues.
What I'm planning will not be simply an NZB downloader, but a fully fledged NNTP client that stores headers locally and allows local full-text searching. However, some newsgroups can have header counts reaching into the hundreds of millions and sometimes further - I've seen a header count of over 3 billion for one particular newsgroup.
So what I'm looking for is an embeddable database capable of handling that many records, with stability and a decent level of performance. I've been looking at NoSQL databases, but I'm not sure whether key/value or document databases would be a better fit.
Or should I tie the project directly into Akonadi and use that to index the headers instead? Akonadi currently uses MySQL as a storage backend, and I do not know how this will perform on the average modern desktop with billion-record tables.
Any advice is appreciated.
What I'm planning will not be simply an NZB downloader, but a fully fledged NNTP client that stores headers locally and allows local full-text searching. However, some newsgroups can have header counts reaching into the hundreds of millions and sometimes further - I've seen a header count of over 3 billion for one particular newsgroup.
So what I'm looking for is an embeddable database capable of handling that many records, with stability and a decent level of performance. I've been looking at NoSQL databases, but I'm not sure whether key/value or document databases would be a better fit.
Or should I tie the project directly into Akonadi and use that to index the headers instead? Akonadi currently uses MySQL as a storage backend, and I do not know how this will perform on the average modern desktop with billion-record tables.
Any advice is appreciated.