Thomas Fischer

The release of KBibTeX 0.10 is approaching with the release of KBibTeX 0.10-alpha2 aka 0.9.81.

Improvements and changes since 0.9.2 are less on the user interface, but on the code behind it:

Many thanks go to contributors such as bug reporters, translators, and everyone else who contributed to KBibTeX!

Source code is available as tar ball, cryptographically signed using GnuPG key 1808CB466328F4380685A3B1A264FD738D861F41.

KBibTeX is a bibliography editor (BibTeX and somewhat BibLaTex) used in conjunction with LaTeX and friends. Based on this code base, a SailfishOS client called ‘BibSearch’ exists which allows to search for bibliographic data in various online sources (IEEE Xplore, Google Scholar, ACM Digital Library, …). BibSearch's code makes use of KBibTeX's C++ code, has its user interface implemented in SailfishOS's Silica QML, and provides just two C++ files on its own to glue together everything.

It has long been my goal to provide a QML or Kirigami-based client of KBibTeX of similar functionality, either to be used as Plasma widget or for Plasma Mobile (thus equivalent to the SailfishOS implementation). Finally I have been able to put together an initial implementation using Kirigami2. There are several rough edges and bugs, and much of the intended functionality is missing, but at least one can search for bibliographic data. If you are familiar with Kirigami, it should be a low-hanging fruit to fix the obvious issues, especially given that there is already a Silica-based client as reference. Code contributions to branch feature/kirigami are welcome!

So far, most of my blog postings that appeared on Planet KDE were release announcements for KBibTeX. Still, I had always planned to write more about what happens on the development side of KBibTeX. Well, here comes my first try to shed light on KBibTeX's internal working …

Active development of KBibTeX happens in its master branch. There are other branches created from time to time, mostly for bug fixing, i. e. allowing bug reporters to compile and test a bug fix before before the change is merged into master or a release branch. Speaking of release branches, those get forked from master every one to three years. At the time of writing, the most recent release branch is kbibtex/0.9. Actual releases, including alpha or beta releases, are tagged on those release branches.

KBibTeX is developed on Linux; personally I use the master branch on Gentoo Linux and Arch Linux. KBibTeX compiles and runs on Windows with the help of Craft (master better than kbibtex/0.9). It is on my mental TODO list to configure a free Windows-based continuous integration service to build binary packages and installers for Windows; suggestions and support are welcome. Craft supports macOS, too, to some extend as well, so I gave KBibTeX a shot on this operating system (I happen to have access to an old Mac from time to time). Running Craft and installing packages caused some trouble, as macOS is the least tested platform for Craft. Also, it seems to be more difficult to find documentation on how to solve compilation or linking problems on macOS than it is for Windows (let alone Linux). However, with the help of the residents in #kde-craft and related IRC channels, I was eventually able to start compiling KBibTeX on macOS (big thanks!).

The main issue that came up when crafting KBibTeX on macOS was the problem of linking against ICU (International Components for Unicode). This library is shipped on macOS as it is used in many other projects, but seemingly even if you install Xcode, you don't get any headers or other development files. Installing a different ICU version via Craft doesn't seem to work either. However, I am no macOS expert, so I may have gotten the details wrong …

Discussing in Craft's IRC channel how to get KBibTeX installed on macOS despite its dependency on ICU, I got asked why KBibTeX needs to use ICU in the first place, given that Qt ships QTextCodec which covers most text encoding needs. My particular need is to transliterate a given Unicode text like ‘äåツ’ into a 7-bit ASCII representation. This is used among others to rewrite identifiers for BibTeX entries from whatever the user wrote or an imported BibTeX file contained to an as close as possible 7-bit ASCII representation (which is usually the lowest common denominator supported on all systems) in order to reduce issues if the file is fed into an ancient bibtex or shared with people using a different encoding or keyboard layout.

Such a transliteration is also useful in other scenarios such as if filenames are supposed to be based on a person's name but still must be transcribed into ASCII to be accessible on any filesystem and for any user irrespective of keyboard layout. For example, if a filename needs to have some resemblance the Scandinavian name ‘Ångström’, the name's transliteration could be ‘Angstrom’, thus a file could be named Angstrom.txt.

So, if ICU is not available, what are the alternatives? Before I adopted ICU for the transliteration task, I had used iconv. Now, my first plan to avoid hard-depending on ICU was to test for both ICU and iconv during the configuration phase (i. e. when cmake runs) and use ICU if available and fall back to iconv if no ICU was available. Depending on the chosen alternative, paths and defines (to enable or disable specific code via #ifdefs) were set.
See commit 2726f14ee9afd525c4b4998c2497ca34d30d4d9f for the implementation.

However, using iconv has some disadvantages which motivated my original move to ICU:

1. There are different iconv implementations out there and not all support transliteration.
2. The result of a transliteration may depend on the current locale. For example, ‘ä’ may get transliterated to either ‘a’ or ‘ae’.
3. Typical iconv implementations know less Unicode symbols than ICU. Results are acceptable for European or Latin-based scripts, but for everything else you far too often get ‘?’ back.

Is there a third option? Actually, yes. Qt's Unicode code supports only the first 2¹⁶ symbols anyway, so it is technically feasible to maintain a mapping from Unicode character (essentially a number between 0 and 65535) to a short ASCII string like AE for ‘Æ’ (0x00C6). This mapping can be built offline with the help of a small program that does link against ICU, queries this library for a transliteration for every Unicode code point from 0 to 65535, and prints out a C/C++ source code fragment containing the mapping (almost like in the good old days with X PixMaps). This source code fragment can be included into KBibTeX to enable transliteration without requiring/depending on either ICU or iconv on the machines where KBibTeX is compiled or run. Disadvantages include the need to drag along this mapping as well as to updated it from time to time in order to keep up with updates in ICU's own transliteration mappings.
See commit 82e15e3e2856317bde0471836143e6971ef260a9 where the mapping got introduced as the third option.

The solution I eventually settled with is to still test for ICU during the configuration phase and make use of it in KBibTeX as I did before. However, in case no ICU is available, the offline-generated mapping will be used to offer essentially the same functionality. Switching between both alternatives is a compile-time thing, both code paths are separated by #ifdefs.

Support for iconv has been dropped as it became the least complete solution (see commit 47485312293de32595146637c96784f83f01111e).

Now, how does this generated mapping look like? In order to minimize the data structure's size I came up with the following approach: First, there is a string called const char *unidecode_text that contains any occurring plain ASCII representation once, for example only one single a that can be used for ‘a’, ‘ä’, ‘å’, etc. This string is about 28800 characters long for 65536 Unicode code points where a code point's ASCII representation may be several characters long. So, quite efficient.

Second, there is an array const unsigned int unidecode_pos[] that holds a number for every of the 65536 Unicode code points. Each number contains both a position and a length telling which substring to extract from unidecode_text to get the ASCII representation. As the observed ASCII representations' lengths never exceed 31, the array's unsigned ints contain the representations' lengths in their lower (least significant) five bits, the remaining more significant bits contain the positions. For example, to get the ASCII representation for ‘Ä’, use the following approach:

const char16_t unicode = 0x00C4; ///< 'A' with two dots above (diaeresis)
const int pos = unidecode_pos[unicode] >> 5;
const int len = unidecode_pos[unicode] & 31;
const char *ascii = strndup(unidecode_text + pos, len);

If you want to create a QString object, use this instead of the last line above:

const QString ascii = QString::fromLatin1(unidecode_text + pos, len);

If you would go through this code step-by-step with a debugger, you would see that unidecode_pos[unicode] has value 876481 (this value may change if the generated source code changes). Thus, pos becomes 27390 and len becomes 1. Indeed and not surprisingly, in unidecode_text at this position is the character A. BTW, value 876481 is not just used for ‘Ä’, but also for ‘À’ or ‘Â’, for example.

Above solution can be easily adjusted to work with plain C99 or modern C++. It is in no way specific to Qt or KDE, so it should be possible to use it as a potential solution to musl (a libc implementation) to implement a //TRANSLIT feature in their iconv implementation (I have not checked their code if that is possible at all).

I am glad to announce the availability of KBibTeX 0.9 Beta 2 (0.8.91) for download. Whereas Beta 1 had some issues and was never formally announced, Beta 2 is quite stable and ready to use for everyone able to compile and install from a tar-ball and willing to test code.

The source code in tar-ball form is available at KDE's download infrastructure:
https://download.kde.org/unstable/KBibTeX/kbibtex-0.8.91.tar.xz
Signed with GnuPG:
https://download.kde.org/unstable/KBibTeX/kbibtex-0.8.91.tar.xz.asc (key: 1808CB466328F4380685A3B1A264FD738D861F41)

Unless there are severe issues, I plan to make a final release of 0.9 in early summer.

KBibTeX towards 0.10

The current master code contains a number of changes due for 0.10, such as refactored preferences/settings system and better integration to GitLab's CI system both to run automated tests as well as to run a Coverity Scan, among others. If you want to test some bleeding edge, pull the code from KDE's Git repository.

A common problem with bug reports received for KBibTeX is that the issue may already be fixed in the latest master in Git or that I can provide a fix which gets submitted to Git but then needs to be tested by the original bug reporter to verify that the issue has been indeed fixed for good.

For many distributions, no ‘Git builds’ are available (or the bug reporter does not know if they exist or how to get them installed) or the bug reporter does not know how to fetch the source code, compile it, and run KBibTeX, despite the (somewhat too technical) documentation.

Therefore, I wrote a Bash script called run-kbibtex.sh which performs all the necessary (well, most) steps to get from zero to a running KBibTeX. The nicest thing is that all files (cloned Git repo, compiled and installed KBibTeX) are placed inside /tmp which means no root or sudo are required, nor are any permanent modifications made to the user's system.

There is a README.txt file explaining the script in greater detail.

The only requirement is that the user installs the usual KDE-related development tools and libraries. If a tool or library is missing, the script will abort, but the error message (most likely some output from cmake) can be searched for in order to learn which package to install. Once this is done, simply restart run-kbibtex.sh until all steps succeed.

I have tested the script with several Linux distributions and gave earlier versions to bug reporters for testing, so I am almost sure that it will work as promised. Please send suggestions or bug reports via email to me.

As promised, here comes the one intermediate pre-release between KBibTeX 0.7.90 (0.8-beta1) and the final release of 0.8: KBibTeX 0.7.95 aka 0.8-rc1. The most important changes between Beta1 and RC1 are the following:

• Update of the ChangeLog (alternatively here) to reflect recent releases' changes,
• Removal of ISBNdb as it is no longer a free service, and
• Various translation updates.

The release is available on KDE's mirror infrastructure:

After a beta version in September and a release candidate in October, there is finally a release of KBibTeX 0.7.
A tag has been set and tar balls have been published.

The only changes compared to the release candidate are attempts to fix online search issues with Google Scholar and IEEE Xplore.

After some delay, I am finally pushing forward towards a final release of KBibTeX for KDE 4. The first step is the tagging and releasing of tar balls for version 0.7's Beta 1.

After quite some delay, I finally assembled a second release candidate for KBibTeX 0.6.1. Version 0.6.1 will be the last release in the 0.6.x series.

The following changes were applied since the release of 0.6:

Update: In an earlier version of this posting, the title called this release ‘0.6.1-alpha1’. Of course, it should have been ‘0.6.1-beta1’ as discussed in the text.

Quick update on the next bugfix release for the KBibTeX 0.6 series: KBibTeX 0.6.1-beta1 (0.6.0.90) has just been released. Differences to the alpha version from two weeks ago are two minor bug fixes and updated translations for Galician and Italian (thanks!).