All posts by Ian Donnelly

How-To: Write a Plug-in

Hi Everybody!

I wanted to write a how-to on how to write an Elektra plug-in. Plug-ins are what allow Elektra to translate regular configuration files into the Elektra key database, and allow keys to be coverted back into regular configuration files. For example, the hosts plug-in is used to transcribe a hosts file into a meaningful KeySet in the Elektra Key Database. This plugin is what allows the kdb tool to be able to work with hosts files like in our mount tutorial.

While there are already a good number of plug-ins written for Elektra, we obviously don’t cover all the different types of configuration files. If you want to be able to use the features of Elektra, you may have to write a plug-in for your type of configuration file. Luckily, its not hard to do. Over the new few weeks I will be writing a tutorial for how to write your very own plug-in as well as explaining all the components of an Elektra plug-in. I will link the tutorials below as I finish them for easy reference, or if you already follow my blog I am sure you will notice them as they get published.

Hello Planet Debian!

Hi Planet Debian!

My name is Ian Donnelly. I am going into my Senior Year at Florida State University this fall but for the summer I am working on a Google Summer of Code project with Debian. My mentor is Markus Raab and I am working with the Elektra team to implement a great new feature for Debian! I am spending this summer working on a system to allow automatic conffile merging by utilizing the Elektra Framework.

Elektra is designed to provide “a universal and secure framework to store configuration parameters in a global, hierarchical key database.” For this project, conffiles will be mounted to the Elektra framework as KeySets. I have written code that performs a three-way merge on these KeySets which is much more robust than merging individual lines. A conffile can easily be mounted to the Key database using the kdb mount command, which I detailed in a previous blog post. Elektra has many included plugins which allow all types of conffiles to be mounted to the Key database. With our newest release, we even added an augeas plugin which will provide compatibility for most of the major conffiles. Once a conffile is mounted, any changes to the Elektra KeySet it is contained in will be updated in the actual conffile and visa-versa.

So far this summer, I have written code for a three-way merge of Elektra KeySets and we have begun to look at the next step of implementing a automatic merge into dpkg. We are currently working on identifying exactly how the merge should take place and where we need to modify dpkg. The basic idea of what will change is that:

  • Dpkg will need to store base conffiles somewhere on the system.
  • Conffiles will need to be edited to have options ready for mounting such as where the keys should be mounted (for a package to support the automatic merge).
  • Mount conffiles during package installs.
  • Attempt a three-way merge during a package upgrade (instead of showing the user a prompt)
  • Fall back to the regular prompt if a conffile can’t be merged without conflicts.

Once dpkg has been patched, we will modify some packages and thoroughly test the new merge operation. It is designed to be non-destructive and to not infringe with any of the advantages of the current system. Package maintainers would need to opt in to using this feature to insure that all packages get tested before using it. However, we have designed this system to require minimal modification in order for maintainers to utilize it.

This project will benefit Debian by removing a large pain-point with users. For experienced uses, merging changes in conffiles can be  a tedious process. For newer users, the current system can be downright confusing. The goal of our project is to make Debian easier to use for everybody while not loosing the features that make it such a powerful platform. Please let us know what you think of the project or if you have any questions.

Sincerely,
Ian S. Donnelly

How-To: kdb mount

I would like to go over the kdb mount command. The mount command is used to mount configuration files to the Elektra Key Database. Once a file is mounted, any changes to the file will be reflected in the Elektra Keys and any changes to the keys will be reflected in the configuration file. So how do we use the mount command?

kdb mount [filepath] [mountpoint] [plugin]

For instance, to mount /etc/hosts to system/hosts (using the hosts plugin):
kdb mount /etc/hosts system/hosts hosts
At this point the hosts file has been mounted. On my system the following key now exist under system/hosts

system/hosts/ian-debian

Which makes sense as the only line in my hosts file is:

127.0.0.1 ian-debian

Now I can edit this configuration file through the Elektra API, the kdb tool, or by editing the file (like always). So if I make a change using kdb such as:

kdb set system/hosts/ian-debian 192.168.0.2

My hosts file will be updated accordingly:

192.168.0.2 ian-debian

So that’s how kdb mount works!

Update: For the sake of clarity, I just want to add a quick example of how to unmount a configuration file from Elektra. The command is:

kdb umount [backend]

For our above example:

kdb umount system_hosts

Week 4: New Release Work

Hi All,

Team Elektra recently released version 0.8.6 to download, more information on that here: http://sourceforge.net/p/registry/mailman/message/32418614/

However, since I am working with Debian, and the Debian package isn’t ready yet I have spent a large chunk of the week working on that. Namely I fixed a few minor lintian errors and worked on packaging the lua and python bindings for Elektra. These two packages, libelektra-python4 and libelektra-lua4 uses a tool called SWIG. SWIG allows C and C++ programs to connect to programs written in higher-level languages such as Python. So these two packages use the power of SWIG to bring the features of the Elektra API (which is written in C) to Python and Lua developers. I personally think it is a really neat feature and allows Elektra to be used more widely than ever before. As our project develops Elektra will support more languages and we are even moving to allowing Developers to write Elektra plugins using other languages.

Additionally, I worked on improving the MergeTools library for Elektra. This library just contains the necessary functions to allow developers to complete a three-way merge of KeySets. This library also serves as the backbone of the kdb merge command. First, the KeySetMerge command now returns an empty KeySet if its unsuccessful. I added operator overloads for == and != to the KeySet.hpp class so developers can tell if the returned keyset is empty. I created a new function that takes in 3 KeySets, their parent keys (for the purposes of merging), and a merge_root key which will be where the merged keys are stored under. The prototype for this function is KeySet(KeySet base, Key base_root, KeySet ours, Key our_root, KeySet theirs, Key their_root, Key merge_root). A function still exists that just takes in 3 KeySets and a key, merge_root. That function attempts to find each root key and then calls the previously mentioned function. Why?

This week I finalized the kdb merge command (not including options). This command takes in 4 root keys, one for base, one for ours, one for theirs, and one for where the merged keys should be stored. The merge command now uses the new function to be able to handle more complex KeySets and so far it works great! In this case we don’t need to make any assumptions about the KeySet since the user needs to pass in the root keys (this is how the kdb commands typically work) so we make good use of this information. Now the kdb merge command only appends the merge KeySet to the database upon successful merge, otherwise it tells you what went wrong.

Also, looking into the main feature of my GSoC project, automatic conffile merging, it seems that it will require patching dpkg. Originally I though package maintainers would have to edit their packages to not define their conffiles as such in order to use the automatic merge. However this is obviously a messy workaround. Patching dpkg however would allow very clean integration of this feature. If this project is successful and the patch makes its way upstream, all Debian packages that utilize conffiles (most of them) will basically allow automatic merging practically overnight. dpkg would attempt a merge on the conffiles during an upgrade and if unsuccessful would just prompt the user like it does now. In most situations however, because of the features of Elektra, conffiles should merge fine. This would make Debian much more user-friendly and could save users a lot of time manually merging conffiles.

Oh, and look for a little tutorial and explanation of the kdb mount command in the next few days. I will post it here.

Until Then,
Ian S. Donnelly

Week 3: Learning Debian Packaging

This was another good week. This week I focused on cleaning up the merge implementation as well as learning about Debian packaging. As far as cleaning up the implementation, I got a few tests written to test the merge function. Additionally I rewrote the merge command code (the code responsible for the kdb merge command, it depends on the implementation of merge I wrote last week). Now the kdb merge command takes in a parent key for each keyset and creates a keyset containing all the keys under the parent key. It then invokes the KeysetMerge function I wrote on those keysets and saves the returned keyset to the key database.

Most of the week I spent learning about Debian packaging. I was able to get some hand-on experience by creating a package for the augeas plugin for Elektra. I modified the control file to include the libaugeas0 dependency and adding in a description of the plugin. Also I created a new libelektra-augeas4.install file to install the share library to the correct location on the system. Then, I tried building the source using git-buildpackage but unfortunately it failed on packaging the plugin because the libelektra-augeas.so file was never built. It turned out that the augeas plugin had to be added to ElektraCache.cmake under plugins_dep (plugins with dependencies) so that the plugin would be built on make. So I fixed that and ran buildpackage again and voila! It worked! I successfully make my first Debian package!

The other thing I looked into was how configuration files get updated during a package upgrade under Debian. It turns out, most package define their configurations as “conffiles”. These conffiles do not get overwritten during package upgrade if the user has edited them. Basically, during an upgrade, dpkg calculates a checksum on each conffile and if it has changed, prompts the user what to do. Instead, the goal of our project is to attempt an automatic merge of the conffile and if successful, used the merge version. If the merge is unsuccessful dpkg will prompt the user as usual. I need to figure out how to change dpkg’s behavior during an upgrade with a conffile so I can build a prototype of a package upgrade merging a file.

Lastly, this week I looked into adding tab completion to the kdb tool. Currently kdb does not try to complete commands using tab completion in bash. It turns out adding tab completion isn’t that difficult. You basically just need to write rules for how tab completion should work for your program and add it to “/etc/bash_completion.d/” I wrote a basic tab completion script for kdb that will complete the commands for kdb. It follows here:

#Proof of concept bash completion for kdb. You must copy this file to /etc/bash_completion.d/kdb and load it with ". /etc/bash_completion.d/kdb"
_kdb()
{
local cur prev opts
COMPREPLY=()
cur="${COMP_WORDS[COMP_CWORD]}"
prev="${COMP_WORDS[COMP_CWORD-1]}"
opts="check convert cp export file fstab get getmeta import info ls lsmeta merge mount mv rm set setmeta sget shell test umount vset"

if [[ ${cur} == * ]] ; then
COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
return 0
fi
}
complete -F _kdb kdb

Its just a proof on concept but its a really neat feature and surprisingly easy to implement. One of the other students working on Elektra, Felix, is also working on tab completion. I think we will collaborate and have this feature implemented for kdb very soon.

For next week I have a few goals I would like to accomplish. First of all I will have a working prototype of a package being upgraded with automatic conffile merging using kdb merge. Second, I want to continue working on the merge implementation to make it cleaner and work on more sophisticated keysets. I also need to add better error handling to tell if a merge was successful or not, and if not why. Third, I want to work on some of the lintan errors for the deb package of Elektra 0.8.6 and try to get some of those fixed to move closer toward release. Lastly I want to updated the kdb man page for Debian to make it more informative for the shell tool. I was able to get all my goals done for this week except the man page and getting some of the lintian errors fixed up. Overall though it was a very productive week.

Until next week,
Ian S. Donnelly

Week 3: First Merge Implementation

This week went much better. I figured out how to properly implement over an Elektra Keyset and was able to write my first implementation of the Merge function. Right now I need to move on to testing my implementation and fixing the bugs. Already I am aware of a change that will need to be made. I am going to need to use full key names for lookup instead of a cascading lookup like I have now. I plan to use the full name for the root of the keyset in conjunction with the keys full names to change the names to look in the right place for each keyset. Additionally I had a video call with Markus and Felix today and I have a clear plan for what the next two weeks should look like. I really feel like I am getting into the swing of things and better understanding Elektra as well as the development process.

So for next week I plan to better test the merge implementation and fix any bugs that crop up. Additionally I plan to write some automated tests for the merge function to make sure it is running correctly. Additionally I am going to work to understand Debian packaging. Also I am going to look at Samba and understand when it checks the configuration files when upgrading the package to see where the merge tool would be used. I want to get a prototype of a package upgrade running the kdb merge tool during an upgrade instead of prompting the user to manually merge the config files. I also plan to spend some extra hours looking at the latest version on Elektra and trying to fix errors with the debian package. I am going to package a version with the augeas_plugin. Additionally I am going to work on fixing some of the lintian errors that are occurring. In other words, it is going to be a very busy week.

For reference, here is the proper way to iterate through a keyset.

KeySet ks;
Key key;
ks.rewind(); //This is the line I was forgetting. It is necessary to call this function to point the KeySet's internal cursor to the beginning of the KeySet.
while(key=ks.next()){
//This will iterate through the KeySet until key is no longer valid (the end of the KeySet).
}

Anyway, I will update you all next week.

Until Then,
Ian

Week 2: Stuck in a Rut

I was hoping this week would go nice and smooth. I finally felt like I was getting into the groove of things after week 1 but I guess I underestimated the challenge of my project. It pretty much goes without saying that I didn’t quite reach my goal for the week of having a three-way merge of KeySets complete. At first, I ran into some difficulty opening two KeySets at once and comparing keys within them. Once I figured that out, I ran into a problem of how to iterate through KeySets. I am still having difficulty with this particular task. Rest assured though, I will figure that out over the weekend so I can work on properly comparing 2 KeySets next week (right now I can only compare their size). Once I figure out how to iterate through KeySets the best way I will post my findings on this blog so that future developers will have a better understanding of this process, additionally I will add some documentation on this process.

Also this week I laid out a plan for testing my kdb merge function and wrote about the general methodology of testing phase 2 of my project, which is applying kdb merge to package upgrades.

Since I didn’t get that far this week I have had to update my schedule. The good new though is that due to some implementation changes I actually haven’t fallen behind. Originally I was going to work on a generic 3-way merge algorithm in C++ first before working on kdb merge. Now, I am working on kdb merge and implimenting a three-way merge of KeySets when kdb merge is executed. So that brings me to my schedule:

Next week I plan to have my basic three-way merge completed and working through the kdb merge command. This will require me to figure out how to iterate though a keyset before iterating through two keysets and comparing their keys. The next week after that I plan to work on common merging errors such as when a key is removed or appened to a keyset. The week after that I pan on testing my merge function and working on handling conflicts. Finally, by the end of June I plan to have a full working version of kdb merge and will then move onto phase 2 of the project, integrating kdb merge into package upgrades.

I think at this point I have a much better understanding of the scope of the project and I think my schedule is more realistic. I will post a guide to iterating through keysets next once I figure out the best way to do so and obviously I will continue my weekly reports.

Until Then,
Ian

Week 1: Real Life Experience

So as I am finishing up my work for my first week I am going to reflect on it here.

The week started with a pretty major annoyance. During the application process I was doing some commits to the Gitorious repo from a crunchbang machine, but I decided to setup a proper Debian install to work from this summer for the sake of consistency. When I tried cloning the repo Monday I realized I had to add my SSH key for my new install to gitorious, no biggie! Well apparently there was some kind of bug with Gitorious this week because after two hours of generating multiple keys and uploading them to Gitorious, it refused to add my key and mark it as ready so I could work with the repo. I ended up cloning the repo over https but it took until Thursday for me to be able to have my new key successfully add to Gitorious. Lesson learned, check your dev environments before you start working.

Once I decided to stop worrying about my SSH key for the time being I decided to begin working on implementing a Three-Way merge algorithm for Elektra KeySets in C++.  While I am familiar with the basics of a three-way merge I decided to research some full featured implementations and see some robust versions of three-way merge. After familiarizing myself with some more robust algorithms I starting coding a header file and realized I didn’t know the best ways of working with Elektra Keys and KeySets. I turned to the Elektra documentation and read over the various functions for Keys and KeySets. I decided some technical details for my three-way merge which I wrote in a Learning file I added to my branch of the Elektra repo. As you can see, I am still trying to figure out the best way of working with Key Values in elektra. Here is a blurb from that file:

For my merge implementation I plan to iterate though KeySets using ksHead, ksTail, ksCurrent, and ksNext. I am writing two functions, one to determine if two keys are equal (from a merge standpoint) and a similar function for keysets. To compare the keys I was looking at using keyValue but I'm not sure if that is a better solution than keyGetString and keySetString. I have to look more into binary keys though to see how I can make kdbmerge compatible with all keysets.

Other than starting the three-way merge I greatly updated the man page for the kdb merge tool I will be writing later in the summer. The main reason for doing this was just to help visualize what the end product should look like so I can make sure my C++ implementation is written to support the necessary functions.

One of the more important things I have learned this week is how different the workflow is when working on production software as opposed to school projects. I have never worked on a piece of production software before. Generally I have done projects by myself or in small groups and we kind of just keep track of what’s going on. This is impossible with larger projects like Elektra. I am new to git, I just learned how to use it when applying for Google Summer of Code and I have to say it’s simply awesome. I have already been using it to version projects for school or personal projects, it’s just so easy and well organized. With that said, it has taken me a while to figure out what all of the files in the Elektra repo do. When beginning to write the merge code, I realized I had no idea how to include the header files I need because they weren’t in the same directory, I didn’t even really know they were. I wrote about that in my Learning file as well, luckily Markus pointed me in the right direction. Here is a little blurb from that file:

Additionally, I just figured out how to include kdb.h and support for keys in my merge. Somehow I guess I missed the src/tools/kdb folder. This is the first time I have worked on such a large project and its much more complicated to view such a large project from a top-level view then I am used to. I feel like I am starting to get a good understanding of elektra now and how all of the parts of elektra interact (especially kdb and the actual elektra library).

I will say that in a short week I have learned a great deal of what real work looks like in this field. As much as University is supposed to prepare us for our careers, a classroom is very different than a work environment. That realization has made me very glad to be a part of GSoC. I feel like this summer will prepare me for my career in a way that I can’t get in a classroom.

This week I planned to have a basic three-way merge completed. I am not quite there. However, I did update the kdb man document and learn a lot about the workings of Elektra so I think I will have  a much easier time next week.

So for next week I want to have a basic, working three-way merge of Elektra KeySets that correctly handles all types of keys and compiles properly when building the elektra source. Additionally, I want to have a full plan written out for how I will be testing the C++ merge implementation as well as the kdb merge tool. Ideally I will have a few basic tests for the C++ merge tool.

Till next week (or if I have something interesting to share),
Ian