Custom Linux Gui: Where to Begin

Custom Linux GUI: Where to begin?

You want to build your own DE (desktop environment). Common examples are GNOME and KDE, although more of them were popular. It actually used to be very popular to build your own DE, and dedicated scripting tools were often created.

You will usually need to combine multiple tools: a window manager, a toolbar program, a desktop manager (icons), session manager, possibly more. Out of that only a window manager is a required part (it usually doesn't make any sense not to run any WM inside X server), and others are optional. You will usually not write these tools from scratch (it is a lot of work), but use already available components.

Note that both GNOME and KDE actually consists of these elements (well integrated). For example GNOME consists of a window manager (metacity), toolbars (gnome-panel), desktop manager (nautilus) and so on. You can change one element into another if you want: it used to be popular to use sawfish as a window manager instead of metacity, keeping all the other elements intact. Now it is popular to use compiz instead of metacity.

I used to use FVWM, which is a window manager with an additional (quite good) scripting capabilities that could be used to create toolbars, menus and other things. One of nice examples is FVWM-Crystal, which is a complete DE built on FVWM and few other tools, written in a mix of FVWM scripting and Python. You could actually change any kind of behaviour at runtime by just opening FVWM's scripting console and typing commands. FVWM-Crystal can be a good starting point if you want to start from something already usable; bare FVWM is good if you want to build your DE from scratch.

[UPDATE: it seems that gnome-shell is very similar in this regard to FVWM. Most of its behavior is scripted using JavaScript (instead of a custom language in FVWM), and is very easy to change. It is also very actively developed, as it is an important part of GNOME3, with very good support for composition. It's a hacker's heaven ;-)]

[UPDATE2: so it seems I was right on gnome-shell. There's a Cinnamon, which uses the same framework (mutter window manager) as gnome-shell, yet builds a desktop which works in a different way than gnome-shell.]

There was a website that was a good source of ideas to implement: lynucs.org. It is down for two years now, but it got archived.

Warning: lots of these components are not maintained any more, especially since GNOME/KDE started to become actually usable. Development in this area is usually driven more by specific requirements of embedded devices: phones or netbooks.

Now, to answer your actual questions:

• Linux/OS flaver: anything easily customizable: Debian, Arch, Gentoo, LFS...
• Tools/frameworks: it depends on what do you want to achieve. If you want to write a custom toolbar you might just use some scripting languages (like FVWM's), custom tools like adesklets, or write your own in Qt (if most of your other components is in Qt) or GTK (if most of your other components is in GTK).
• Personal UIs: there were lots of them, and sites like lynucs.org was a nice compilation of them (lynucs listed components used in each desktop).
• Complex transparency effects: older tools usually do not handle it natively, or do a simple things like just reusing part of desktop wallpaper as its own background (so-called fake transparency). Most probably you'll need to write your own code, f.e. as a plugin to compiz.

Create a custom login screen GUI for linux

The "login screen" shown on GNU/Linux systems to login to a graphical session is part of the desktop environment you are using. Various alternatives exist for the zoo of alternative environments out there. Examples are kdm as offered by the KDE desktop environment, gdm as offered by Gnome, xdm, wdm, many others exist. Start by reading the wikipedia article about kdm to get an impression.

You'd have to implement a replacement for that part which serves the same purpose, if the existing options to configure it are not enough. So best probably is to start reading about those. That should give you an impression about the requirements.

Roughly explained you have to create an executable that can be executed by the system and configure that as "desktop manager" in the systems configuration. The exact location of that configuration again depends on the distribution, but the general approach always is the same.

The logic of such a program has to somehow answer the question of whether a user is allowed to login to the system by whatever means chosen. Typically this is done by using PAM in background ("Pluggable authentication modules"). Maybe start by reading an introduction to PAM to gain insight into how things work, which may be a bit confusing at first glance due to its endless flexibility. That framework is used on modern unixoid systems to separate the requirements for that logic from the actual implementation. So you really want to learn about how PAM works too. It is an unbelievable flexible framework where already endless strategies exist, so countless adapters for all sorts of existing authorities like databases, LDAP servers, simple file based authorities, IMAP servers, you name it: everything you can imagine. So chances are that you do not have to implement anything, but just can pick something implemented by someone else and configure / tailor it to your needs.

The logic itself does not have to be complex, it just has to answer that simple question in an clear way: is the user who requests access granted or not. For that it will (or will not) have to consult some existing authority. How exactly that works obviously is completely up to you. The real difficult thing probably is to access and control the graphics hardware without having a running graphical desktop environment to build on (since that is only started afterwards). So you probably have to deal with some hardware abstraction layer. Which again means that you probably want to be using some language like C or C++ for that job, scripting languages are less well suited for such job, also because they introduce all sorts of additional requirements of what has to be installed on a system which becomes unusable if some element inside that chain of utilities break due to the lack of a login screen :-)

How to build Linux system from kernel to UI layer

I apologize in advance for a very long winded answer to what you thought would be a very simple question. Unfortunately, piecing together an entire operating system from many different bits in a coherent and unified manner is not exactly a trivial task. I'm currently working on my own Xen based distribution, I'll share my experience thus far (beyond Linux From Scratch):

1 - Decide on a scope and stick to it

If you have any hope of actually completing this project, you need write an explanation of what your new OS will be and do once its completed in a single paragraph. Print that out and tape it to your wall, directly in front of you. Read it, chant it, practice saying it backwards and whatever else may help you to keep it directly in front of any urge to succumb to feature creep.

2 - Decide on a package manager

This may be the single most important decision that you will make. You need to decide how you will maintain your operating system in regards to updates and new releases, even if you are the only subscriber. Anyone, including you who uses the new OS will surely find a need to install something that was not included in the base distribution. Even if you are pushing out an OS to power a kiosk, its critical for all deployments to keep themselves up to date in a sane and consistent manner.

I ended up going with apt-rpm because it offered the flexibility of the popular .rpm package format while leveraging apt's known sanity when it comes to dependencies. You may prefer using yum, apt with .deb packages, slackware style .tgz packages or your own format.

Decide on this quickly, because its going to dictate how you structure your build. Keep track of dependencies in each component so that its easy to roll packages later.

3 - Re-read your scope then configure your kernel

Avoid the kitchen sink syndrome when making a kernel. Look at what you want to accomplish and then decide what the kernel has to support. You will probably want full gadget support, compatibility with file systems from other popular operating systems, security hooks appropriate for people who do a lot of browsing, etc. You don't need to support crazy RAID configurations, advanced netfilter targets and minixfs, but wifi better work. You don't need 10GBE or infiniband support. Go through the kernel configuration carefully. If you can't justify including a module by its potential use, don't check it.

Avoid pulling in out of tree patches unless you absolutely need them. From time to time, people come up with new scheduling algorithms, experimental file systems, etc. It is very, very difficult to maintain a kernel that consumes from anything else but mainline.

There are exceptions, of course. If going out of tree is the only way to meet one of your goals stated in your scope. Just remain conscious of how much additional work you'll be making for yourself in the future.

4 - Re-read your scope then select your base userland

At the very minimum, you'll need a shell, the core utilities and an editor that works without an window manager. Paying attention to dependencies will tell you that you also need a C library and whatever else is needed to make the base commands work. As Eli answered, Linux From Scratch is a good resource to check. I also strongly suggest looking at the LSB (Linux standard base), this is a specification that lists common packages and components that are 'expected' to be included with any distribution. Don't follow the LSB as a standard, compare its suggestions against your scope. If the purpose of your OS does not necessitate inclusion of something and nothing you install will depend on it, don't include it.

5 - Re-read your scope and decide on a window system

Again, referring to the everything including the kitchen sink syndrome, try and resist the urge to just slap a stock install of KDE or GNOME on top of your base OS and call it done. Another common pitfall is to install a full blown version of either and work backwards by removing things that aren't needed. For the sake of sane dependencies, its really better to work on this from bottom up rather than top down.

Decide quickly on the UI toolkit that your distribution is going to favor and get it (with supporting libraries) in place. Define consistency in UIs quickly and stick to it. Nothing is more annoying than having 10 windows open that behave completely differently as far as controls go. When I see this, I diagnose the OS with multiple personality disorder and want to medicate its developer. There was just an uproar regarding Ubuntu moving window controls around, and they were doing it consistently .. the inconsistency was the behavior changing between versions. People get very upset if they can't immediately find a button or have to increase their mouse mileage.

6 - Re-read your scope and pick your applications

Avoid kitchen sink syndrome here as well. Choose your applications not only based on your scope and their popularity, but how easy they will be for you to maintain. Its very likely that you will be applying your own patches to them (even simple ones like messengers updating a blinking light on the toolbar).

Its important to keep every architecture that you want to support in mind as you select what you want to include. For instance, if Valgrind is your best friend, be aware that you won't be able to use it to debug issues on certain ARM platforms.

Pretend you are a company and will be an employee there. Does your company pass the Joel test? Consider a continuous integration system like Hudson, as well. It will save you lots of hair pulling as you progress.

As you begin unifying all of these components, you'll naturally be establishing your own SDK. Document it as you go, avoid breaking it on a whim (refer to your scope, always). Its perfectly acceptable to just let linux be linux, which turns your SDK more into formal guidelines than anything else.

In my case, I'm rather fortunate to be working on something that is designed strictly as a server OS. I don't have to deal with desktop caveats and I don't envy anyone who does.

7 - Additional suggestions

These are in random order, but noting them might save you some time:

• Maintain patch sets to every line of upstream code that you modify, in numbered sequence. An example might be 00-make-bash-clairvoyant.patch, this allows you to maintain patches instead of entire forked repositories of upstream code. You'll thank yourself for this later.
• If a component has a testing suite, make sure you add tests for anything that you introduce. Its easy to just say "great, it works!" and leave it at that, keep in mind that you'll likely be adding even more later, which may break what you added previously.
• Use whatever version control system is in use by the authors when pulling in upstream code. This makes merging of new code much, much simpler and shaves hours off of re-basing your patches.
• Even if you think upstream authors won't be interested in your changes, at least alert them to the fact that they exist. Coordination is essential, even if you simply learn that a feature you just put in is already in planning and will be implemented differently in the future.
• You may be convinced that you will be the only person to ever use your OS. Design it as though millions will use it, you never know. This kind of thinking helps avoid kludges.
• Don't pull upstream alpha code, no matter what the temptation may be. Red Hat tried that, it did not work out well. Stick to stable releases unless you are pulling in bug fixes. Major bug fixes usually result in upstream releases, so make sure you watch and coordinate.
• Remember that it's supposed to be fun.

Finally, realize that rolling an entire from-scratch distribution is exponentially more complex than forking an existing distribution and simply adding whatever you feel that it lacks. You need to reward yourself often by booting your OS and actually using it productively. If you get too frustrated, consistently confused or find yourself putting off work on it, consider making a lightweight fork of Debian or Ubuntu. You can then go back and duplicate it entirely from scratch. Its no different than prototyping an application in a simpler / rapid language first before writing it for real in something more difficult. If you want to go this route (first), gNewSense offers utilities to fork your own OS directly from Ubuntu. Note, by default, their utilities will strip any non free bits (including binary kernel blobs) from the resulting distro.

I strongly suggest going the completely from scratch route (first) because the experience that you will gain is far greater than making yet another fork. However, its also important that you actually complete your project. Best is subjective, do what works for you.

Good luck on your project, see you on distrowatch.

How do you make linux GUI's?

X is a hideous layer to program for and, despite your intent to avoid Java, QT or any of the excellent UI abstraction layers, you'll be doing yourself a disservice by coding to that level. I've done it (a long time ago when Motif was in its infancy on the platform we were using) and I would not do it again if there was an easier way.

Your use of the phrase "native programming" confuses me a little. If you want to learn native programming, it's to the APIs that you choose to call. Using similar reasoning, you shouldn't be coding in C either, instead opting for assembler (or direct machine code) since C provides an abstraction to the hardware.

If you want to learn X programming, that's fine. You'll end up with a lot more control over your interface but almost everyone else will be out-performing you in terms of delivery of software. Myself, I'd prefer to code to a higher-level API that I can use on many platforms - it gives me both faster delivery times and more market potential.

You don't build a house out of atoms, you build it out of bricks. My suggestion is to use the tools, that's what they're there for.

---

Related Topics