Why not to kill people who write documentation?

Well… as one of my friends says: “It’s not worth it – you’ll go to jail as if you murdered a human!”.

You can always say that if you are using a free library, “don’t look a gift horse in the mouth”. If you don’t like this library, don’t use it. Or write your own. Or fix it by yourself. This may also happen in case of a commercial-licensed library, but even then you can say: don’t buy it, use something else, if you don’t like it. Theoretically.

But on the other hand, why to distribute a library (or tool) for which the usage rules are to be yet determined by experimental usage? What profit may anyone have from them? The cost of use for such libraries actually grow towards what was expected at the first glance, and more importantly, they do not well contribute to good opinion about such a library, no matter how useful it is. If you release such a library, you are a liar because you lie about the real costs of using this library. Especially if you release it for free. You count for help? So say it explicitly: I need help.

I wish every library is so thoroughly documented as, for example Qt is. Unfortunately there is a lot of libraries that are recommended as good solution for something, but… well, if you have problems with using it, you should experiment (read: waste time), try to read more times to better understand (read: waste time) or ask on forums, of course, searching for information about it first (read: waste time). How “free” is a library that requires you to pay so much with your time?

Take a look at GWT API Documentation and find a class called “Grid”. Find a method called “prepareCell”. What does this method do?

Checks that a cell is a valid cell in the table.

Checks and what? Especially that this method’s return type is void (this is Java so a value cannot be also returned through referred variable). I have always thought that if a function is going to “check” something (not “make sure of that!”), it performs some verification and returns some status to indicate the result of checking. The “check” name not necessarily indicates only a verification; it can always make some fixes accordingly. But even if so, it’s nothing said about it.

Ah! It throws an exception of type IndexOutOfBoundsException. But well… if this is said so enigmatically, a developer should rather pay little attention on this and treat this exception as something that shouldn’t have happened and would be thrown only in a case when there is something that they could prevent against (because the same is said about, for example, String::charAt() function).

It’s not explicitly said, at which circumstances the exception is thrown (of course, the author of the javadoc comments didn’t specify it and the documentation mentions about this exception only because it’s automatically generated). It may mean various things: either this exception is a form of reporting the result by this function (that is, one of possible results, but more strictly, the only result), or this exception is thrown just in a situation when a user did not verify something important and let invalid index values to be created.

There are then two possibilities of why such a function exists:

  1. This function does some enigmatic “checks” and is mandatory to be called before operations on particular cell. It is provided only because it’s useless to check these things before every instruction that manipulates with the cell, so you should only ensure that it was called before you access particular cell.
  2. This function verifies whether the cell with specified coordinates exists; it returns normally, if so, and throws the exception otherwise.

I tend to think that the second one is what this function does, but it’s not said explicitly. And I showed here not the worst example of incomplete documentation. The fact that this is a good example of how idiots use exceptions in public API, which I have already described in my article about exceptions, is yet another “flower” in this funny library.

It reminds me one of humor pages when there was an explanation about the difference between ‘memcpy’ and ‘memmove’ functions: “memcpy function copies the memory, while memmove moves it”.

I have always said that if you want to make sure that your call will not end up with unexpected results, read the documentation of the called function and make sure that you have managed every case of result it may provide. This advice, however, seems to be wasteful, when you read such a documentation.

So, if you can’t rely on documentation, the only thing that you can do is to experiment. Of course, there’s nothing better than just read the sources (Qt library has been always distributed together with sources – just for a case when, even if impossible, the documentation would be not perfect enough). But unfortunately, many times even the sources isn’t a good source (!) of knowledge. Sometimes the source code is so dirty, complicated and indeterministic that it’s not possible to go through it by simple reading it. The ultimate thing to do seems to be just… run your application under debugger, and… yes, debug the library.

Yes, I’m not kidding. I was developing a Windows application at ~2001 year, using MFC. It was really fortunate that MFC came up with its source code. Otherwise some bugs would be just unable to find. It’s especially important in case of Object-oriented library. For example: I am overriding some method and the problem is that at the time when my method is called some data are not yet ready. So? How would you determine at what exactly point of code my method is called if I don’t have source code of the library that calls it? It’s not even needed that the library be object-oriented, it’s just enough when the library is using callback calls (polymorphism is actually a callback-based technique).

You say, the library should come up with good documentation, which states thoroughly, in which places my callback function is called. Na’ive :). Usually it would just say that the function does some specific thing… and that’s all. People are sometimes too stupid to remember that if a function is virtual (so, overridable), then the most important is to describe, at which circumstances this function is to be called and what’s this function purpose (so that the user’s implementation can be written in a conformable way). If the function is both virtual and implemented (that is, not abstract), then both descriptions should be provided: what the original implementation does and what the overridden implementation should do. Usually, though, the documentation is too weak to provide this complete information.

Same thing is about exceptions, return values, status etc. For example, there’s a lot of APIs, in which the author(s) cannot precisely describe, why a function may fail. They just write “it returns -1/returns 0/throws a XXX exception in case of failure”. And the function may fail because, well, it may fail. And the function, well, throws an exception, and, well, you should catch the exception in case this function throws it. While the basic thing about the failure for the API user is to grab the difference between errors that can be prevented from (by some early checks) and errors that may occur by independent reasons. This is impossible if a function’s documentation states, practically, that it can, well, throw an exception if it has a caprice to do it. It’s a pity that it never says that it can raise SIGSEGV if it has a caprice to do it.

I would like not to say anything more about MFC or otherwise this article will become thrice longer. Maybe something has changed since the last 5 years, when I was using this library last time, but this library had also errors in documentation – for example, a famous error in the Find dialog’s documentation, which states that the dialog should be deleted when no longer used, while by tracing the source code you can see that in response to WM_NCDESTROY it calls some virtual function PostNCDestroy that is in this particular dialog class overridden and implemented as “delete this”.

An example of a good documentation (beside Qt) is the set of POSIX man pages. For example, let’s look at the documentation for ‘socket’:

RETURN VALUES
     A -1 is returned if an error occurs, otherwise the return value is a
     descriptor referencing the socket.

ERRORS
     The socket() system call fails if:

     [EACCES]           Permission to create a socket of the specified type
                        and/or protocol is denied.

     [EAFNOSUPPORT]     The specified address family is not supported.

     [EISCONN]          The per-process descriptor table is full.

     [EMFILE]           The per-process descriptor table is full.

     [ENFILE]           The system file table is full.

     [ENOBUFS]          Insufficient buffer space is available.  The socket
                        cannot be created until sufficient resources are
                        freed.

     [ENOMEM]           Insufficient memory was available to fulfill the
                        request.

     [EPROTONOSUPPORT]  The protocol type or the specified protocol is not
                        supported within this domain.

     [EPROTOTYPE]       The socket type is not supported by the protocol.

     If a new protocol family is defined, the socreate process is free to
     return any desired error code.  The socket() system call will pass this
     error code along (even if it is undefined).

Now let’s look at the “other side” – let’s take, for example, Berkeley DB, which is a very old library, with very simple features, since last time supported by Sleepycat company, which was some years ago taken over by Oracle (so, yes, this “product” is finally supported by Oracle). To have a shortcut, let’s take a look at the documentation for BDB’s C++ API:

Opening Databases

You open a database by instantiating a Db object and then calling its open() method. 

Note that by default, DB does not create databases if they do not already exist. To override this behavior, specify the DB_CREATE flag on the open() method. 

The following code fragment illustrates a database open:
#include <db_cxx.h>

...

Db db(NULL, 0);               // Instantiate the Db object

u_int32_t oFlags = DB_CREATE; // Open flags;

try {
    // Open the database
    db.open(NULL,                // Transaction pointer
            "my_db.db",          // Database file name
            NULL,                // Optional logical database name
            DB_BTREE,            // Database access method
            oFlags,              // Open flags
            0);                  // File mode (using defaults)
// DbException is not subclassed from std::exception, so
// need to catch both of these.
} catch(DbException &e) {
    // Error handling code goes here
} catch(std::exception &e) {
    // Error handling code goes here
}

It’s not the point that the complete API is not described, for example, not all open flags are described (they are described later). The point is: there is no description of the function’s return value (it’s not even stated that it is void, although it isn’t :); in the example there are exceptions caught, but there’s not explained the reason, why there can be any error when calling Db::open().

This is the description signed by Oracle. There is also other description, for example, one that I have found in the libdb package on Cygwin. In this case the Db::open function seems to be much better described:

Errors
ENOENT
The file or directory does not exist. 

The Db::open method may fail and throw DbException, encapsulating one of the
following non-zero errors, or return one of the following non-zero errors:
DB_OLD_VERSION
The database cannot be opened without being first upgraded.
EEXIST
DB_CREATE and DB_EXCL were specified and the database exists.
EINVAL
If an unknown database type, page size, hash function, pad byte, byte order,
or a flag value or parameter that is incompatible with the specified database was
specified; the DB_THREAD flag was specified and fast mutexes are not available
for this architecture; the DB_THREAD flag was specified to Db::open, but was not
specified to the DbEnv::open call for the environment in which the Db handle was
created; a backing flat text file was specified with either the DB_THREAD flag or the
provided database environment supports transaction processing; or if an invalid flag
value or parameter was specified.
ENOENT
A nonexistent re_source file was specified.
DB_REP_HANDLE_DEAD
The database handle has been invalidated because a replication election
unrolled a committed transaction.
DB_REP_LOCKOUT
The operation was blocked by client/master synchronization. 

If a transactional database environment operation was selected to resolve a
deadlock, the Db::open method will fail and either return DB_LOCK_DEADLOCK or
throw a DbDeadlockException exception.

If a Berkeley DB Concurrent Data Store database environment configured for lock
timeouts was unable to grant a lock in the allowed time, the Db::open method will fail
and either return DB_LOCK_NOTGRANTED or throw a DbLockNotGrantedException
exception.

There is one problem there: it’s not clear in this description, in which situation there can be an exception, and in which situation there can be a return value informing about the error (it’s described in another place, at DbException). Moreover, putting ENOENT on top makes it a bit exceptional and suggests that this exactly value can only be returned, while only the others can be send via exception (I guess it isn’t true). The problem is that users usually do not read documentations from board to board, but read only some fragment that is currently interesting to them. In this document there are even no links or redirections to other parts, and users tend to state that if something is not described, it works “some default way” – for example, if some special case isn’t mentioned, users think there are no special cases.

It’s hard to say, how much it has to do with the poor quality of the library API itself (it can be configured to either return errors by value or throw an exception) and how much with the documentation only. I have made so far two approaches to BDB, as it seemed to be a very simple (so, flexible) library for use with a very distinct place; the last one was memory allocation tracer (based on malloc_hook). Unfortunately, it proved to be useless due to some strange operations done with memory and it was crashing too often by unknown reason (version 4.5). Finally I had to use a client-server method and send the data to another process for collection.

Only things we have done by ourselves are free. Well, even this isn’t true. We still have to spend time, and time costs.

In case of libraries, the documentation for the library is moreover important, as problems with this will then expand when making software using this library. For example, take a look at this description from Elementary library (part of EFL library):

EAPI char* elm_entry_utf8_to_markup  (const char *s)

This converts a UTF-8 string into markup (HTML-like).
Parameters:
 - s: The string (in UTF-8) to be converted 

Returns:
The converted string (in markup)

Well… ok. Let’s take this a good deal: it returns a string; we can use it in our operations without any restrictions.

Ha! No way! Probably you can suspect something when you can see that the return type is ‘char*’, not ‘const char*’? Yes, of course, you are right: this function allocates memory by malloc() and returns a pointer to a dynamically allocated array of characters. So, once you finished with this array you got from this function, you must call free() in order to release it (or store it somewhere, where you usually keep only malloc()’ed memory).

No word about this thing in the documentation? Oh, well… for intelligent programmers it should be obvious that it returns allocated object because this pointer could not come from anywhere else (static local variable as buffer is a racy solution). A bit more funny thing is that in previous version this function was returning a string different way and it didn’t have to be freed afterwards… (and return type was const char*).

And that’s how it’s so easy to get into memory leaks. This is the biggest problem with C language. Its problem is not that it’s so raw and so low level. The problem is that this language does not even give you abilities to at least mark your intentions in the source code and count for any help from compiler to point you things that are errors in regard of this intention. It means that every smallest thing must be very thoroughly documented especially in case of libraries having C API (including those written in languages other than C). The function receives a pointer to X as argument? You must describe whether this pointer is an iterator to an array, or a pointer to the array’s beginning (with size in another argument or known from elsewhere), or a pointer to a variable, or maybe a pointer to a single object, maybe a pointer to an object to be consumed (freed afterwards) by the function. A pointer just plays too much roles in C language. For example (this spreads also to C++, but in this language you at least have alternatives in form of some advanced pointers), you cannot just “compare two pointers” and expect that you’ll receive some certain result. If you want to compare two pointers, you must know exactly what one of these pointers hold, and also what possible meaning is of the value of the other pointer. It’s because if you don’t know these things, you are not allowed to compare pointers, in particular, you are not allowed to use the result of the comparison in any next evaluations. For example, if your pointer is a pointer to single object or a pointer to function – it may contain some value of pointer or NULL. If this is an iterator to an array, it may have a value of a pointer to an element of this array from its beginning to the one element past the end (and before the first, too!). Moreover, we can put a constraint that a pointer, no matter its destination, is not allowed to be NULL. Effectively, whenever you have a pointer used in some interface, all aspects must be thoroughly described (again: single object, array of objects, nullable or not, producer/consumer).

Same thing about int – it’s even much more universal type. The fact that two functions return integer values doesn’t simultaneously mean that it makes sense to compare them. Well, yes, too many people do not even realize it (that’s why I tend to think last time that the C language can be preferred only by people, who are – concerning their professional experience – programming children, who just do not realize the threats carried by this language). For example: can you add values of X and Y coordinates? Yes, you can add, but what logical interpretation such a value has?

Every such problem costs later the time spent to detect and wipe bugs, and also the time for experimenting costs. As I said, in worst case you would have to even run your application under debugger together with the library and debug the library itself. You would eventually have to develop this library – dear customer, make pleasure to yourself because the “free developers” prefer to stuff you in…

Sorry.

People, who want to give credit to free and open software must first consider that this software in major cases suffers of poor documentation, and this is the main reason of why this software tends to generate higher costs than the software, for which you have to pay. Currently I can see only one significant good thing introduced by free software: it at least defines the basis for what you may have for free. It forces then those, who want to sell software, that in order to make money for this software they have to do something more than just create, test and distribute it.

Unfortunately the usual situation is that when there is some kind of software existing as free software, and then no-one even is going to produce a commercial version of this kind of software. They prefer to produce something that’s not available in free software world… with poor documentation, as usual.

That’s why the only software that looks professional is the commercial software created for developers, and software produced commercially for contracted customer, whom you’d have to pay high fines in case when the software is poor. Although it’s not unlike any other kind of products.

There wouldn’t be anything wrong with this but one thing: poor documentation is nothing else than cheating. Let the authors of poorly documented software state a disclaimer in the beginning: “this software is very poorly documented” – and we’ll be ok, we know what we are starting with. As this is a shooting oneself in foot for commercial software, the authors of open source software should consider this rule very seriously. This will contribute to more fair approach for the users, and also may be a call for help for creating some better documentation. Only advantages. But not for people, who count for that they can make profit from cheating.

Don’t kill them. Walk them around. Read the documentation before you decide whether to use a library. And if the documentation is poor, do not hesitate to drop it.

Posted in Uncategorized | Leave a comment

Exceptions konsidered harmfool

Introduction

Some time ago I found a very old article, where the author says that the exception handling should be skipped as much as possible, and exceptions should be caught immediately and handled exactly in place. And that this is worse than goto because the points where the execution stops are invisible. He restrained himself in another article, saying that there is, well, no good exception handling in general. Yes, I’m talking about Joel Spolsky. “Uh, oh”, speaking his language.

There are two general faults in this thinking. The first one is to call “hidden” something that is only written in different place. The second one is to think that a crash is the worst thing that may happen to a program. Only two possibilities are as to why state things like that: either to be a complete ignorant, or to be for a long time too familiar with procedural style and have no idea what to do in case of having exceptions in a language. At most, to know exceptions only from Java libraries, where they are used the most possible wrong way. As Joel Spolsky is an old and experienced programmer, I would mercifully assign to him this last possibility.

Theory of error handling in imperative languages

If you have several instructions, where each one can fail in its execution, you should detect this problem and… well, do something. We’ll reconsider this “do something” later.

If you are using the return-value error handling (I wanted to say “old-fashion”, but even the old-fashion APIs don’t rely only on return values), the following things have to be considered:

  • every call may fail, so you must execute the functions separately, checking the post-call status
  • every function must have some tools to determine the execution status – by a return value or additional variable passed via reference
  • if the status is returned by the return value, and the return value means something in case of success, there must be selected some special value for the case of failure

It means, for example, that you cannot build a chain expression, where a return value of a function is passed to another call. Consider:

object->GetFactory()->GetObject(value)->create();

Use( getValue(), getStep() );

Unfortunately, if “object->” returns NULL, GetFactory will fail – so you’d better do it the following way:

if ( !object )
 handle_error();

Factory* f = object->getFactory();

if ( !f )
  handle_error();

Object* o = f->getObject(value);

if ( !o )
  handle_error();

if ( !o->create() )
  handle_error();

Value v = getValue( &status );
if ( status == error )
  handle_error();

Step s = getStep( &status );
if ( status == error )
  handle_error();

Use( v, s, &status );
if ( status == error )
  handle_error();

This has made the code more resistive to runtime errors, but this form is totally unreadable towards the previous form. In the first one you can clearly see the connections of operations and see the whole logics, while in the second one every single operation is intermixed with error handling (best if so concise as here!). Seeing any logics in this code is almost impossible.

The exception mechanism seems to be a remedy for most of those problems. Consider:

try {

  object->GetFactory()->GetObject(value)->create();

  Use( getValue(), getStep() );

} catch ( null_pointer_exception ) {
  handle_error();
} catch ( value_exception ) {
  handle_error();
}

This way you keep the expressions concise and simple, while still the failures at every step will cause terminating the execution and running handle_error().

I’m not writing these obvious things to show why exceptions are better than reporting via return value (I haven’t shown their all advantages). If you want to reconsider whether exceptions are better than reporting problems via return values, you have to consider this topic in two separate aspects:

  1. Reporting the result of a predicted condition, about which we can’t grant anything yet (we are just checking it).
  2. Reporting the problem, when an operation is expecting some condition to be true and it’s false, so the operation has been abnormally terminated.

The above example shown how exceptions are better than return values, but it’s because they refer to case #2. Let’s try something in case #1, done by return values:

if ( !fs::is_readable( "somefile.txt" ) )
    return NO_FILE_TO_READ; // nothing to do

ifstream z( "somefile.txt" );
size_t size = z.seekg( 0, ios::end ).tellg();
z.seekg(0, ios::beg );
string s;
s.resize( size );
z.read( s.data(), size );
if ( !z )
    return FILE_READ_ERROR;

size_t pos = s.find( "lockable" );
if ( pos == string::npos )
   return NO_OCCURRENCE; // nothing to do

... 

Could these checks be replaced with exceptions? Well…

try {
 ifstream z( "somefile.txt" );
 z.exceptions( ios::badbit | ios::failbit );

 size_t size = z.seekg( 0, ios::end ).tellg();
 z.seekg(0, ios::beg );

 string s;
 try {
  s.resize( size );
  z.read( s.data(), size );
 } catch ( ... ) {
    return FILE_READ_ERROR;
 }

 try {
    Extract( s, "lockable" ); // will throw exception if not found
 } catch (...) {
    return NO_OCCURRENCE;
 }

 ... 
} catch ( ... ) {
   return NO_FILE_TO_READ;
}

Does it still look better than the value-return version? I don’t think so.

Saying something like that the information about the exception is hidden (well, I state that you have never tried in your program to use some function, for which you neither have a complete documentation nor source code) is the old-fashion procedural thinking. It’s not true that this is dangerous, as long as you do not violate some simple rules per role:

  • the library user uses resources that have predictable conditions of releasing and keeps resources releasable until the operation is complete
  • the library author reports exceptions only because of reasons that the user could have predicted (and prevented from happening), by the reasons reported from some underlying calls, or other kinds of termination events, optionally

I would agree with Joel Spolsky that the exceptions should be handled immediately, but only in one case: if the exceptions were used by the library some stupid way – for example, like the way the IOException exception is thrown in file operations in Java standard library. This can be called “direct translation”, as this is simply translation of exceptions into return values.

The problem isn’t in exceptions. The problem is that exceptions is a mechanism that should be used only for reporting some selected kind of errors, not all possible errors (that is, calls that do “check some condition” should never report the “failure check” via exception). Exception is a mechanism that allows to report an error together with causing an abnormal procedure termination (because this is yet another way to exit the procedure in addition to “return”). So if you consider, which way is better to report an unrecoverable error with abnormal procedure termination, the answer is simple: exceptions, not return values, because the exception mechanism:

  • ensures immediate termination without engaging return value
  • ensures exception propagation in case when it’s not handled by the caller
  • in RAII languages (like C++), it also ensures automatic destruction
  • in non-RAII languages, it provides a mechanism to release resources at the end (when both exception was caught and the program normally continues)

So, there’s no dilemma whether to use exceptions or return values because this is like considering real object-oriented with C++ vs. manual pointer-to-function-based object-like programming in C. The real dilemma is in different place: in which practical situations “special situations” should be reported by exceptions.

Don’t prevent the prevention

Another aspect, which is astonishing to be heard for such an experienced programmer as Joel Spolsky, is to believe that crash is the worst thing that may happen to your program. Actually I think I should be jealous to those who think that – I would be happier to believe that there’s no worse thing to my program than a crash. Unfortunately I don’t have this solace of unawareness. I can only hope that the reason standing behind his statement of handling exceptions in place was something different than preventing programs from crashing, but I’m not sure of that because, well, this is the only practical consequence of that.

Try to look at it this way: imagine that you are driving a car, you are riding a bit fast and urgently you have a crowd of people in, say, half a kilometer ahead of you. You’re trying to brake, but well… the brakes don’t work. You are still riding. The distance to the people decreases. You don’t have any other way; there are buildings on both sides of the road. You can’t reverse because you’re riding too fast. In such a situation you have only two choices – either you crash on a building or you’ll let this car ride into this crowd of people and squish some of them.

These two things refer to things that may happen when you forget to handle an error reported by some function you call: the first one happens when you forget handling an exception, the second one when you forget handling an erroneous return value.

If you understand this example well enough, you’ll understand that someone, who is trying to do all possible things to prevent the program from crashing, is a complete idiot. Fortunately creators of standard libraries usually aren’t idiots and that’s why trying to do strlen() with NULL, or passing NULL to std::string’s constructor usually ends up with crash. Error handling should not prevent the program from crashing (although it may do its best to prevent vital data from being destroyed). Error handling should prevent the program from continuing, which usually means that the sub-functionality or even the whole program, if it’s unable to continue because of this condition, should be abnormally terminated (that is, crashed).

There is, of course, one more thing that can stand for the idea to prevent the crash even with the cost of destroying user’s data. When a user can see that the program crashes, his first thought is “what a s*** stuff!”, while when there is some destruction in the data, the user may think that, well, nothing wrong happened, probably they won’t even notice that something has crashed. And even if some data destruction happened, you can always say that, well, this is something wrong with your operating system or you have installed some malicious software. Of course you still risk that there is someone inquisitive who can prove that your program has a bug and you intentionally hold the crash by the cost of data destruction, and you are doomed. If you want to prevent the crash just to not let the program make you black PR, just try to make the process of saving the data and restoring the program to the previous state fully automatic. For example, the Opera browser I am using is the most crashing web browser out of all web browser I have been using – but Opera still has many advantages over other web browsers and even if there is some crash once per a time, it quickly recovers to the previous state after automatic restart. A short conclusion is: don’t try to trick users because it is enough that one smart user detects you and you are in the ***hole.

The useless handlers

Once you realized that crash actually isn’t such a bad thing, you can now think well whether you really would have to handle the error (in case of return values you have to have some way to propagate the error!), stating that the error handling is not a means to prevent the program from crashing.

In Tcl language, for example, there is a mechanism that works merely like exceptions – status codes (they work merely the same way). In this language they are used not only to report errors, but also as a mechanism for implementing the break/continue statements for loops. This language may teach you one very important, but controversial thing: Don’t handle an exception at all, if you can’t do anything better than there would happen when it’s not handled. In other words, let an error crash your program, if all you can do is to quit the application with error message (which for average user is same as “crash”). Simply because the Tcl built-in error handling will do exactly the same, with only a bit different error message. For example:


set fd [open somefile.txt r]
set contents [read $fd]
close $fd

do_something_with $contents

If the file was not found, [open] will end up with error and will crash the script (stating that this above is executed in a “bare interpreter”, not from within some procedure that may run it in its [catch] block):

couldn't open "somefile.txt": no such file or directory
    while executing
"open somefile.txt r"
    invoked from within
"set fd [open somefile.txt r]"
    (file "somescript.tcl" line 1)

Ouch! Well, so let’s add some error handling to our script:


set failed [catch {set fd [open somefile.txt r]} errmsg]
if { $failed } {
 error "File 'somefile.txt' not found"
}
set contents [read $fd]
close $fd

do_something_with $contents

Just to receive an extremely different and much better 😀 result:

File 'somefile.txt' not found
    while executing
"error "File 'somefile.txt' not found""
    invoked from within
"if { $failed } {
                error "File 'somefile.txt' not found"
}"
    (file "somescript.tcl" line 2)

Uh, oh. Yes, I know, I was a bastard – I have used the [error] command to handle the problem. I should have used something else. OK, go on, use something else. You’ll probably want to use [puts] command to display the error message (just to make it different from the human-readable stack dump that Tcl interpreter returns by default) and you’ll do [exit 1] (to make it return result 1 to the shell, what is also done by default by Tcl interpreter in case of error). Or, as you have the Tk package available, you’d prefer displaying some message window with the error message? Well, put the whole script (that is, the part that is independently executed – proc’s may be outside it) into a [catch] block and show the message window when it returned nonzero value. Just once for the whole script.

You can do all these things, then take an average user and make them use this program till getting this error and ask them, what happened to this program. Every “average user” will tell you “it crashed”. So what’s the difference?

If you really want to do something “really different”, do not handle this situation via exception. The error when opening a file reported by exception is exactly the right thing to do because it’s the problem that you could have predicted and you didn’t. You still could:

if { ![file readable somefile.txt] } {
   puts stderr "I kindly inform you that the file 'somefile.txt' is not possible"
   puts stderr "to be open for reading. Please make sure that this file can be read"
   puts stderr "and try again."
   puts stderr "Thank you for using Tcl for writing scripts"
   exit 1
}

set fd [open somefile.txt r]
set contents [read $fd]
close $fd

do_something_with $contents

Please note that this time we didn’t handle the prospective exception thrown from [open] command. But why should we do it? We have prevented the exception from being thrown!

Of course there still may be cases like “the file was deleted in between”, but this kind of error is qualified to different category, and we’ll talk about this later. In this place I can only tell you that if this happens, a crash is quite intended.

Remember that some users will try to intentionally make your program crash and they may also infringe the statements of the operating system in order to make it happen – don’t try to do everything you can to prevent it because, simply, you can too little. Try to do your best to prevent your program from crashing during normal use (but not in runtime!), not to withstand any abnormal situation. The most valuable thing that the user can have from using your program is data. Your program should never allow that the data are corrupted, lost, overwritten or leaked. Your program should not allow to compromise the safety and security of the data. And even if the only way to protect the user data is to crash the program – let the program do it and do not deliberate too much because this is exactly what you should do.

So why should I handle exceptions?

There are generally only two reasons why you may want to handle exceptions:

  1. The exception mechanism for this particular case was optional and you turned it on because you currently prefer this way (example: exceptions thrown from iostream operations in C++ std library, only if the exception flag is set by ios::exceptions).
  2. The procedure that ended up with exception was running in a kind of “subsystem”, it isn’t crucial to your application, but at most to this subsystem. This way only this “subsystem” is treated as “crashed”, while rest of the program normally continues. All errors of this kind are errors that could have been predicted (example: vector::at() reports exception when running out of bounds – you could have predicted this problem by checking the index before calling vector::at) or may be “totally unexpected” errors.

Of course, this above is true only when the library has been created with the principles of art. There should be also third reason: the procedure was written by an idiot and this exception was the only way to report an error, so you just have no choice. An example is IOException in Java, which is reported in all I/O operations including close() and it’s not implicitly propagated (it’s not a derivative of RuntimeException).

When we’re after the case #2, if you want to handle the exception, you should handle it always near the line between the subsystem and the higher level system unit. If you cannot extract the subsystem – just handle it on the highest level, or even don’t handle it at all (let it crash your program). In particular, handle the exception in this case only when you are able to recover and continue rest of the program. If you are not able to recover, then, well, let the exception crash your program – trying to “do something” in this situation is like trying to cure someone whose brain is already dead.

The leftist fool programmers

Error handling is a very underestimated topic in programming. Everyone thinks that if there is a potential problem, we should prevent it – so, for example, check if a pointer you have received in your function is not null.

Really? Check and what? Send an error message? Exit the application and lose all the data and process? Continue with a bit failure? Try to continue as far as possible and report error? Does your function consider failures in execution? Did the function that calls your function predict that failure and plan error handling accordingly? How can we recover from such an error, and – much more important – how can we determine such a problem on the testing step in the development process so that this problem won’t occur? Or maybe it’s still OK that this error occurred because it was caused by an external resource on which we have no influence?

How many times did you consider all these things when planning the error handling?

Why the hell should I check whether the pointer I have been passed to isn’t NULL? Ok, it may be reasonable for public API to prevent stupid users from doing stupid things (although I think C++ can afford for some better ways to do such a check). But, for example, why should I check that some function, returning a pointer, returns a non-null pointer?

There’s a lot of really stupid advices of how to prevent errors and problems in your application they may cause. Well, there are feminists, there are socialists, there are vegetarians, ecologists, anti-globalists… and the programmers also have their own version of idiots. They propose simple solutions for complicated problems and cause this way other kinds of problems. One of those stupid advices is that you should always use ‘strncpy’ function instead of ‘strcpy’. To see, how stupid this advice is, see an article on Maciek’s Inspirel site. This is a case of proposing simple statement kind of “you should never”, which is a typical leftist approach. The engineer approach can at most state “you should do this only if you have some logical explanation that things you do make sense”.

If you want to consider whether checking a pointer for NULL makes sense, first of all you should make sure that the function in the description in documentation states that it may return NULL and check in what situation it does it. If it declares that it does, and it’s not possible for you to prevent it, then you should check for NULL and in response take an action that is consistent with the error description (in particular, check which resources you need won’t be available and which operations cannot be done because of that). If the function returns NULL only because of some invalid input data – check the data beforehand. If the function declares that it never returns NULL (for example, the function returns a pointer to a common class pointing to one of 3 singleton objects), or it would return NULL only in a situation that you have prevented, then checking for NULL is useless.

Ok, I still should make a small exception of these statements because I base on another statement that programmers document their functions completely, while it is often, unfortunately, not true. So, if the function doesn’t have a complete documentation, or you suspect it’s out of date, or it says something like “returns NULL in case of error” and says nothing more about errors (which is everything all the same for a “user programmer”), the only way to go is to check the source code and see for yourself. And if even this thing isn’t available, you should gently refuse doing any work that is based on such a function, or at least sign a special declaration that you are not responsible for any bugs in this code.

In practice, people do “dumb null checks” just “to have a clear conscience”. People, who do it should rather go to psychotherapist. You should remember that one stupid null-check will not prevent even one tenth of prospective bugs in your application from destroying it (as I said, functions in C standard library do not check pointers to strings whether they are NULL). Instead of preventing the crash, cause the crash by reading the first 4 bytes from under the pointer – as long as the system features memory protection. If you really want to aim for preventing bugs in your application, you should perform checks on multiple levels (usually by static analysis), detect all possible cases of program’s states and behavior, and eliminate those that you don’t want. Ensure predictability by static declarations and flow analysis. If you really don’t trust the underlying code, try to run it in some separate environment. That’s really all you can do.

Also, if we’re talking about null pointer – almost every operating system has means to kill the process that tries to dereference the null pointer. So, are you trying to prevent an error in your application or prevent a crash? If you are trying to prevent some mistakenly returned NULL, you’re really doomed – by a simple statement that you are using a function for which you are not completely sure that it works. You’d say there’s no program without bugs? Theoretically you’re right, there is only one small problem: if you have a bug, it’s not possible to predict what the program will do. Are you trying to “do something” (the favorite saying of the leftists when they want to satisfy their thirst of solving problems) stating that you found this problem? Ok, do something – call abort(); that’s the best thing you can do.

Before you choose, make yourself conscious of what your choice is. If your choice is either crash or crash, then why do you think that the second crash is better than the first one? But if your choice is crash or unpredictable behavior – the crash is still better. If you are trying to prevent your program against unpredictable behavior in case of unexpected conditions – the only way to prevent your program from unpredictable behavior is to crash. At best you may lie to yourself that you can do something better (usually), and instead of calling abort() throw an exception, counting for that “someone above this place” will be able to save something.

In case of this dumb null check people often forget about one more thing: if you do any check for a condition that you think is critical, you should also plan some possibilities to test this “recovery code”, right?

When I was adding a feature to some application that should predict multiple access from threads and needed to have mutex locking, I was obliged to predict problems like not enough system resources or errors when locking/unlocking (the platform was not too much reliable). Handling of these errors was a bit complicated. Moreover because of that I have written test cases where mutex operations were stubbed and I could setup them to cause one of these errors. Having that I at least knew that even if these errors are very unlikely, there is some potential profit of this error handling code.

However I have seen many times some code, which I have traced only theoretically (just by navigation) and found that even though on some function levels these unlikely errors (reported by return value) are handled and propagated, after some call the return value was ignored. I didn’t need a test case to prove that this just won’t work. The error was maybe very unlikely, but if this happened, the handling just wouldn’t prevent the program from doing serious destruction. So why this whole error handling was added while it doesn’t prevent anything?

Moreover, this is one of the serious powers of exceptions. Exception is just something you can’t ignore without excessive handling code. The mission of an exception is to crash your program; handling the exception is to shrink the crashed part to a least possible range. The earlier you make yourself conscious of this unique treat of exceptions, the better it will serve your development work in case of error handling.

Imagine, for example, that with some configuration the malloc() function may call abort() instead of returning NULL, when the allocation failed. It’s a correct approach because people, despite that they usually check this returned value, they rarely do anything reasonable in response. Practically since the very beginning people were conscious that leaving the problem in hands of the user (that is, by reporting the problem by return value) may cause so many damages that it’s even better to crash the program without letting the user know. This may be the evolution of error reporting:

  • reporting the problem by return value; this requires to select an “erroneous value” as one of possible values, or return the value and status separately; however users may always ignore the error status and continue the program, usually leading it to calamity
  • call abort() in case of problems – this is a very drastic error report, but at least it doesn’t leave a user possibility to ignore the problem
  • raise a signal in case of problems – this usually ends up with crash, although the majority of “killing” signals give opportunity to handle them and do something better than crash; it proved to be a good solution for cases like division by zero
  • throw an exception in case of problems – this is merely like signals together with longjmp, with much better regard of non-trivially-destructible objects (C++ only) and language syntax support

So you should not consider whether it’s better to report by return value or exception, but rather: a) whether leaving a potential of ignoring this error is dangerous, and b) whether exceptions are better than signal handler + longjmp (they are).

The negligent make it twice

People intuitively throw exception in the last possible situation, not in the very beginning, before any significant steps were taken. That’s why catching exceptions is usually the wrong way of error handling. But when you check the conditions in the very beginning and take care that the “throwing” situation won’t occur, what’s the reason of catching exceptions?

For example, imagine that you hear: before you go to the till, check if you have enough money. Well, good advice; you have collected things you want to buy, you brought them all to the check-out point, just one small thing – check if you have enough money. And what? Leave them there and go out? Get them and return money later? Resign and repeat the shopping after you took enough money? Of course, every sane man will response: why not check for enough money before shopping? This way you don’t waste time to perform the shopping, if you can’t finish this operation correctly. But if you are sure that you have enough money, when you are currently shopping, why to check this condition again when going to the till?

Of course, you should have the “last chance to crash” in such situation. However, when this occurs, the only thing you can do is to drop all things you have and run away. It’s way too late to do any “fix” that would allow you to continue shopping! You may fix it by going to a cash disposal terminal, but not when you are at the till; this can be done only before shopping!

The clue of the problem here is that if you have an exception in some place, it doesn’t mean that the problem happened there – it only means that only there you had a chance to see it. In other words, it may be a result of something wrong that already lasts since some time. I’ll give you another example:

If you index an array, and you rely on exceptions – should the exception be raised in a place where the array is indexed out of bounds, or rather in the place where the invalid index value was created? Additionally, in Ada language arrays may be specified the integral type for indexing. This integral type may be a limited type, where valid values are exactly those that match all valid indices in the array. Having this, there’s just no way to index out of bounds because the out-of-bounds index value has no chance to be created.

But wait. This doesn’t mean that the problem disappeared. Of course not – it just occurs in different place. But it definitely occurs earlier, that is, closer to the place where the problem happened, which gives you probably a bit more chance to recover. For example, if the index comes from some external procedure, you can just state that the procedure got stupid and ignore its request, continuing rest of the program. While when it happened when you are indexing, usually you can only crash.

Please note first of all that many these exceptions are thrown in an “unexpected” situation; they just throw exceptions because they can’t continue or they do their best to prevent your application against going out of control – but when this happens, it’s usually way too late to do anything reasonable. There should be then some possibility to detect this situation a bit earlier, and if it is, you should do it. In particular, when you are going to perform some set of operations that need some resources, check for all resources whether you have them just at the place where you may have them, also check for conditions to be satisfied in the first possible place where you know what to check for.

For example: if you want to read some data from two files during your job, open both files in the beginning and keep them both open even if you don’t read from one of them for some time. You may spare a lot of resources not wasting time to perform your operation (and not giving a user hope that you can do it while you can’t) that would fail nonetheless in case one of files can’t be open. Then, once you are sure that both files exist and can be read, start your operation. If then there is any problem during reading from any of these files, it will be reported, but this is something you couldn’t check before starting your operations (by a case, this is a kind of error, which is very rare comparing to nonexistent file).

Only if you plan your error handling with this in mind, can you say that you really have control over your application.

Errors occur, problems happen. Mind the important difference between “error” and “problem”. When a problem happens in an application, there are several things that may happen:

  • an error is reported immediately
  • the problem causes invalid changes in data, which causes another operation to cause a invalid changes in data, which… finally some of them report an error
  • same as above, but it never ends up with an error (but does a lot of mess in your program)

In various VM platforms (like JVM or .NET) many works were done in order to make the last one never happen. First, they throw an exception when null pointer is dereferenced. Second, they ensure that in case of normal references, every non-null reference points to an existing object, and also throw an exception if referring to released object, in case of weak references. But as long as you are using universal data types (and too many people forget how universal the ‘int’ type is), you will never prevent the second one. And even though you finally catch an error, it’s usually far too late to do anything, even dumping the data to some file, because the data may have been already corrupted. Because to detect problems early there is needed a data type with weak abilities to “carry the problem” (which means, as you know, that it should be less expressive).

So, if you think that enough things have been done to prevent any problems to last too long in languages like Java or C#, you’re completely wrong. They only prevent data mess done out of the frame of current data types. But there are still a variety of errors that can be done in the frames of current data type, especially if these frames are so wide, as it happens with ‘int’ type. Using a simple ‘int’ you can index arrays out of bounds, you can refer to an existing object that was just moved to a different index (did you think that “iterator invalidation” cannot happen in Java?), you can refer to an object you have remembered in a local variable instead of reading this pointer by calling some method. And the VM will not prevent you from doing these things.

Think well, then, in which place you’d like to hold on because of some erroneous situation, and rely on that the granted condition will not unexpectedly change. Because if you don’t think well, you’ll get unrecoverable errors with undefined program behavior, and you’ll have to make your job twice.

How APIs can increase your bug risk

Imagine that you call some function fx() which returns a pointer value and it’s specified that it may return null pointer value in some specified case. Then you call another function xset(), which accepts a pointer value for the same type and it’s specified that this function may accept null values and this means some-special-conditions (in short: fx() declares that it MAY return NULL and xset() declares that it accepts NULL values). Now imagine that you do:

xset( fx() )

And? Will xset() function do exactly what you want for any value that may be returned by fx()? In other words, you probably want to pass a valid pointer from fx() to xset(), but did you really want to pass the null pointer value to xset() when fx() returned null pointer value?

If not, it may end up with a problem in your application that you haven’t detected.

This kind of error may happen in C, C++, C#, Java, Ruby, Perl, Python an many many others (Vala is a notable exception due to non-nullable references by default, however if you operate with nullable types in the above operation you’re in the same situation as in the others). You can’t do it in Tcl, but in Tcl you can do similar bugs like having an empty string in a variable, in which you expected an integer. Pay attention that there’s no way to detect this kind of error in the above languages, even in runtime.

The same happens, in Java this time, if you do:

 char c = characters[str.indexOf(match)];

If you have called String::indexOf() then the return value may be -1, if the searched occurrence was not found. Unfortunately, it’s next passed as an index of an array, which will end up with IndexOutOfBoundsException. Well, in this case the problems were still prevented, though maybe not with the most readable form. But this exception will not prevent errors in every case. Now, grab your chair, and watch this:

 char c = characters[str.indexOf(match)+10];

This is not fiction. This is a real bug that I have once found.

It’s best if the error occurs exactly in the place, where the problem happens. Unfortunately, errors occur usually far forth to the place where the problem happens, that is, in other words, various instructions transmit the problem and throw it to each other like a hot potato, until one of them throws it to the ground (that is, until the error occurs). If any, needless to say. Comparing it with the example with array indexing: a problem had happened way before the array indexing: an invalid index value was created. If we do it in Java language, we’ll get an integer value, which maybe contains invalid index, but we’re not yet indexing – only when we’re indexing, the JVM will throw the out-of-bound indexing exception.

But the problem isn’t always when we’re indexing. The problem may happen when we are only computing this index. Or computing it from some else value, that was taken from some else place… and so on. The further we are from the real place where the value was spelled up, the harder it is to find the real problem. And exceptions are thrown not in the place where the problem happened, but rather when hiding this bug under the carpet was no longer possible.

Of course, you’d say, this should be then handled by exception!

I know that in Java standard libraries there is a custom of making the most possible stupid use of exceptions, but this one, despite that it can provide so dire problems, is an example, where this stupidity didn’t reach.

It shouldn’t be handled by exception because the indexOf function is not meant to retrieve some existing thing from a string that you know it is there. It’s meant to search for an occurrence in a string and provide you with the answer whether it is there or not. The answer that this occurrence is there is same good as the answer that it’s not – the -1 answer is a bad answer only from the perspective of the user of this function, if they know that, according to the application’s logics, this occurrence must be there. If such a response was handled by exception it means that the guy who was writing it was a complete idiot. It might be handled by exception, but only when – for example – the indexOf function is passed an additional argument, which has a name kind of “THROW_EXCEPTION_IF_NOT_FOUND”.

So, is this return value still good?

Not exactly. It’s good that this is not reported by exception. It’s bad though that this return value is a good looking integer index value, if mistakenly used in an expression. The best way is to return some “value”, or both the index value and status, and make sure that the status has been checked before the value was used – maybe even that would throw an exception in case when a user mistook the value. In Java you can use Integer class (this will make use of the “optional” property of reference type in Java that I have already described). In C++ you can use boost::optional or some similar technique (say, a pair of bool and this expected value).

If you ask what should be then done in case of close() (in Java’s streaming classes), which may want to report error (by exception), things that should be done are the following:

  • this may throw only when the flush() operation can’t be finished (in order to complete writing) – just try to flush first and you should be fine
  • ok, so flush the buffers and close() should not throw any exceptions, you can ignore any errors because no errors will be reported

What? Really all errors?

Not exactly, but we really should not worry much about it (it doesn’t mean that you can leave the catch body empty – it only means that you should rethrow this exception via RuntimeException). Really. Well, let’s explain it in more details.

Levels of errors

We have various kinds of errors and which way of making this error possible to recognize at the right place depends on this kind.

As long as the “level of logics” is concerned, we have the following kinds of errors, that is, problems that have happened in a program:

  1. Preconditions to be checked beforehand (level 3 – preconditions)
  2. Problems that were found because the preconditions are not satisfied (level 2 – negligences)
  3. Weird errors that occurred in some category where they should not have happened (level 1 – calamities)
  4. System damage (level 0 – cataclysms)

Before you approach to predict any problems that may happen to your program, the first thing for you is to make yourself conscious that the only levels you should worry about are levels 3 and 2. You MAY be prepared to handle errors for level 1, but please, don’t be so naive that you can normally continue your program when they occur; try rather to close your application kindly, safely, and first of all quickly (before the system will kill it). If you get a problem of level 0, please call abort. No, not exit. Really you should call abort unconditionally. Don’t even try to check any data, “norover” saving them!

What kinds of causes may be assigned to particular levels?

Level 0 is something that is in the system, including hardware. For example, if reading of memory causes weird data to be read, then as long as you are able to detect this problem, you should crash. This can be also caused by lacking system resources (for example, you can’t create a semaphore), or sometimes by lacking some basic system service. All these things are resources that we believe that are crucial not only to our application, but just for everything that is running on this platform. That’s why we believe it will never happen. Some APIs allow to report errors of this kind, but usually you should handle them by simply calling abort(). APIs for languages that support exceptions should use exceptions to report them, however it’s hard to find this because these errors are usually reported on C level and there are no many C++ wrappers that translate them to exceptions. Generally it depends on how reliable the system is meant to be – the more reliable the system should be, the more kinds of errors are of level 0.

Level 1 concerns errors that happen usually because of some external conditions that the user couldn’t check before, but still not something that is not predicted to work. In other words, these are “user level” errors that should never happen, but not because of crucial system resources. It may be because of some problem in the library, maybe it overestimated something that it will work or by some external service that is needed by the application, and somehow it doesn’t work (but the service isn’t any crucial part of the whole system). For example, transmission or network errors are unexpected problems – how important it is for you, depends on how important the network connection is for you. This level also contains the cases when exceptions of level 2 were thrown, but they shouldn’t have, as the preconditions were satisfied. They should be handled by exceptions, although this may also be reported by return value optionally. However if return value is used to report this error, the error state must be remembered so that the next operation will not be tried.

Level 2 is the already mentioned “last chance to crash”. This is a report of a condition that should have been satisfied before calling some API, it wasn’t, and the API did not accept the conditions under which it was called. Errors of this kind should be at best reported by exceptions and handled on a high level. When this happens, it’s usually too late to “do some reasonable alternative”. An exception thrown at this level is always because of a “programmer’s negligence” because they could have prevented them by checking required conditions (level 3).

Level 3 is just the condition, which may be satisfied and may be not, both results are acceptable, maybe not in the program’s logics, but in the API logics. For example, your program is working with text files that should have some specific format. You should read the file and check if it has correct format; if it doesn’t, the program should display some information for the user about invalid usage and then normally continue. This exposes a “fork situation”, as we expect that both alternatives are more-less the same good or bad, we just make the program behave different way. This kind of problems should never be reported as exceptions as the only alternative; they should be always reported via return value, although reporting via exceptions may be provided as well, optionally.

Note that optional exceptions thrown at level 3 is a thing provided only for user’s comfort. It’s important to provide this alternative because for the API designer it’s not possible to predict whether this unwanted situation will be for the API user just some alternative situation or a serious error. Reporting level 3 errors only as return value will cause the problem described by the first example in the beginning. On the other hand, reporting them only as exceptions will cause another problem, described by the second example.

Remember that the topmost reliability of your software will be such as the reliability of the system and additional resources you use. Your application can be 100% stable by itself, but if the system or additional resources cause problems, you can’t do much, and practically, you can’t do anything. So the first rule for error handling is: don’t fix the system and dependent libraries. If your system is little reliable, change the system; if your library is not reliable – use another library.

That’s why you should ignore errors of level 0 and 1, letting them crash your program when they occur. Users should focus on levels 2 and 3.

Generally we can define the system environment as:

  • the basic services – crucial system resources that must always work, resources provided by these services must be always provided every time they are requested; at most there may be some limits defined which must not be overrun by the application; errors when allocating resources from these services, as long as the preconditions are satisfied (if not – level 2 error), are level 0 errors
  • the extension services – system resources that are not crucial and there are predicted situations when the resources may be not available. Depending on various conditions, errors when using resource from this level can be: level 1, if the resource wasn’t available, but it should; level 2, if the resource wasn’t available because a user didn’t satisfy some condition that was required; level 3, if the resource was never granted to be available and, well, it just wasn’t. The difference between level 1 errors here and level 0 errors from the basic resources is that the level 1 means that there are problems with applications of little importance because of some extended services’ damage, but the crucial part of the system is still functional. Level 0 means that the system is totally unstable.

Let’s try to classify particular types of errors:

  • can’t open a file with a name for which we didn’t check it exists: level 2 because you could have checked whether it exists before opening
  • can’t open a file, despite you have checked that it exists and can be read: level 0 because this is not your fault, nor is it fault of any unpredictable condition; it’s a system’s problem
  • can’t write to file because the file size exceeded the limit: level 2 because you could have checked the limit beforehand, so it’s your fault
  • can’t write to a file stream, despite that you have this stream correctly opened and no limit has been exceeded: level 0 because writing to files is a crucial system resource
  • can’t connect to a socket because the network is off: level 1 because it’s an error that depends on some external conditions, so it’s beyond the system’s basic guarantee
  • can’t connect to a socket because you passed incorrect parameters for the operation: level 2 and you should know why
  • can’t connect to a socket despite that you passed correct parameters and network is working: level 0 because this is one of the crucial system services that have strong guarantees
  • can’t read from socket because the network connection has been dropped: level 1, by the same reason as when trying to connect a socket when network is off
  • can’t read from socket because there were no data sent to the socket and the socket is working in a non-blocking mode: level 3 because there are no guarantees that the nonblocking socket will ever provide any data to read
  • can’t read from a stream that was not open for reading: level 2 because, as previously, it’s your fault
  • you search for an occurrence of a string in a bigger string and it wasn’t found: level 3 because no one granted that this string is there
  • you dynamically cast your object to another type and the dynamic cast failed: level 3 because no one said that this object is of this class unless you check it out

There can be, this way, the guarantees provided for that particular errors will never happen:

  • level 0: strong (internal, basic) guarantees: it must always work or you have a total crash
  • level 1: weak (external, extended) guarantees: it should work, but if it doesn’t it’s not a critical problem
  • level 2: your guarantees: if you have done your job well, this error will not occur
  • level 3: no (or private) guarantees: no one said it will ever work – you can try and you may succeed (or maybe you had given some additional guarantees that it will work, but this is your matter and your problem if this isn’t true)

However, please note that a lot of system API in today use has been created at times when system haven’t been given any strong guarantees. Because of that even problems like “I can’t create a socket” by whatever reason (no matter if it’s “incorrect parameters” or “resource temporarily unavailable”) is reported by -1 return value (which is normally the descriptor ID). It means that you may even allow some errors to cause other errors (for example, blindly use the value returned by socket() function in the next call to read() and respond on the error reported by read(), which reports also problems with invalid descriptor – this is practically the only way to go when you use network descriptors in multi-threaded environment). If the only system API that you have reports every error by return values (not even by signals!), it simply means that this system (theoretically) provides no guarantees about anything – in other words, everything may crash. A programmer, who must use such an API, is in a very bad condition because according to the rules of this API they must be prepared for a situation that nothing works. The need to worry about all the errors, including those that happen in 1% of cases, causes that the user is forced to spend 90% of their coding time with planning the error handling, so effectively it results with very poor error handling. Then, being accustomed to such kind of error, makes a user do “bug prevention tests” like those mentioned in the beginning, which causes very poor quality of the software.

Remember also that every handling and recovery that you plan must be tested. So you must be able to provide stubs for also system functions so that you can make them intentionally cause error, so that you can observe whether your program correctly responds for this kind of error. If you’re not going to test your error handling – don’t do error handling (I would even say it’s obvious that if you’re not going to test your code, you should not write it). Do a crash instead. I’m telling you truth – letting your program continue after the error will make it end up much, much worse.

I think this is one of the main reasons why termination semantics has been accepted in C++ and preferred to resumption semantics.

Exceptions should be exceptional!

Concerning the levels described above and skipping the 0 and 1 levels, we have two aspects (levels 2 and 3) in which we can consider whether in particular situation the problem should be reported by exception or by return value.

When you want to obtain some resource, which requires some preconditions, the correct way to go is the following:

  • call the resource checking function
  • if this failed, do not perform the operation and do some recovery
  • if this succeeded, perform the operation, IGNORING any exceptions (allow them to self-propagate)

The general rule for using exceptions is: make a user of a choice to not handle exceptions if they don’t want to do it. This concerns errors on every level:

  • level 3: provide an alternative (and default) API that reports errors via return value
  • level 2: provide a user with complete information about how to prevent this kind of errors
  • level 1: let it be something that in normal situation would immediately call abort() – by using exception you only give some users possibilities to limit the number of problems caused by the crashed application
  • level 0: the library should not report anything, even via exception, but do abort()

In a very short description: treat reporting exceptions as something “better than abort”.

The optional exceptions for level 3 may just provide a bit more comfort for the user so that they can build simple and clearly looking expressions. For example, problems when the key wasn’t found in the tree-like dictionary may be handled this way:

Value p1 = dict.find( key1 );
if ( !p1 ) return;
Value p2 = p1.find( key2 );
if ( !p2 ) return;
Use( p2 );

or this way – if you think that it’s more comfortable (usually it is, and it doesn’t bring a problem of interrupted-in-half operation):

try {
   Use( dict/key1/key2 );
} catch ( KeyNotFound ) {
   return;
}

The semantics of ‘find’ are that we are only check if the key is available and return status of this operation. But we may optionally want to get the status information via exception or no exception just to have the operation encoded in a simple expression using overloaded ‘/’ operator.

BTW., please note that if exceptions in C++ had been predicted to be used for anything else than unrecoverable errors, then they would have been designed with possible “resumption semantics” (as Common Lisp has). But it was chosen that exceptions in C++ support “termination semantics” only. You can refer to “Design and Evolution of C++” for details. I’m talking about C++ because only this language’s designers were reconsidering this topic – designers of Java and C# didn’t think at all, but simply copied the C++’s solution directly (the “finally” clause was also provided first in Borland C++ as extension). Of course, if you ask them, they will deny and say that they took them from Haskell :D.

So don’t let yourself be said that error handling is exception handling. Error handling is early checking in order to prevent operation from being executed in unacceptable conditions. In other words, error handling is preventing exceptions from being thrown (not from being propagated!).

Why strict exceptions in a language are bad

The developers of Java language think that the best way to report problems from the procedure is to throw exception. This is the first stupid idea (no matter that it’s not strictly adhered to because, as I have shown, the indexOf method does not throw an exception). And next, they think that the exception should be declared to be propagated between calls for every function that’s on the call path, in order to ensure that no exception will be left unhandled (as if it was impossible to ensure the same thing without requiring to explicitly declare propagation – especially that Java compiler does not use the C linker to complete the application, so it is in even better situation than C++). These two interconnected are dumb in power of two.

Think a while about it. If Java forces you to handle exception in case of a least problem (with no option to handle it by return value) and forces you to declare the propagation explicitly, it means only one thing: exceptions reported in Java libraries are expected errors. Because if they were unexpected, they would be thrown only when there was some predictable problem and not needed to be explicitly declared.

Above you have possible situations when exceptions should be used. If you now compose it with the above statements for Java, there’s one thing clear: the main idea of exceptions in Java is to handle errors of level 3 (not 2!). That’s more than clear. Because if this was any kind of unexpected problem, it would be handled by a derivative of RuntimeException.

As you know, strict exceptions were introduced in Java in order to provide ability to prevent unexpected exceptions (that is, prevent at compile time a situation that is handled by std::unexpected() in C++ – but remember that in C++ you can at most narrow the range of exceptions allowed to be propagated, while in Java you can only widen the range of propagated exceptions). Authors of this statement seem not to realize the depth of their stupidity: they think that there is a situation, when an exception should be expected???

Yes, an exception may be expected. Only in one situation – when a problem in a function will be able to be reported only and exclusively by exception, and you should catch the exception in place (as you know, catching the exception in place is simply translating exceptions to return values). If you ever have a situation that you have a need to handle the exception in place, it simply means that the function that throws this exception was written by an idiot.

No, I’m not abusing anyone. Read thoroughly any book that was written at the time when this idea was being developed, or documentation of any of old programming languages that were added this feature. You’ll find the same thing everywhere: the main idea of exception is to handle the problems automatically on higher level than the level where they occurred, and handle them once in a group (not in every place when they were able to occur), and this can only be achieved when the exceptions can be freely propagated. Do you imagine, for example, that in the first languages that featured exceptions (e.g. Ada) the only possibility to “catch” exceptions was to install the exception handler only once per function? It was only C++ the first language that introduced the try-catch block and allowed this way that you can have exception handlers in one function in multiple places or even on multiple levels.

Let’s consider this sorry IOException in Java thrown when some IO operation (read or write) was failed. The stupidity in this case does not rely on that this problem is handled by exception (as you know, this is a level 1 kind of problem, so it should be handled by exception). The problem is that this kind of error shouldn’t happen in normal situation, so it should be handled on a high level as something unexpected. Unexpected errors are similar to errors like NullPointerException. Why then isn’t IOException a derivative of RuntimeException?

If there is some situation when limiting the exceptions is needed, a user should decide by themself whether they want to catch only selected exceptions and do a crash in case of the others, sink exceptions into one other kind of exception, or let every exception to propagate. In Java, not only must you explicitly declare propagated exceptions, but you also can’t add new limitations for expected exceptions. For example, sometimes you’d like to let IOException propagate very “highly”, while sometimes you’d like to not let NullPointerException out of your function.

The first and foremost dumb idea standing behind strict exceptions is that “free exceptions” may easily lead your program to unexpectedly crash, so there’s a need for a tool that will prevent these crashes (if you have read my earliest statements thoroughly, such an idea sounds now very funny). It’s still stupid, but even stating that it has some little reason, why a compiler and the set of language rules must be such a tool? Why can’t there be some other (external and independent) tool to be used? Let a user write the (trial, “proof-of-concept”) programs as they want to, let them even unexpectedly crash. But let there be also an additional tool to detect possible unexpected exceptions.

Why such a solution would be better? Well, because the main drawback of strict exceptions, like with any other thing supported by a compiler, is that if it requires too much effort to be cleared using the “good way”, they will be cleared “evil way”. This is like fixing const-correctness in C++ – if it requires too much effort, users will end up with const_cast. It’s also like too strict rules for passwords: if they are too hard to remember, users will write them down in some visible places, causing this way that the system is much more vulnerable than with less strict password rules. In Java, if the compiler can’t let you go until you handle an exception X, people will handle it with empty body, just to make the compiler shut up. This empty body will then stay unchanged because there is usually more important things to do, and will be possibly quickly forgotten (especially when tools like Eclipse allow to automatically add the code for exception handling to fix the code that does not compile – of course, exception handler’s body isn’t provided :)).

Ok, but why am I talking about this? Will my complaints for strict exceptions change anything? Can I hint any solution but using another language than Java?

Yes. You can easily stop using strict exceptions by just not using them, that is:

  • don’t specify exceptions in your methods
  • when a library function throws an exception, create your own wrappers for this library that will not throw declarable exceptions
  • if it’s too much work to create these wrappers – catch these exceptions in a reasonably near place to where they are thrown and rethrow them via RuntimeException
  • all your exception classes should extend RuntimeException, not Exception

The only problem you may have is that there’s no software that would detect the “unexpected exception” situation and you can’t even use the exception filter like in C++ (indeed, Java’s exception filter is something completely different).

The system matters

There are three general levels considerable by the application, and on each level there are some important rules that should be kept in order to have a good and clear error handling:

  • basic system level: the API designer must explicitly define guarantees and possible preconditions for the resources provided on this level; the API user must strictly adhere to these preconditions if they want to be granted anything
  • extension level: the API designer should provide exception-based error handling in a “should not continue” problem and describe possible guarantees; the API user should be conscious about the weak guarantees provided by this level and be prepared to respond for a situation when something doesn’t work
  • application level: the API designer should always provide return-value-based error handling in the API and possibly, for the user’s advantage, some exception-based alternative; the API user should remember to adhere to the described preconditions, make sure they are satisfied, possibly install exception handlers if they think there will be any use of them

Separation of checking file existence and opening a file is important – if you, for example, use two files to read and one file to write results to, you should check whether you can open the first two files and whether you will be able to write to the output file before you start the operation (imagine: you read 10GB from one file and 20GB from the second file and then after 1 hour of reading you realize that you can’t open file for writing because – guess what – yes, you don’t have privileges to open the file, for which you have been given a path). You’d say, then, what about problems that may occur during your program is running? For example, the file existed in the beginning, but when the program is trying to write, the subdirectory of the file to be written was deleted? Well, these things belong to the other category: system safety level, interaction safety, trust to the other code being currently run (they are all level 1 errors). So, if your program should not work in “untrusted” conditions, don’t be so strict to predict every possible failure. Sometimes you can let your program crash, especially if it’s running in conditions for which this program was not predicted. And if your program is running in conditions that were not predicted, it means that the problem is on much different level and your program is most likely not the only application that has problems.

Note that opening both files once you know this is possible is a solution, but probably not in every system. In POSIX systems, for example, you can open both files, keeping the writable file uselessly open for some time, because even if some other process deletes the whole directory where this file was located, the file will be at most unlinked, so I/O write operation will succeed anyway. In Windows it’s a bit different, that is, the system will not allow the directory to be deleted because the file can’t be deleted as long as it is open. But there still may be a system, which does not protect the written file and the I/O write operation may cause runtime error. The difference is very important – this last system gives weak guarantees for I/O operations this way.

If you have any doubts whether I/O operations should report level 1 errors or level 3 errors, then remember, in which practical situations I/O errors are reported. Because both may be reported! For example, the I/O errors in many language’s standard libraries may be also errors caused by incorrect formatting, e.g. reading a series of ciphers that could not be turned into a valid int value. Thinking about writes, oh, maybe transmission errors? Forget it; not in today systems. In today systems the memory and filesystem are integrated and if the system cannot write to disk because of some disk damage, it’s system’s problem, not yours. The system must fix it somehow and ensure that you are always able to write to a file. The system never reports transmission errors. Different situation is when your device is not system-integrated – some external device or network. In this case you may at most get errors like… EOF. That’s all. For example, if the device is damaged or the network is broken, the file (even being written) will receive EOF. If you check what I/O error flags can be set in C++ standard library, you’ll see that there are three: failbit, which usually means formatting errors, that is, high level operation problems, badbit, which should practically occur when the stream operates with a device with weak guarantees, and eofbit when EOF was reached. It simply means that, for example, in POSIX systems, when operating with a file stream, the badbit flag will never be set. The creators of C++ standard I/O library were rather not aware of the shown above levels of errors, but feeling that both exception and return value may be useful depending on some conditions, they left the mechanism very flexible. Because in practice eofbit and failbit are level 3 errors, while badbit is level 1 error.

What we can do to make error handling better

If you don’t have bugs in your program, you should let prospective exceptions of level 2 be propagated without handling and be sure that your program will never crash. If you have bugs in your program, celebrate if they only cause a crash. If you don’t want that your program crash, eliminate code bugs – not block exception’s propagation! If you really want to eliminate unwanted state and behavior, there are two general good practices for that:

  • create a possibility to run a code in two different configurations: release configuration, where you allow errors to destroy your application, and debug configuration, where you catch any possible problems early, but don’t worry much, what would happen to the application in this case
  • use a lint-type tool that would perform thorough verification and detect potential problems and hint appropriate places where runtime verifications should be provided

So, first thing is to make your code prepared to run in a ‘debug’ configuration, where you verify everything, and in a ‘release’ configuration, where errors may destroy your application. So, if you think that some code may potentially be distrustful, just verify its results in ‘assert()’s. This way you still have possibility to verify whether the code ran correctly (when you run in a ‘debug’ configuration), but you won’t overload the code with a lots of useless verifications, when you run in a ‘release’ configuration.

Why to use verifications only in ‘debug’ configuration? Because in this case this is usually unimportant how you plan the recovery from errors. You can crash, you can stop in a debugger, you can display error and go on, even risking destruction of your data – you are not running in a production environment, but in test environment, so even most dangerous errors shouldn’t matter. If you state the matter this way, it’s really unimportant how thoroughly you have planned the error recovery. For this case there is only one thing important: every problem must cause error in the earliest possible place where the problem happens. Moreover – you are testing the application to make sure that a) every problem is reported as early as possible, and b) every error that was caused by internal reasons will not happen in any possible scenario.

And second good practice is to use lint-like tools. These tools are compiling your code according to your compile rules and create a special form of resulting files that will be used to perform a complicated verification of your code. This way you’ll be reported all dangerous situations like possible indexing out of bounds, reading an uninitialized variable, dereferencing a null or invalid pointer or uncaught exception. And it will also determine whether this problem has no chance to occur and won’t report problem for this case.

Once you have these things checked, it’s better that you trust to the API that was provided. Distrustful you should be only in case when you are running something REALLY distrustful, that is, the code that is executed may be even especially predestined to destroy your application, gather secret data or cause harm to user’s systems. Such an distrustful code should be run in an incubator environment, with limited access to particular elements of the system (in this statement, applications running as JVM code is a good example, but not the only good example, of course). Such a code has no chance to be verified beforehand by neither running in a debug configuration, nor by lint-like verification. And therefore it should be run as an interpreted code so that any problem it may cause will not cause problems to the application.

If you cooperate with another application, it depends whether you really need it for your work, or this is just an optional functionality. Anyway, when the interoperability malfunctions, the functionality that relies on it can’t be used. In effect, it only depends on how important this functionality is. In short: if your application cannot live it without this service, make it suicide; if not, well, make it cut off its hand and keep it alive. But the problem of failed service cannot be fixed by your application!

Always remember that you pay for every additional verification because of dis-trustfulness. But if you verify, be sure that your function is really prepared to run a code that may fail. The verifications should not be added because an application may potentially fail, but because this failure means something specific for your function. If you really want to be distrustful, you should not write programs at all, and you shouldn’t even run cars, travel by planes, give your money to a bank or even watch television. You just can’t prevent every possible failure. And in many places verifications won’t help you much, if something has been really direly destroyed.

Bottom line

It’s not a problem to handle exceptions, like there’s nothing simpler that you can do with a pointer value than checking it for null. The problem is what to do in the problem handling procedure. Before you program a verification procedure, ask yourself, what would you do in such a case. In other words, what’s the level of meaning when particular problem occurs.

Don’t also fall into easy simplifications that you should verify every pointer value against null or that you always should handle the problem in the place it occurs. You should, however, recognize the level of the problem and take according steps: maybe you should propagate the error as it is (in case of exceptions, just allow an exception to be propagated), maybe you should do some recovery in this place, or maybe you should report another kind of error. Maybe also sometimes it’s enough that the execution is broken and no additional steps have to be taken.

There are no simple ways to handle problems and errors. Every case should be treated individually, although the errors can be generally classified in four levels of errors. Coding standards may at most specify what exceptions should be used, how to propagate problems between execution levels, how to manage resources and how to report problems with particular resources. But it should never specify what problems should be ‘always’ verified, where to handle errors and what to do if an error was found.

On the other hand, the decisions whether particular situations should be reported by exception or return value is really simple (so simple that it’s even strange why so many libraries break it): use only exceptions, if the situation to be reported could have been predicted and prevented from happening, and use exceptions as an option for return values to report any kind of error. In other words: your library should not force user to handle exceptions – the program should be able to be written and should have notions of working even if no exception is caught.

The main and most important thing when doing error handling is to focus on errors that are likely to happen and plan error recovery for them thoroughly. Exceptions were predicted to make this thing possible, not to get into a degenerated API as the Java’s standard library, which is the main reason of why many good programmers despise exceptions.

Posted in Uncategorized | 1 Comment

Multiple inheritance considered nice

0. Introduction

This article describes logical aspects of multiple inheritance and should clear your doubts about whether multiple inheritance (especially in C++) can be logically consistent.

Multiple inheritance, another buzzword in object-oriented programming, is currently treated as a “dire” feature. One meaning of it is that it shouldn’t be supported (and this is what Smalltalk, Java, C# or Vala do), the other is that it’s “not for kids, but for professionals”, so it’s supported together with conflict resolution, which leads to stupid and unmaintainable code (Lisp, Eiffel, Python), and effectively to ban multiple inheritance. I would add C++ to this last list, however it at least does not do this stupid conflict resolution and additionally features virtual inheritance that allows to prevent “diamonds”. None of the languages does the right thing, though, that is, strictly disallow diamonds and implicit common base method overrides.

The main problem isn’t the multiple inheritance. The main problem is that people seem not to understand the OO inheritance itself.

1. What is inheritance?

Yes, well, let’s start from the beginning because there’s too much people who have their own idea what inheritance is and, moreover, what inheritance is not.

I should first define, what class is, of course, but let’s not get to such a low level. As everyone, who is familiar with OO, knows, class is an entity with statements, which defines what a potential object of this class should do and which way. Having that, inheritance is a relationship between two classes, which means that the “source” class (called “base class” or “superclass”) plays a partial role in the “destination” class (called “derived class” or “subclass”).

Please pay attention that this relationship, despite being similar in result, is realized different way in dynamic languages (like Smalltalk) and in static languages (like C++). Let’s say, universally, we have a class A and we create class B that inherits A, then finally we make an object O of class B.

In Smalltalk this relationship means a kind of “fallback delegation”: whenever we call method M for object O, and the B class does not define M, the A class is tried to do M. And so on towards the bases, until Object class is reached, and if M is still not found, it ends up with exception.

In C++ inheritance really means derivation – if we define B this way, then all methods of A are meant that they also are members of B (every method identified as A::M is also identifiable as B::M). There is no such thing as “fallback” – all methods of A are incorporated into B, and when there is a method M called on object O, the method is “searched” in the B class only and exclusively.

In Smalltalk then, as the only type in this language is a reference to object, you can just “try” to call some method you think this object handles. The class of which this object is is stated to provide some features, so when you define your own class, you usually would like to provide another implementation for what the object’s class has already provided. The main treat then seems to be the possibility to define my own methods with the same name as those defined in base class and this way change the behavior of some method from the base class, which was calling this method in its body. That’s probably why the fact that B inherits A is called “subclassing” in this language.

The perspective of C++, despite that is exactly the same in effect, makes different impression. In this language the main treat of inheritance seems to be “extending” classes, that is, making classes import contents of another class and take it as their own. Or, in other words, add new things towards the base class. That’s why probably it’s usually called “derivation”.

However it’s maybe not explicitly visible that both these things result in the same, that is, both aspects are important: “subclassing”, when you partially change the behavior of the base class, and “extending” when you add new features towards the base class. Inheritance is both these things together. However in Smalltalk it’s not obvious that “extending” occurs, especially that in Smalltalk, as it’s a dynamic language, it’s hard to extract partial contents of the object, that is “subobjects”, which is normal in case of static languages. But anyway, in any of these languages you can “override” methods from base classes, as well as add new features to a class.

Stating now that Java is a Smalltalk descendant with added static types and C++-like syntax, it looks moreover stupid that the “extends” keyword is used to define a feature that by OO purists is called “subclassing” (in Smalltalk it is “subclass:”!). Moreover, the “implements” keyword would also be treated as subclassing (especially when an interface can be a type of reference, so it can be used for “classification” as well!). It was then much better that C# followed the way of C++ by not using any keyword, since any keyword there would look stupid.

OO purists (would say “mind purists”, who have their minds pure of common sense) usually say that extending is only a “side effect” and the most important thing is subclassing. They just don’t realize that there is also “extending”. The majority of people simply do not understand that inheritance in general is these two aspects.

Some people say, for example, that you should not derive, if you are not going to subclass. But why not? I want to have an object that will do things I need, and partially these things are fulfilled by some existing class. Why then, shouldn’t I derive?

Or, what should I do instead? Create another class from scratch (oh, maybe I should copy-paste the code from the class that partially does what I need)? Should I delegate, that is, place this “base class” as a field and define method forwarders? This will only result in the same thing and will cost me more effort. Or maybe I should create external functions with the code I meant to put into the new methods? That’s a good alternative solution and has some advantages (for example that multiple such solutions can coexist without conflicts), but also it can’t be used when such extending requires additional state particles in the object (implemented by fields).

In Smalltalk the problem is also that once you defined a class, its fields cannot be reused. In Smalltalk the matter is very simple – there is no syntax for exposing fields (even for derived class). In languages, where fields can be public and there is also a “protected” section, which makes parts of the class available for deriving class, it’s possible to create such an “extension” towards the base class that makes use of some fields from the base class. Well, I can understand that “protected” can be treated as a “wtf kind of encapsulation”, but I don’t understand why exposing fields has to be “non OO”. In particular, I don’t understand why exposing or hiding fields has to be different thing than exposing or hiding methods. Especially when “properties” is growing to a rank of feature, either “logically implemented and partially supported by some tools” (in Java) or even built into the language (C#, Vala, also Qt extensions to C++).

If you explain the things as if inheritance is only subclassing, and this usual inheritance is the only available inheritance in the language, then it’s very easy to explain, why multiple inheritance is “not true object oriented technique” – this will be covered later.

2. What is virtual inheritance?

The usual inheritance relies on the statement that the base class is a direct base of a class that declared this derivation, that is, if this class is derived “through” some other class intermediately, it is only its indirect base and rather not interesting part of the base class. The virtual inheritance (the only language I know providing this feature is C++) is a bit different – every virtually derived class that occurs in the class hierarchy will always become a kind-of direct base of every its deriving class (even indirectly) and also this class will become a “shared superclass” for all classes in this hierarchy. It’s something like only the derivation procedures undergo derivation, while the (virtual base) class itself is provided as a base class of each class separately, although as common for them.

For example: in the following hierarchy (== marks virtual inheritance):

V ==> A --> B

not the class V itself is intermediately derived by deriving A, but the delegation procedures for class V, and in consequence, class V becomes a direct virtual base of class B, too (and of any other class that derives, even indirectly, from A).

The subobject layout is also different. If V was a normal base class of A, then in an object of class B there is a subobject of class A, which also contains a subobject of class V. If V is virtual base, then subobject of class A (in an object of class B) still contains a subobject of class V, however it’s also a subobject of B. A subobject of class, which is a virtual base, becomes a common subobject of the derived class and the base class. However from the class B perspective, the A subobject does not contain V – class B just took it over as its own. Despite the fact that V is still a subobject of A, the fact that it’s a subobject of B overlaps this first fact. In other words, both object of class B and subobject of class A think that the subobject of class V is their direct subobject.

The practical consequence of this fact isn’t seen as long as we use only single inheritance.

3. What is multiple inheritance?

It’s theoretically simple – one class derives more than just one.

That is, we are extending many classes together: we use multiple other classes in unison to combine a base for another class. We use particular classes as contents so that the object of our class will contain multiple subobjects belonging to multiple classes.

But wait! This is quite OK as long as we’re talking about extending. Moreover, as long as we are talking about independent extending (methods of derived class do not interfere with methods of base class, which means, among others, that methods of derived class do not mess with fields of base class). What about subclassing?

Of course, we still have problems like naming conflicts, but as long as it concerns sibling classes, it’s very simple to be solved. If the name of called method can’t be unanimously resolved as one of base class’s method, the compiler will complain, and this way you should either refer to method specifying from which class, or – better – create a method in the derived class that will redirect to this one that you meant, or even “rename” this method (that is, provide method forwarders with different names).

Multiple subclassing, on the other hand, is something that simply doesn’t fit in object-oriented programming. There’s simply no such thing. It’s not possible to define it logically.

In particular, what does it mean that a class named B “subclasses” a class named A? It’s simple: a class named B specializes or precises the term defined by class A. But what if we are defining class C that “subclasses” A and B simultaneously? If “subclassing” is defined as making “more specific”, “narrower range”, “specialization” – you can quickly come to a simple conclusion that this may concern only and exclusively one class. The only possibility to have one class that effectively subclasses two others is to have something like a “glued class”, that is, a composite of several classes, where each component subclasses only one class. And of course, none of these “components” have a thing to do with each other.

If we look at it from Smalltalk perspective, it’s not possible to subclass multiply by two reasons. The most characteristic reason for Smalltalk is that in this language the base class is used as a “fallback” when the method is not defined in the object’s class. Where to “fallback” then, if the class doesn’t define the method? The first one? Each base one by one? Find the first class (in the derivation order) that “understands” it and throw an exception if none does?

Of course, this problem is soluble – technically of course, not logically. For example, Python solves it such a way that it just tries to resolve the method by searching the base class, then starts to search the next base sibling if the previous search failed. In particular, it does not even regard naming conflicts – if there are two methods of the same name in both base classes, the first one takes precedence anyway. It means that derivation order also defines priority, or significance – a method from the more significant class overrides the method from the less significant one.

But being soluble doesn’t mean that this is a good reason to use this solution. The problem isn’t that this is some hack or something. The problem is that this solves the technical problem of searching for a method by name, but does not solve the problem of impossible multiple subclassing in the OO theory. Of course, this conflict problem relies on method names, not on provided features (so it would be said that this shouldn’t have anything to do with the OO theory). The problem, though, is in the impossible subclassing in case when base classes form a diamond. Providing technical solution for this problem leads to accepting a logically wrong code.

In the light of this, there’s no wonder why even the official documentation of Python strongly discourages using multiple inheritance.

If you are lemming, you can now quickly realize that the multiple inheritance is bad and shouldn’t be used. If you are engineer, you know that the real world will quickly surprise you.

4. Classes in the real world

Let’s reconsider then, whether multiple subclassing is something that is acceptable in the real world. Let’s reconsider first, why there is something called class hierarchy.

Class hierarchy (classification) is something that came from – well known from many other domains – “categorizing”. This functions like multiple boxes to which we can put many small things. Depending on the category we assign to particular thing, we put this thing to the particular box. As you can see, one thing cannot be in more than one box at a time. This is the main treat of categorizing: there can be only one category to qualify particular thing.

However this is a very strong (and very humane!) simplification in describing the world. This is because categories exist on some specified “level”, or there can be “categories of categories”, or whatever we can call it, in other words there are “category types”. When we talk about these boxes, to which we put things, these boxes describe categories of only one type. While there can be multiple types of categories and each object can be categorized from multiple perspectives. This is normal thing because an object cannot be just one treat – usually objects combine many different treats in one. However every object has always only one treat of specified type (if we think some object has multiple treats of one type, it means that the category type is incorrectly defined).

It simply means that an object may belong to multiple categories, but each category must be of different category type. If we call the category “class”, so that we “classify” these objects, we may have something like “class types” that group classes, and this way an object may belong to multiple classes at a time. There is only one condition: every class, on whatever level of derivation, must appear in the hierarchy tree only once.

5. Amphibia, as a subclass of car and boat

I take this example because it’s the best known example of multiple inheritance and simultaneously an example that multiple inheritance is possible, correct, and does not require any hacks – it just works in the real world. Simultaneously it introduces a big confusion for object-oriented languages. Especially because it proves that languages that do not feature multiple inheritance are stupid.

The amphibia class is then a subclass of car (because it rides as a car on roads) and of boat (because it can swim as a boat). Why is it possible that one class can unify these things in one? Simply because boat and car do not have any common parts.

No? What about engine, steering wheel, controls? We’re still talking about an engine-powered boat, as this amphibia can be powered by its engine also when swimming.

Pay attention that boat and car are totally different things. A car can ride on roads, but it’s not possible that it ride on deep water. Also a boat is “riding” only on water, while it cannot ride on roads. Even though we’d say that they both move, their definitions of movement do not interfere with each other. They are just moving in separate environments, so they are different “types of moving”. Moreover, now depending on this environment our amphibia behaves only like a boat or only like a car at a time. It is, then, something like two different objects combined into one.

However, they have some common parts. Yes. But this common part can cover different functionalities (not dependent on the fact that this machine is running on water or land), which cooperate with the additional treats. So this one part is common (so in OO it should undergo virtual inheritance) and the other parts provides separate features.

Well, so engine, steering wheel, or controls, are just parts of it. This is just another class category. How does it work then?

Very simply. Usually if you define some class, which is deriving from another class, you need to define a level of significance, that is, whether we are limiting the use of object (which is said to be the main thing that inheritance does, although it’s not true), or we are extending the use of object, towards some base class.

Referring to this example, if we define EnginePowered class, we define something that can be powered by engine in general. We can have also two classes, Car and Boat. These two classes don’t have any common parts – they are quite separate. Now let’s consider that we create new classes – EnginePoweredCar and EnginePoweredBoat (please note that Car can also be a part of train and does not require to be powered independently). By defining such classes we bind two different treats in one, that is, treats belonging to different categories: EnginePowered is a class of totally different category than Car or Boat. Moreover, EnginePowered can be treated as a technical detail that may be used for something else, while Car or Boat is the main treat. So, EnginePoweredBoat should be defined as a class that derives Boat and virtually derives EnginePowered.

This way, if you now define Amphibia as a class that derives EnginePoweredCar and EnginePoweredBoat, it will be composed (practically) from Car, Boat, and – virtually – EnginePowered.

But how does it work that despite that EnginePoweredCar and EnginePoweredBoat override the same functions from EnginePowered and do not “conflict”? It’s because they are linked special way – it’s not much important whether they are automatically switching between road mode and water mode, or whether both are operated simultaneously. The main reason why this is possible is that it’s possible to define what engine and steering wheel and all other controls operate with details of Car and Boat. This is because the control elements of car (wheels) and control elements of boat (rudder) do not conflict with each other.

6. Virtual inheritance as a solution for multiple inheritance

So, again, in which case the virtual inheritance should be used and when it is reasonable to use multiple inheritance?

Let’s imagine that we have some class A that defines whatever things can belong to it as a category – there is a large variety of objects that can be classified as A. Now you define B that derives A but limits it to some narrower range, so much less objects belong to B. You also define C and D. Please note that B, C and D are sibling classes, and as such, they should belong to the same class category. In which case then, in similar situation, should A be derived virtually?

In such a case, when defining B makes the range enough narrow that it is not physically possible that there can be any object that would belong to C or D (that is, to any imaginable other classes that would become siblings to B). In other words, beside the fact that these classes subclass A, they do not have any least common treats.

It’s very simple. As I said, in virtual inheritance the subobject of virtual base is common to everyone in the hierarchy from this class down. It should then express the exceptional common treat of all its subclasses, and there are no other common treats.

An example of virtual inheritance may be also a mule. As mule is a mix of horse and donkey, you may say that horse is animal and donkey is animal. However mule isn’t animal just because both donkey and horse are animals. Mule is animal because it just IS animal, independently on the fact that horse and donkey are both animals, too. It’s just yet another case (subclass) of animal, despite the fact of coming from horse and donkey.

I express this because there can be classes that derive from some other class and may implicitly have some common treats. In this case there’s no point in deriving virtually their base class, as these classes cannot be separated anyway.

I even tend to think that the virtual inheritance isn’t a required feature to make it possible to have a well working multiple inheritance – it would be enough that the “diamond” forms are disallowed, but you can derive from the “conflicting” class in the class, in which you derived these two classes. That is, if you have A deriving from V and B deriving from V, and C that derives from A and B, you should derive it also from V to make this code accepted. Such a solution would do the same as virtual inheritance. The virtual inheritance is needed in C++ mainly because this keyword changes the object layout to a bit less efficient, but the only possible to implement a shared subobject. But this solution is able to be implemented in languages like Python.

7. Diamond

Diamonds do not exist.

Ok, let’s start from a different place. Centaurs do not exist.

Why do not centaurs exist, or simpler, why cannot they exist? Because they are composed of a horse that was cut off its head and a man that was cut off his… well, say, loins. You cannot compose objects (or their classes) by taking a damaged part of the one and a damaged part of the other, this way composing something that cannot even co-function. For example: where should this creature have lounges and stomach, or even more funny, how could the backbone of such a creature be constructed? Try to find a centaur’s skeleton drawings on the network and you’ll see that the backbone is bent in the place where human body meets the horse body – a very strong bending, which cannot be withstood by an average man’s backbone and not break. Such a creature does not exist, but not because the nature did not happen to evolve anything to it (there would be no such problems with Shokans), but because it is not possible to define such a creature – neither technically, nor logically.

Centaur is an example of combining two classes that have common parts, or IOW, two classes that partially play the same roles. We can extract parts of horse and man on the lowest level, as mammals. But then, there are more specific treats that make them two different classes. However they are still not so different that the common parts can be disappeared or extracted as a “virtual base”. In case of man and horse, the “mammal” is the virtual base, however next both classes add new features of the same category (but working completely different way), so there are still many treats that are in conflicts with each other.

So, if we have two classes that manage different sorts of terms (riding in case of car and swimming in case of boat), we can safely create a class that derives from them both, because these classes cover different features. We can’t do the same with horse and man because these classes both cover features of the same categories, they only do it different way.

There is also similar problem with widgets. Note that despite that C++ is such an old language now and there is a lot of GUI libraries, and moreover, some of them are now being created with regard of standard C++ (!), none of them allows for multiple inheritance for widget classes, in particular, a new widget class cannot derive from more than one widget class (but of course, there’s no problem in having a class that derives, say, a widget and a stream). It could be theoretically possible that the Widget class itself be virtually inherited in every basic widget, so you can compose widgets.

That’s not true. It’s different in different libraries, but it’s very well seen in Qt. In this library, there is a virtual method in QWidget class, called ‘paintEvent’. You just override this method and then inside it “do whatever you want to make this widget look as you wish”. What would you then do in your ‘paintEvent’ method in a class that derives from two widget classes, while in both base widget classes you have ‘paintEvent’ methods that do totally different things? If you just allow them to paint (that is, you call them both one after another), they’ll collide (the second one will scratch the result of the first one). If you want that they paint at different coordinates you should either fake the widget different sizes during repainting (which is really a “tuff” hack!), or you can fake some margins around so that you can move the fake widget’s sizes and position to force underlying paint-s to draw at correct coordinates. Moreover, you’ll have to regard these faked coordinates in many other methods, like those of sizing, also you need to “route” the mouse clicks to appropriate underlying class, etc. etc. – the widget class just isn’t predicted to manage “glued two widgets”. While it’s very simple how to create a composite widget: just use a container widget that can lay out the widget parts on itself. This is what you should do when you want that one subwidget stand beside the other. Your class may at most derive from the layout widget (in particular, this is a solution for Gtk; in Qt you should derive from QWidget and use QBoxLayout for organizing the layout).

But no matter what a combination of two (or more) widgets you’d imagine, there’s no possible combination that would excuse using multiple inheritance to implement it. You’d say it should be possible to combine Label and Button, this way creating button with label. No, that’s wrong – this is just another combination of widgets; in Gtk the GtkButton class is a container widget, so you can put anything inside it; similarly this should be done in other libraries. Whatever you’d imagine as a combination of widgets, it will always result in the same: you should at most derive from just ONE widget class and if you need any subwidgets as components, you just manage them as independent ingredients.

Remember: when you have two classes that even partially play the same role, you just can’t derive from them both simultaneously. If this “role” is defined by the base class, this class is never derived virtually in case when the extension of the derived class is tightly bound to the implementation of this role. So this would form a diamond.

Do not look for diamonds. Diamonds do not exist.

8. The correct multiple inheritance

From the technical point of view, the correct multiple inheritance is when you adhere to the rules of the language. From the logical point of view, you can use multiple inheritance, but there are also some logical rules. I’ll show now, how the multiple inheritance should be used in a language that features multiple inheritance with virtual inheritance on request (although the only such language I know is C++).

If your hierarchy of objects is not polymorphic at all, there’s no problem with having multiple inheritance (case #0). In this case there is no, even potential, subclassing:

Non-polymorphic derivation (no subclassing)

If you inherit multiple classes, but only one of them is polymorphic, there’s still no multiple subclassing. It’s then multiple inheritance, but only single subclassing (case #1). In this picture, * marks the polymorphic class:

Derivation with only one polymorphic base class (single subclassing)

If you inherit multiple polymorphic classes, only two cases are allowed for object-oriented programming:

  • These two inherited classes come from totally different hierarchies and they do not have any common polymorphic classes in the inheritance tree (case #2). This is a bit spoofing of the OO theory (the resulting class is rather a kind of “class composite” than a class), but at least it does not form a “diamond”:

    Deriving two polymorphic classes with no common root (compositing)

  • Both classes are derived from the same abstract class, but this abstract class is virtually inherited by both of them (case #3). In the below picture, double line shows the virtual inheritance. Virtual inheritance means that every virtual base will always become a direct superclass of any class that derives it indirectly. Please note that in this case you have partially “separate features” derived, plus common feature of both classes, but derived individually and separately:

    Multiple polymorphic inheritance with virtually derived common root class

In other words: there is only one case of multiple inheritance which is a disallowed multiple subclassing: when two classes derived by some class are subclasses (even indirect) of some other class (case #4). As you can see, this is the only case when you can see the well known “diamond”, which causes such a confusion in OO theory:

Multiple polymorphic classes with common base class (diamond)

Please note that in a language, in which every user defined type that may be fully derived is class, and every class has implicit root base class (for example Java, C#, Smalltalk and practically a vast majority of OO languages), particular cases of multiple inheritance are impossible or limited:

  • case #0 and case #1: there is no such thing as “non polymorphic class” (in C# you can derive between structs, but there is no derivation between structs and classes)
  • case #2: It’s limited, in such a sense that only one of such polymorphic bases can be class; others must be interfaces (it results from a simple rule that there is a root class for all classes, so every case of two or more classes being derived will always form case #4)
  • case #3: It would be possible in such languages, if they supported virtual inheritance (they usually don’t; only some Smalltalk dialects support delegation as a simple replacement for multiple inheritance)

Virtual inheritance could be a solution for languages that do not feature multiple inheritance, however it must be allowed that the root object class be a virtual base (not always a direct base); otherwise any double inheritance will form a diamond. Such a solution is possible in languages like Java or C# (possibly also in Vala – there should be no obstacles for this in GObject system).

Note that virtual inheritance means that no matter how all base classes have implemented some methods from this virtually derived class, your class must implement them anew. Although unfortunately C++ still allows that if only one of derived classes (e.g. BaseB1 in the above picture) reimplemented a method from the base class (BaseAn) then the derived class (X) will take it as its own. The conflict may be only in case when at least two of derived (or intermediately derived) classes have overridden some methods from BaseAn (for example, both BaseA1 and B). This is unfortunate and the compiler should issue a warning about this because such a behavior is rarely expected and usually unobvious. But usually what you expect is that BaseAn is also a direct base of X, so X should reimplement all methods from BaseAn. Even stating that it can call implementations of these methods in its base classes. Because this is what you usually expect when you derive multiply and have some common base class.

Languages that allow for multiple inheritance, but do not feature virtual inheritance (nor the forced class rebasing, as I have mentioned), practically support the worst coding practices because instead of making diamonds disallowed, or at least making reimplementation conflicts disallowed (as C++ does), they try to solve the technical problem, that is, to accept a counter-logical code. To use multiple inheritance correctly in these languages, you should always manually find the common base class and override all overridable methods in this class, even as simple redirection to some other method, because otherwise your class’s behavior will quickly go out of control. In particular, you should not allow that the automatic conflict resolver resolve your method call, but you should rather manually define which method should be called. Otherwise your class will form a “Frankenstein”, which partially behaves as one of classes that you have derived.

9. Lessons learned

Of course, a usual programmer has no influence on that the language they have to use to make software have some features implemented stupid way. But at least they are able to keep some rules that will, simply, rely on just not doing stupid things even if a language allows for it.

If you heard that multiple inheritance is bad because it breaks the OO theory, and this theory has been created by observing the real world, you can always respond that amphibia is a case of virtual inheritance existing in the real world, so it’s maybe this theory what is broken. On the other hand, don’t treat multiple inheritance as a method to combine everything into one for any purpose. You should research it logically – whether it should be derivation, but maybe containment, wrapping, composition, proxies etc. And first of all, keep the rules that define what kinds of inheritance are allowed.

Multiple inheritance can be considered nice – as long as you don’t look for diamonds.

Posted in Uncategorized | 1 Comment

Sweeping bugs under the carpet

A bit differently to the other articles, this is predicted mainly for managers and team leaders, who should be especially interested with increasing software quality. Although many programmers may still get some advantages from it :). The first version was published on blogspot, May 2009.

0. Basic statements about software quality

Software quality is one of the most important interest of the organizations occupied with producing software. What is interesting, it’s much easier to write bad software than, for example, create a bad bridge or a bad car. Reasons are somewhat similar, of course, but producing a software, despite it being an engineering domain, is a high level of complexity, and this spreads to every aspect of the process, including quality.

When you produce software, everything is added levels of complexity (comparing to other engineering domains): planning, creation, and verification, too. That’s why every case of simplification is welcome. However with this simplification there are two main problems:

  1. Many topics are only “felt” or “intuitively understood”, not elaborated
  2. Incorrect approach to simplification results in “re-complication”

The process of “re-complication” can be compared to trial of compressing a container of water. As you know, water is no air; if you try to press it strongly from above, the container will expand horizontally. Water just can’t be compressed; you can at most heat it to make it boiling and turn into gas, which can be compressed. But again, increasing the pressure of the gasified water may shift its boiling point and make it condense. This is what happens when you try to restructure something which’s structure you don’t understand.

Things like that may end up with “re-complication” and spawn something that is harder to understand and use. This is what happens to various approaches to simplify the software engineering processes.

First of all, too many people forget that the most important characteristics of software engineering is that it’s a process of implementing some logical ideas using some intermediate statements, called often “programming languages”. In particular, it’s too often forgotten that the “logics” is the core of programming, not the language used to express it. A programming language is only a most wishful method to transform the logics into a working program, so it’s only a secondary product of software engineering. Whatever process support you’d spell up, stating that it bases on the programming language used to write the program, will be a secondary-product-based support, as well as secondary would be any help, any results and any simplifications that they would provide.

This highlights for us the following non-obvious characteristics of the software engineering:

  1. One of the most important things is transformation reliability (how good your translation from the logics to the programming language is)
  2. Learning programming techniques should more base on logics and less on the programming languages. Programmers whose thinking manners base on programming languages or platform specifics, will never make reliable software (also thinking on the platform level may often lead to bad code optimizations)
  3. Testing should also base on logics stated for the program rather than on what can be determined from the sources

The needs to focus on logics have important consequences. For example, it’s best if you don’t rely on only the source files to keep track of the logics of your program – it’s always much better if additional information is provided, which contains logical statements that cannot be reliably coded in the programming language. That’s why it’s very important that the program does not only consist of source files, but appropriate descriptions, documentations, diagrams, and many other documents are provided and accordingly maintained. Also all automatic tests should first of all rely on the logical statements of how the application should work rather than on whether the program produces correct results.

1. Correctness and incorrectness of the program

The program should behave as it is desired. This is the first mistake that programmers do. The program is created not to fulfill the requirements, but to fulfill the customer’s desire.

So, the first level of the software need is the desire. This is what we really want to have.

Next level is the requirement. This is what we described as what we want to have.

Next level is the HLD. This is what the designers understand our description of what we want to have.

Next level is the implementation. This is what programmers have implemented of what designers understand of what the customer has described that they want to have.

And, finally, we have a product. Product is an output of the final testers, who have confirmed the implementation. Effectively, the product is a finally-confirmed output of what programmers have implemented as they understood what designers have understood what customers have described they want to have. In simplest form:

Desire -> Requirement -> Design -> Implementation -> Product

Every arrow in the description above means a place where the bugs can be provided. When we’re talking about the software quality, though, we usually talk about the bugs that occur during the implementation phase. Because the input for the implementation is usually the logics and the the logics should be translated into a programming language.

The problem is that in general the program is meant 100% correct if the product is 100% consistent with the desire. Even if we focus on the implementation only, the correctness of the implementation relies on how it reflects the statements of the design. The problems with verifying it is driven from the following reasons:

  1. Some things needed for the implementation are not specified by the design. They should not matter for the final results, however there must be found a way to eliminate them in final verification.
  2. Desired results of the implementation should be translatable to logics, in order to have something reasonable to compare with the first statements
  3. Complete results of the implementation should be able to be translated to the complete logics, containing both things stated by the design and things needed to introduce into the implementation to block undesired behavior. Parts of the program that do not work as it is desired, should be blocked.

This is one of the reasons that the modern software production processes involve customer into the process. Because the real verification should be the verification of both ends of the chain: the product is correct when it is compliant with the desire.

Of course, we need these things that prepend the implementation. We need a design to create the implementation; we need requirements to prepare the design. But the first verification of the final version of the implementation may rely on comparing with the design, maybe comparing with the requirements – but the software company cannot compare it with the desire because only the customer knows what the “desire” is.

Of course, some companies think that they did everything correct if their product is compliant with the requirements. Such companies, however, should not be treated seriously because such an approach is treating the customer ridiculously. The customer is satisfied if they received what they desired, not what they were able to describe as their desire.

But focusing only on the implementation, there still are things that no program should ever do and there’s rather a requirement of every imaginable customer that the program will never do: crash or change the internal document data in an undefined or untraceable manner.

Bottom line is that in final result we should have a product, which is an implementation of, indirectly, a customer’s desire, which should do things that have been stated (which is the compliance verification) and not do things that were not stated (which is often forgotten).

2. Logical input and logical output of the implementation

No matter what theory was used do design and implement the program, the following steps are always true for the software implementation process:

  1. The design defines how the final program should fulfill particular requirements and states desired scenarios
  2. The executive part of the implementation is using variables and generally various other changeable data
  3. The external and system parts of the application the changeable data introduce its own logics, not always desired, however indispensable for implementing the design statements
  4. The final implementation contains statements required by the design and requirements, plus some additional logics

Every program must use variables. Even if shifted to very narrow role, they still are variables and they will always be variables.

What is a variable in general? It is some data that may be written and read by the program, which means also, that some procedures can behave differently, depending on the value of the variable.

Is the variable part of the logics of the program?

I should make here a very long text that has no reason, or just leave an empty space so that you can reconsider the above question. But I’m too lazy to do that. Just read this text and this should make you sure that … eee … know the answer? Now? So, what do you think.

Variables are not parts of the logics of the program. Variable is a tool we use to implement the logics of the program.

Ok, now few words about the terms I am using:

  • state: a logical unit that may change values when modified and that can be read a value from
  • state value: a value that can be assigned to a state at a time

Maybe it would be better to say “a variable is in state” than “a state has a value”, but I would have to remind that I don’t mean a variable in a programming language. This sounds better for me, at least because a variable is only used to express this state, not always perfectly. In these terms, “state” is something like a “stateous unit”.

We can also say that the program is in some “state”. It means – in these terms – that the state of the program can have appropriate state value. Of course, programs can be in many states; the number of the values that can be achieved by the program’s state is called the “degrees of freedom“. Degrees of freedom can be a property of the state, no matter how implemented.

Like it is with many other things, states can be hierarchized, dependent on each other, and so on, so finally, states may be simple (just one single Particle State) and complex (consist of many other Particle States or complex states). The whole program, then, has its own state, which is a complex state, and which consists of all particle states in the program, unified. The number of degrees of freedom for this state is equal to multiplied all its particle states’ degrees of freedom, minus tristate-blocked values (if you have a tristate: [nonapplicable,true,false], then the nonapplicable value does not produce any new result when multiplying with another state) and self-elliminating values (there may exist two equal values of a complex state despite some of particle states may differ).

There’s really a lot of things that can be told about this topic, but I’ll stop here as it’s not the subject of this article. I’ll try to describe it more thoroughly elsewhere.

Every program must work with states and so its particular execution statements rely on values of particular particle states, so these states define possible execution scenarios. How much such scenarios exist, it depends on the degrees of freedom of the state (depending on level or execution period, for which the scenario is defined, states on this very level).

So, our input was the design that defines what scenarios should be possible to be passed. Our output is an implementation, which contains many various potential scenarios, of which several have been allowed to execution and others have been blocked by appropriate ending them by a dead end.

3. Physical output of the implementation

First, let’s reconsider, what we really need to have a product.

To have a product we need to have a binary file with program and co-accompanied files.

To have the binary file with program we need source files.

And we can say “that’s all”. If we need everything we can “compile” into the final required form, we don’t need any other things.

And that’s the reason why many companies do not create or maintain other things. Creating and maintaining them costs a lot, so we spare costs by not having them.

Good. But what would you like to do with your source files next? Your product is ready, you can throw them out.

What are you waiting for? Why do you waste your disk space? Delete!

Why not? What will you use them for?

Ah, you want to improve it with some additional features? You’re going to create a new, enhanced product, with more functions, more useful, so that you can get more money from this?

Good, but you still have only source files! Are they any better than binary files?

What? You think you can just continue with the development? Very funny.

So, again. As I said, requirements is logics, design is logics, and implementation is logics, too. It means that the source files contain only the statements that explain how “some logics” should be implmented by the machine – but they hardly explain what this “some logics” is! Imagine that you have an engine and you’d like to improve it. You get the engine’s schematics. But how would you do any work to make it if you don’t know what’s the machine’s purpose!?

Another example, a bit more from software engineering: If you have a physical value of a distance and speed, then having given the base distance and the speed we can calculate the time as

double t = s/v

But we can write it also as

double a = b/c

What does that above mean? We have used three variables of ‘double’ type – what would this ‘double’ mean when you have a=b/c?

This is how the majority of statements look. Of course, some identifiers may be self-explanatory, there may be some comments, but it’s usually never enough to get notions about what logics it tries to implement.

And at this time, you wish you had some documentation…

Of course. Because only this additional “documentation” can contain any straight explanation about the logics that this code implements. So, if you want to make enhancements you need sources, but you also need some explanations about the logics – if you don’t have them, the only way to go with the development is to first reverse-engineer the code.

That’s why it’s important that the implementation is being created together with the LLD (Low Level Design) AKA DLD (Detailed Level Design).

I don’t mean LLD in the “waterfal sense” (which is indeed very stupid thing). LLD should be a part of implementation work.

The LLD is a bundle of documents that should be created during the implementation development and should complete the source code with various additional descriptions and schemes that will next help understand the logics of the program. And because these two things are so interconnected with each other, many parts of the LLD are created by automatic documentation generators (like Doxygen). Of course it will never be a “complete logics” because the logics will always be more or less floating, but it should explain enough logics so that everyone who will next try to continue the development will know what they’re doing.

If you have only the source code, which is, worse still, very sparsely commented, to be able to continue the development you need to perform the reverse-engineering first.

If you’re going to create unit/component tests for the code for which you only have source files, you can create only tests that will confirm the correctness of the source code for “whatever logics they may employ”, but not for the logics, for which the implementation was created.

If you’re going to confirm the correctness of the program, for which you only have source files… you’d better go to a fortune teller. Or roll a die.

4. The needs of reverse-engineering

If you think that you can “start coding” when you only have source files, you have probably never got involved in the strictly technical software engineering. Even having a full documentation, an engineer still should spend some time to study it – although this time will be relatively short. When there is no documentation, the engineer should spend much more time with analyzing the source files and – what is much harder – guess the business logic standing behind this code. If you think that “this reverse engineering will be still cheaper than writing a program from scratch”, you may be astonishingly surprised. You might have probably heard sometimes that “this code is so tangled that I would spend less time by writing it from scratch than fixing it”. Often it is said so by programmers because it’s more satisfactory (and less boring) to create something new than dig in the dirt of the existing code. But unfortunately they often just tell you the truth: the time spent with writing this from scratch may cost them less time than “reinstating” the existing code to full understanding. If you have some more resources to spend with this project and you think that it’s valuable for you to collect this information, you can give the project to two separate teams, one to create it from scratch, and one to reverse-engineer the old code. I would beg for 80% that the first team will finish their work faster.

The reverse-engineering by just looking at the program how it works and trying to duplicate it manually is the simplest method and theoretically is highest cost. Any platform-code-based reverse-engineering (disassembly-based) is complicated and practically impossible to be done manually, without very advanced and intelligent tools. This theoretically would cost less, but tools that can do it reliably are expensive. And finally the reverse-engineering of the program’s source code is theoretically cheapest. But it still costs time to spend with the source code to see how it works and what it does in practice, plus guessing what the author’s intention was to write it.

This job involves not only to navigate through the code, but usually also run it several times in various different conditions, maybe also under a debugger, and trace various execution paths in the program. Worse still, you should also consider that the code may be in some parts written incorrectly or some parts of it was a “wrong way”, abandoned and not used – just the developer had no time to delete it from the source. And of course, during this job you should create the LLD documents concurrently; otherwise this job is just wasting time.

Time spent with this isn’t that short as you may think. Restoring the logics from the existing code will always take more time than spelling up these logics in your mind (because when reverse-engineering you try to discover it using trial and error method). The only way to spare time in this first case is when you don’t have to understand all of the logics of the program (according to my observations, if you wanted to analyze everything in the program, you’d have to spend more than twice more time than it was used to write it). That is, you just need to understand some small parts of it, while you don’t even touch other parts (if my observations are fair, you should not analyze more than 1/4 of the total source lines; otherwise you’d better write it anew).

Time spent with analyzing the source code may be less than spending the time with analyzing the same program for which you only have binary files. I say “may be” because this is often an illusion. In average estimation, you should most safely state that it really doesn’t matter if you have source files or not, if you don’t have any least documentation. Moreover, if you have a description of what the program is doing and how it works, it would help you much more than source files. Of course, it happens that source files help you “continue development”, but this happens only if you have a big and expanded program and you need to modify only a small part of it. Analyzing the source code may have a big educational value, but well, keep in mind that this is not a value that directly contributes to works on the software.

Especially that the public software tools evolve, and today we have at our disposal many advanced and complicated tools that make our task easier than it was many years ago. If your application has been released in first version more than 5 years ago, it’s quite probable that a lot of manual coding and private utilities had to be written to make it work. Simply because there was nothing in this topic on the market at the time it was created. They can be now replaced by some existing library, usually even with LGPL license. That is, even omitting the fact that now that you have this application you are a bit wiser and can take advantage of lessons learned, the new version of the application written from scratch may take less time to write than it did for the old version of the application – due to much less source lines that need to be written.

5. Why simplifications and automation are so hard?

Pay attention that the source file in a programming language is an attempt to encode the program logics into a form understandable by the machine. The programming language is not the language to encode the logics, but just another attempt to automate and simplify the process of making the machine “execute” this desired logics.

Let’s even skip the advantage of portability that today languages (other than Java and C#) bring us, which is an advantage, indeed. Let’s focus on the simplification that the programming languages bring us.

Encoding a program logics in a language comprehensible for the machine requires to write it using appropriate machine commands. Enhancing the level of logics is possible when we have a program that can translate this level of logics into the machine language.

But it’s not the only role of the programming language. The programming language’s role is also to employ the logics of particular level.

If you ask me, what this “logics” is I’m talking about since a very long time, I wouldn’t be able to tell you in one simple sentence. Because logics is the logics. This is what you think the program should do. But what logics really is (say, “how it would look like”), it’s very hard to say. There can’t be any strict definition of logics because first thing you take up when you attempt to create the implementation is “how would we do it?”. And once you state such a question, this is where you translate your logics into computer language (so it’s not “logics” anymore).

That’s why many people may think totally different things about what the logics of some program is, even if they’re going to create the same program. You can think of the levels of logics like of the levels of network protocols, or layers of the application. This is basically the same thing: there is some underlying layer that would work for the sake of the higher layer so that it can exist and do its job. This is something like using a low-level transport protocol to work as a transport layer for some higher-level protocol – like, for example, SyncML protocol over HTTP.

So, similarly, you employ bytes of memory and FPU to implement your real numbers and you employ real numbers to implement speed, distance and time. The real number is not a computer level – computer level is only bytes of memory and floating-point operations done by FPU (or emulated). This is a logical level, then a higher logical level would be the terms of speed, distance and time.

Logics is fuzzy by nature. I’m not talking about the fuzzy logics of course. I’m talking about what various people think about some things being done some specific way. Logics is fuzzy, which means, that there are no strict rules, what logics is behind some specific task. For example, when I’d like to perform some operation for each object from the list, should I define the operation as “do something with the object and do myself for the rest” or “perform the following thing for every item one after another”? Is the second thing only an attempt to implement the first, or both are an intermediate description that can be understood by compilers? Is modifying the object in place a normal and quite logical thing, or logically I should treat every object as unmodifiable and rather create a new object instead of modifying the existing one?

Both can be logical or illogical (that is, an intermediate description). That depends on what you compare it with, depends on your knowledge about how to play with items created by a computer, how you perceive the surrounding world, how you would create similar thing not using a computer. Yes, it also largely depends on your imagination.

Also, some “rules” of what logics can be used to implement any higher level logics, should be developed and implemented by creating particular programming languages. But whatever level you’d reach in these attempts, there are two problems:

  1. The highest level of logics for the application is the description of this application itself – logical terms refine as the level of logics increases, but becomes more and more specific for the problem and this way we’d reach the application itself. Such a level is useless for our general simplifications because in a language of description created as a highest level of logics of application X is a language in which only application X can be written.
  2. Describing such thorough logics as statements in the programming language may be, unfortunately, really effort consuming. Using, for example, the ‘double’ type for speed, distance and time, is good enough, you’d write it quickly and make a readable program. You may want to use the Speed, Distance and Time types for them all, assigning them some special mark as the unit and define dependencies between these types and various equalities and equivalences between particular physical units (like N/kg being equal unit to m/s2). You can do it. This work, sometimes, may be really worthwhile. But really, really in very specific situations. Generally, these things will only contribute to extend the time to create the application and contribute to no advantages.

So, whether we want or not, we should be conscious of the following things:

  1. The application will be always written in a language, which’s level of logics is lower than the level of logics for the application
  2. Whatever automatic tools we create to help us make software, they will only work with a strictly defined language, which means, that they will not do things that require anything above its level
  3. Any tools that would verify the application’s correctness will work always only for that level of logics, for which we have source files, so they will only verify the correctness on this very level of logics.

However, it’s really worthwhile to make the language more schematic and using stricter definitions (that is, for example, that use strict types for variables rather than allowing a variable to be of any type). Only such languages can help you with encoding the rules for the correctness in the language itself. Even if such level of logics, which will be checked for correctness, is low, for totally-dynamic languages the level is much lower.

Dynamic languages, especially those totally dynamic, like Tcl, are no higher level languages at all. They are much like assembly language, there’s only a different platform that runs the program. Languages that do not employ static types at all or do not allow to create any restricted types are all low level languages.

As I have mentioned about abilities to start coding when you have source code, but no documentation, this is also similar when it comes to automation. Automatic tools that would process the source files will be only able to do a research based on the source files, but they will not guess the actual business logic. If the difference between the levels of the final logics and the source code is very high (and it usually is true for C language), you can only use tools that will find any “general purpose bugs” in the source code, which is important, but still not focused on the matter of what software is.

6. What useful automation can we count for?

Ok, now that we have several tools on the market, which can do for us many useful things by analyzing the source code, let’s try to reconsider, how useful they may be, stating that they are working for particular programming language.

So we can count for:

  • Determining improper use of language and semantics (that is, potential errors)
  • Checking code complication level
  • Checking for deterministic execution (for languages that have abilities for nondeterministic execution)
  • Checking for non-generic programming
  • Checking for bad style
  • Checking the testing level
  • Creating call graphs and class diagrams (inheritance and collaboration)

This check for “testing level” is used to check what fragments of code are being passed by particular test cases. It’s called “Code Coverage”.

As you can see, these automatic tools can only do things that can be described by the programming language or extract and evaluate the information that can be contained in the program (written in that language).

One of the important tools are those that can create class diagrams basing on the source code. These tools are really worthwhile; it’s really much better if you don’t create your code with blocks and generate then from tools. As a programmer, you should focus on writing the code, nothing else. A good, experienced programmer will write the code, from which class diagrams will be extracted by an automatic tool (for example, Doxygen), much faster, than someone who would create the class diagram and generate the code.

And if there’s something important that cannot be written in the programming language but is required that the tool better understand the structure? Then the tool should provide additional facilities that the user can use to help the tool guess what they meant by writing particular code.

The “lint” tools (like Klocwork) help determining many potential errors and dangerous situations, also help detect unreadable, too complicated code and many other things that decrease maintainability. The same about bad style.

The topic of unit tests is slightly more complicated, though. That’s why it’s hard to evaluate, how good a tool that works with unit tests can be.

Since generally all these tools have exactly one destination: help developers write better code. Nothing more. Developers, who use these tools can still write the same bad code anyway – the tool isn’t a magic bullet, nor it is any gauge that can be used to show the code quality. By the way, there is no such tool at all. The main problem is that the tool does not operate with the logics, which means that:

  • It can only check for “general purpose bugs” and “direct value-based result”
  • It can only check the correctness of the code that is present (but not detect problems because of a code that wasn’t written)
  • The measurement based on execution path is meaningless because it has nothing to do with the real logical layer – the execution scenarios

Which leads us to a simple statement: the correctness can only be checked by application-specific tools or manually, having the thorough requirements description. Both require effort and both are strictly “human-based”.

7. What things can be done to make our code better?

Well, I even have a better question: what can we do to ensure the maximum reliability of the quality?

Yes, I know this is a tough question. Especially that it doesn’t bring any help to managers.

But this is really all we can do to enhance the quality: make the quality conditions predictable. Once these conditions are predictable enough, we can at least know whether the quality is good or bad. Various tools and techniques can only help us enhance this reliability rather than the quality itself.

You can say: we should have unit tests, we should have testing procedures… yes, seems wise. But if you say that when the program is already written, you are not wise. You are dumb. (Although we have a saying: “A wise Pole after a detriment”.)

The most important factor that may enhance the quality’s reliability is the program’s testability.

You know, the most simple way to debug the program is to put printing hooks inside so that we know what the program is doing. If you think that this is an “obsolete” technique and really primitive as compared to debuggers, you probably never got involved technically in any software engineering, and never, for example, developed an application in real-time system. Don’t forget that when you are debugging a program, then every instruction is being executed much longer time than it takes in “real conditions” (which sometimes really makes a difference), moreover, the instructions that are executed similar time than the “external” ones, now execute with a really great time difference. Programs also often contain timers, handlers for some events that are synchronized in time, and the program when traced line by line just won’t behave the same way as without debugger. I don’t even mention threads, which make more mess in this.

This is the first thing. Second thing is that if you want to test some small part of a big program, you need that the program be in some particular state – and in order to enter in this state you need to prepare the program for this somehow.

It’s not a challenge to write a big program. It’s a challenge to write a big, testable program. A program is testable, if you are able to extract some parts of the execution statements, even tear off the current execution environment and run under controlled conditions. If some part of the code is not able to work in a “torn off” environment, it means that this part of code is not testable. It’s very easy to write a program that contains a lot of hidden, false, or unnecessary depndencies. Not only may they be not testable, but it may be even very hard to try some real proram scenario to run in a controlled environment. Or at least it would require a lot of effort to “configure” the program’s state that is the starting point of this scenario.

In short, it’s very easy to write a big program, which does things that you absolutely don’t know. It’s especially important in case of environments that do not provide any abilities to debug the program – for example, device drivers for embedded real-time systems. The truth for this case is – if you can’t reproduce it with the emulator (or even worse, you don’t have any emulator), you are in… a hole. The only thing you can do is to print some logs. Stating that you at least have something to print on.

Anyway, it is important that the environment in which you run the program be close enough to the production environment and provide testability. But much more important is whether your program is easy to do the following: take the function you suspect of providing the problem, reinstate the environment (that is, set the global state to the required value), run, and collect enough information that help you determine the problem. You can do it only if:

  • This function is stateless and deterministic. In this case you just feed the function with input values and expect some result after return.
  • The program provides access to this function via some text-based interface, or even a “scripting engine” built into the program

If none of these above conditions is satisfied, the function isn’t testable – which means that you don’t even have notions whether it works or not!

If you have functions like that in your program, it means (at least) that your program has low testability. The program is moreover little testable, if it relies on values in some global variables, these global variables are pointers to other variables, objects, these objects contain more pointers to objects etc. Moreover, if you are trying to hide the true nature of the object being held by some pointer by using the generic pointer (void*, gpointer, Object, id, or whatever it’s called in your favourite language) and casting. The ideal situation is when the programmer, when looking at some pointer variable, always knows what an object would be assigned to it, what it would do and why it is there. And, moreover, such a situation occurs rarely and objects are rather being obtained from the place that owns and manages them, rather than remembered.

At this point there are currently three digressions I should make because of things that came to my mind:

Too much data sharing. This is the true nature of garbage-collection-based languages. The only situation when the garbage collector helps is when the object is shared between some parts of the code (because in case of objects that are owned by just one owner, there’s no problem: the owner takes care of allocating and deallocating the object). Now it looks like that it’s always much better than the object is owned than shared. That’s why it’s usually better than you have a shared pointer to an object and use sharing only when you really need it, than providing “sharing for everyone” and encourage people use object sharing for greater scale, enhancing this way the “unreliability” of the source code.

The problem of global variables, Global variables, yes, have always been accused of making problems, which is correct. So, creators of various programming languages (Java, C#) made it impossible to create global variables and requested that every variable or function must be a part of some class (this stupid statement fortunately did not span to Vala language). But if something is useful (and, moreover, dangerous, don’t know why, they usually come together), programmers will always find a way how to simulate it or do something similar. That’s why the idea of Singleton is so widespread throughout the code in Java and C#, feigning as if a “high educational programming” was used, while this Singleton is nothing else as another mutation of a global variable (that’s a topic of another article – Singleton is a totally different thing in C++, in which it’s a tool to solve the problem of predicted dependency-based initialization, and different thing in Java, in which it’s just a global variable emulation).

Since the problem with global variables is not that they are global. The problem is that the global variable is one of the cases where the sharing of the data occurs. Because the core of the problem is not the global scoping – the problem is sharing. Global variables are only one of possible specializations of data sharing. Ok, I would say it much more directly: the real danger of how the shared data can be harmful for the software quality is when the data is written. Sharing is much less harmful, if we have only one writer. The real problem is when there are many readers and many writers.

Dynamicity and expressiveness work against correctness and testability. That’s one of the reasons why Java has become so popular – it provided a good balance between the advantages of Smalltalk (virtual machine, garbage collection, OO) and advantages of, say, statically typed languages (practically C++, but Java’s creators would respond with indignation for suggestions like that – Java was, still, based on Haskell!). The dynamicity of Smalltalk was the main cause of problems in this language and introducing static typization, although it strongly works against object-orientation and breaks the language design stability, it contributes to decrease bugginess of the software. Static types were created not only in order to enable more optimizations on the low level. Static types are also used to precise what you really want that this part of code be up to. These types will make any debugged code more clear. The more “universal” and “dynamic” your type is, the more time you’ll waste to guess what’s going on there. That’s why types that are “maximum expressive” are simultaneously “maximum error-prone”.

So, the following things should be taken into consideration, so that the code is most possibly testable:

  • Abstain from data sharing
  • Strongly control data recursion and cyclic dependencies (at best avoid if possible)
  • Kill off false dependencies between functions and data (try to decompose the data structures so that functions do not access more data than they actually need)
  • If a function doesn’t require state, do not put any state into the function (try to always make the function stateless and deterministic, if possible)
  • For states (data that must be shared), determine the degrees of freedom and scope
  • Be prepared to provide a command-interpreter functionality so that it can be tested externally
  • Do not hesitate to create more static types that more precisely define your logics

8. How to get known of how reliably our code is tested?

Remember one important thing, which I have already mentioned, but I’ll repeat it because it’s too often forgotten:

The source code only describes what the computer platform should do to implement the logics of my program. So if there is any bug in the program, it may be:

  • A “general purpose bug”, that is, a bug that will always be a bug no matter the logics this code implements
  • A “implementation bug”, that is, the implementation does not reflect the desired logics
  • A “logical bug”, that is, bug in the logics itself

There can also be various more detailed bugs, like “interoperability bug”, that is, two pieces of code are bugless by themselves but buggy when composed together. But let’s focus on these most simple parts (these complex incorrectness may result from these bugs above).

Automatic tools may only help you with this first one. The others require either additional code, application, library, whatever, to perform specific tests. Or just manual verification having requirements-based scenario description in hand.

If you want to make sure that your implementation does not contain bugs:

  • You can use “lint-like” tools, to check your program for possible “general purpose bugs”
  • You can make and perform test scenarios to check whether your logics is reflected correctly by the implementation, to check for “implementation bugs”
  • You can research your logics to check whether they do not contain “logical bugs” (you can also test your logics working in the implementation – perform logical scenarios)

You can also try to determine the existing logics that are in the code and try to compare it with the stated logics. Well, this seems to have notions of a really good approach. By the way, this is exactly what the group code reviews are predestined for: let everyone interpret the code that they are seeing and check whether everyone interprets it as the same logics.

But there is another problem: there’s no tool that would translate the code source back to the logics. Moreover, logics cannot be really written, nor there can be any machine-level possible translation between logics and the implementation. If it did, 90% of problems in the software production would have been already resolved.

The source of the problem is that the logics, which is a base for the implementation, must be elaborated by a human (who is unreliable and can elaborate stupid logics). Then, it should be implemented by a human (who is unreliable and can make bugs). Finally, the program is checked for correct logics, by a human (who is unreliable and can verify it incorrectly). Or it can be verified by automatic tests, which would be a strict and totally machine-executed code that reliably performs the verification. Of course, these automatic tests must be first written by a human (and guess what… the human is still unreliable! :)).

Whatever tool for verification, quality tracing, statistics, obtaining data, classifying and many many more… it will always end up with a statement that a) the problem of reliability is in human and b) you cannot produce software not involving human.

And that’s why writing software without bugs is impossible.

The only thing we can do to enhance reliability of the software is to decrease the bugs. And every method that can significantly help in detecting bugs and potential bugs and threats and everything else that can threaten the code reliability, is always welcome.

Effectively, you have two possible approaches of how to try to make sure of the software quality:

  1. Use automatic tools to verify the source code. You’ll only get known that the source code does not contain general purpose bugs. Maybe you’ll also receive information whether the code has some bug because it didn’t pass the testcase. Of course, you don’t have a tool that verifies whether the testcase is correctly written, so even the testcase pass doesn’t make you sure that the code is correct.
  2. Rely on experienced programmers and give them all possible tools that will help them analyzing the code.

The advantage of #1 is that you don’t have to rely on those unreliable humans. Whatever thing you’d use, you are sure that the tool didn’t make any mistake. The disadvantage is that you don’t really get any reliable information about the software quality. Note also that no matter how you can rely on these tools, they still weren’t written by “gods” or computers… Not even mentioning how naive they are.

The disadvantage of #2 is that by relying on humans you can miss. If you rely on “bad people” you’ll fail. And of course, people are always unreliable. However, the advantage is that if you rely on people, and you’ll not believe more to the tools than to their words (that is, if you just trust them), there’s a great chance that they will respond with trust to you and whatever statistics you’ll create, but won’t use against them, will be more reliable.

If you even try to make any statistic behind their backs and watch them – don’t ever let them know that you’re doing it. If you do, they’ll always find a way what to do to make these statistics display good results, of course, not by bettering the code quality, but by tricks (people are way too smart than the tools). You cannot show even one smallest trace that you are creating these statistics and use them for anything. Not even a trace for suspicions. What it means in practice is that you may only watch them and save the information for yourself. You cannot make any decisions or changes in managing work, if they base on these statistics – if you do, your programmers will detect you quickly and quickly they’ll trick the code so that statistic look better. So, don’t use this tool for yourself – give it to programmers so that they can make the code better.

So, as you can see, there’s no better method to make sure of how our code is tested than just check the logics of the program that is written and try to determine whether this is exactly what we want. And this can be done only by humans.

I don’t say “reliably”. Humans are always unreliable, you well know it :).

9. How to approach to testing programs in order to test reliably?

Contrary to what many information technology scientists believe, programs do not have “execution paths”. Programs have “execution scenarios”. And programs are correct only if they are prepared to only execute correct scenarios and they prevent executing bad scenarios. When testing a program, we do not test, whether the program correctly passed some execution path. We test a scenario (either good or bad) to check whether the program lets it correctly pass (if good) or rejects this scenario quickly enough (if bad), and if so, it correctly recovers. Success scenarios are those that should work and are defined by requirements. Failure scenarios are needed to be made because of logical constraints or platform’s irreliability and they are “discovered” during the implementation. Failure scenarios aren’t less important than success scenarios (I would even say, more important!).

You can compare it with an army. Some army got an order to perform some steps, which also include going through some path. Of course, the path is important, it’s also important whether the path can be passed etc. – but it’s not the core. When we want to check whether the army did the right things, we need only to know whether the army performed correct steps and, simply, fulfilled the orders. If all particular elements of the order were fulfilled correctly, we can say that the army has correctly fulfilled the orders (not whether it passed correct paths).

So, first, most important thing is to define scenarios. The “scenario path” may pass through various execution paths and the program may get in various states (out of all the states contained in its degrees of freedom). That’s why there points on this scenario paths where we have a kind of “common conditions”. It’s like when you ride from one place to another throughout a town. You can enter various crossroads and jump from one highway to another. But you can also meet various conditions at these roads (usually traffic jam :)).

We are writing the code in particular programming language just in order to fulfill all correct scenarios. So, remember:

  • You want to have particular scenarios passed, so you write the code that should perform this scenario
  • Parts of the things you are using during the code spawn the “common points”, that is, points where various different conditions may occur and they may influence on the kinds of scenarios that may be passed at particular conditions
  • By using functions, on which’s result you have no influence (like system functions in kind of memory allocation, resource gain etc.), some scenarios, of course not desired by you, are spawned
  • In result, there happen to be created scenarios, which were not planned and which may lead your program to do things you don’t want it to do. You must catch every case when such a scenario has a chance to be spawned

But there is another problem: catch and what? Well, this is a kind of different situations: there occur scenarios, which are created by some conditions which you must have accepted, if you wanted to use some particular functions, so effectively you must add new scenarios to your program logics just because there are scenarios that just are, that is, new situations that you don’t need to occur, but they will occur, so you must plan, what you’ll do facing them.

In result, all scenarios, that you have planned, plus all scenarios, that you were obliged to create to respond for scenarios “automatically spawned” by the fact that you use some particular functions. Once you have all these scenarios documented, you must then select the common points, and be sure, that:

  • Every entering the common point has limited number of degrees of freedom of the global state (limited to the scope of this point)
  • Every exiting the common point will take a path that is assigned to any of scenarios that were planned by you (including planned scenarios for error handling, that is, scenarios that do not belong to the initial statements of the logics of your program)

Now, the test cases. The test cases may be either unit tests or component tests, it doesn’t really matter. What matters is that you make sure that every planned scenario and every theoretical scenario is tried and that the execution conditions are under control. And yes, only if all these scenarios are used in these tests can you say that you have the 100% coverage.

The tests are not required to perform always one particular scenario, nor there is a need that each test test different scenario. You should just plan tests as you think would be most appropriate. In ideal case, tests should be assigned to particular scenarios that go from one common point to another common point. Note that it’s always best if you have common points and have them in “well balanced” number. If you don’t have common points, it usually means that you have them, but they are “little common” – that is, your scenario is spanning through too long path. The common point is something like a pair of execution point and value of the global state (for all particle states which are available at this execution point). So, in order to have the common points well defined, you should have a) well limited number of particle states at every such point, b) predictable number of values of the “local state” (all particle states within this execution point’s scope). Note: the identity of a common point is a particular local state’s value and the execution point. If you have another combination of states, it’s another common point. So if you have too much common points, it usually means that your execution points usually have access to too many number of particle states. This is exactly the case I was talking about as “false dependencies” and the overuse of data sharing.

The only problem is that this is not automatically measurable. As I said, scenarios are also part of logics, so by nature, it’s unable to handle by automatic tools. In other words, automatic tools, no matter how intelligent, will not write test scenarios for you.

The problems that may cause bugs, then, are not only the code that exists and behaves incorrectly, but also the code that doesn’t exist, but it should exist. Such a code would be required to prevent some scenario to go out of control (that is, to prevent spawning a “singular scenario”). There’s no tool that may detect this just by looking at the source code. When, for example, a function returns an integer value, the tool cannot guess, that the values can be -1, 0, and every value greater than 0. Or that some function that receives the pointer, can accept null pointers and treats them in some special way, while some other function, receiving the pointer of the same type, doesn’t accept null pointers and treats this situation as error.

Summing up: when you create a program, you should have scenarios describing how the program should behave. Stating that you will have to use parts of code, which’s behavior and results do not depend on you (depend on user data, system resources, other libraries or directly other’s code), you will have to create additional scenarios that will manage (prevent) situations that you don’t want to occur.

Effectively all test cases you create to test your program do not test the paths or the behavior of the code (in contradiction to what many theoreticians and managers think). They test scenarios. Scenarios are logics, additional (say “tightening”) scenarios are logics, and singular scenarios are also logics. The program is correct only if it correctly behaves for predicted scenarios, contains tightening scenarios that prevent unwanted situations, and doesn’t ever run into singular scenarios, that is, scenarios that lead the program to behave in unpredictable way.

Remember: a program may work correctly for every possible scenario that programmers can spell up, but users can spell up more scenarios than you can imagine. Note, however, that if you make sure that you have limited the “kind” of data with one “statement”, for which the program will always behave the same and you treat data outside this kind as rejected always, your program will always behave correctly. But in order to do that you need to make sure that the complexity of your data is managed by the complexity of the code; otherwise the above checks for correctness will be useless.

The most important lesson from this is in two points:

  1. Program is logics, correctness of the program is logics, tests are based on logics. Despite that the program and tests are encoded in some programming language, the languages are nothing but a way to describe the logics. Logics is the key. Always.
  2. There’s no possible automated way to check whether your program is correct. Tools can only help you detect potential problems. The basic thing you have to strongly adhere to when you write your program is that the program is written clearly, predictably and with high testability.

10. How to make sure that programmers are doing their job well?

Yes, it’s malicious.

Of course, there exists no way to check it! For example, if you are a businessman and some programmers work for you, and you have absolutely no idea about software engineering – you will not be able to evaluate it any way. There are no tools, no technologies, no methods that you can use to evaluate them. There is only one way to do it: employ someone, who is a good programmer and whom you trust. Of course, the worse programmer this guy is, the less reliable can be his observations. You should know also his character and be sure that his reports aren’t tendentious or over the top.

This is also not unusual in any other kind of business. Trust is something that must be built by years and can be wasted quickly. It’s also one of the most valuable thing in business. Of course, you don’t have to trust fully everyone you employ, you just need a key person on a key stage. Someone you trust that they know that you need reliable information to make business decisions, while they also trust you that you will not make wrong use of these reports.

If you do not base on trust and believe more to the reports done by automatic tools than to the reports of trusted programmers, you may start receiving reports that do not reflect the true situation. In every report there are things like “white noise”, that is, data that change the nature of the report, but they are driven from the “independent” causes. When no one knows that these data are collected and no one knows about the methods of measurement, there is a great probability that this “white noise” can even up. But when someone knows about the method you use and the data you collect and, especially, the conclusions you drive from them – you can be sure that the people being “measured” are much wiser than the tools that base on simple equations, and this “white noise” will turn into “red noise”, that is, the “disturbing, but evening up” data will turn into the data that “disturb, deform, corrupt and distort”. And in result you’ll get nothing but the “true lies”. When you get into this situation, your programmers will most likely focus on how to sweep bugs under the carpet, not to fix problems.

Who is dishonest, they will see enemies everywhere around. Be sure that you know it, especially when many things depend on you.

Posted in Uncategorized | Leave a comment

References considered harmful

0. Introduction

References?

Yes. Of course, references are indispensable (I don’t count languages that state they don’t need them because I haven’t ever written anything in such a language), but it doesn’t change the fact.

References are always needed in a language which operates with large objects – and when you operate with something more than hello world or a simple “school” equation training, you will create large structures and dynamic data, which means that you need large objects and containers. If you want to operate with variables simultaneously, you need references.

Although references are something indispensable, they have also various properties, of which many provide lots of problems.

I don’t mean only references in, for example, Smalltalk. I should rather speak about pointers, as this is how it’s called in many programming languages. In case of C++ I mean exactly pointers, not C++ references, as C++ references have been named this way because they keep the “referring” feature of C++ pointers, but do not play many other roles that pointers do (and have also other limitations, for example they are not changeable). For C#, on the other hand, I mean references, as pointers in this language are used only as a low-level feature. Let’s state then that by “references” I mean reference in most of the languages and pointers in C, C++, Pascal, and other languages that have pointers, but not references 🙂

1. Maybe types

Well, ok, as I explained what I’m going to talk about, now I’ll talk about something different.

In some languages (especially functional languages) there are sometimes optional types, that is, a meta-type with any inside type being a value type, while the variable of this type may not only have values of exactly this type, but it can also have additionally a special value stating that there is no value at all. In C++’s boost library there is a template type ‘optional’ that does merely this thing, in Haskell it’s called “Maybe”.

A funny thing is that in C++ the ‘optional’ type is defined similar way as smart pointers, despite that it’s not a smart pointer. But there’s no other way, of course, to save the original type’s features and add some this meta-type’s specific.

Funny because the pointer itself does exactly the same thing. Of course, pointers should not be used in C++ to implement the “optionals” (in favor of boost::optional) mostly because the pointer by itself does not take care of the object of value type. But for objects of object type it’s completely enough.

But even leaving the matter of implementation beside for a while, it seems that ‘optional’ has more common things with pointer. Yes, if you, for example, create a pointer to bool type (and create two global constant variables initialized with true and false), then you may have a reference variable that points to true, points to false, or points to nothing, that is, its pointer value is null.

2. Nullable means optional

In other words, references that are always nullable (I mean, then, all mentioned languages but Vala), play the role of “optionals” (or Maybes). You can have a reference referring to some existing object, but you may have also a null reference, which does not refer to anything.

This is a new treat in OO languages; new and very unfortunate. It’s not true for Smalltalk, almost the only such programming language (including its dialects and extensions, like Self or Strongtalk). In Smalltalk you have only objects, but there is no such thing as “no object”; there is no “null” in the, say so, Java sense. This role is played in Smalltalk by ‘nil’, however nil is not a reference to nothing – it’s also an object, like any other being in Smalltalk. It’s only special in such a sense that if you send a message to (call a method for) this object, it will just do nothing (I didn’t checked what is returned by such a method, but I guess it just returns nil). It may end up with exception, probably, if you try to, say, add a number and a result of this call, which will not be able to be coerced. But still nothing wrong happens. You probably may check every argument you receive whether it isn’t nil, however it rarely makes sense; you’d rather check whether the object is of some class, or just try to call some method, which at worst will fail (usually by exception, called from doesNotUnderstand:). Anyway, in Smalltalk, despite that there is some ‘nil’ in it, it rarely makes sense to check the reference for it.

While in case of C, C++, Java, C# etc. (Vala does it behind the scene) manual checking every reference value received by a function for null is not so rare (I would even say, very often). Even if, usually, you would never have any situation of passing null there.

So this is the main problem with references. Maybe they do not do such mess in Smalltalk, but they do in the others.

Well, you can have a very similar thing to C++’s bool* type, or even better thing, with Java’s value wrappers – for example java.lang.Boolean. It’s a perfect trinary logic type: it can be true, false, and (nomen omen – see Haskell) “maybe”, when null. The actual problem with making Smalltalk way “everything object” in Java (even integers) was not only performance (although significant). The actual problem was that in Java if you create a method that gets 3 integer numbers in the beginning, declared as java.lang.Integer (not int), you’d have to first check them all whether they are not null, or otherwise you theoretically risk NullPointerException.

This was for me one of a very shocking experience when I came to the project being written in Java (after some experience with C++). So far I was sure that when I pass a string to a function then it can be only empty in worst case. This time the string passed to a function could be not only empty, but also null, and moreover, on the null string I could not perform any dumbest operation, even check if it’s empty. And moreover, I couldn’t do anything to prevent nulls against being passed to the function. This time was the first (but not last) when I felt like returning to C language.

Especially when I discovered a nice hack to omit the null String problem ("valid".equals( s ) instead of s.equals("valid")).

3. You are doing it wrong

First of all, then, you have to reconsider that if you are using a reference, you simultaneously allow such an option that this reference can be null. Effectively then, you’d better have some additional type with a reference that is never null.

In C++ you should first of all avoid using plain pointers in favor of some of smart pointers that can verify things for you. And first of all you’d better make sure that your data type should be a value type or an object type. This way, if you have a value type, you should operate this type by value only, and use boost::optional if you need optional types. If object type, then you should also make sure that you allow its reference to be null in specific situations.

There’s no much things that you can do in Java. Every user defined type must be a reference type there. However you can create a generic type, say ValidRef, that will wrap any reference type, return its object via get(), and set the object via set(), while by passing null to set() will always end up with exception (not of RuntimeException class). There is still one problem, though – you can’t prevent passing null where a reference of ValidRef is required. Anyway, this is still not a solution.

In C# you can use structs for types that should be value types, and such types cannot be ever nullable. You can still make types optional by using ‘?’ mark.

In Vala the situation seems to be the best, as this language supports defined nullability. Unfortunately it’s not perfect in this statement. As types are non-nullable by default, while they only can be made nullable (by adding ?), it should be then forced that a type returned by an external function may only be nullable (and it can only be non-nullable only in some specific cases, explicitly defined). Unfortunately it’s a relatively new feature and it’s used not in every place where it should be, so this nullabiliy is only visible by generated null-checks. While it should be defined that a nullable reference cannot implicitly convert to non-nullable, it can only be done explicitly, and such a conversion may end up with exception if the source value is null. In other words, it would be desired that as Vala already has such a great feature, it should make this nullability check statically supported, with also good dynamic support that will throw exception not when a null was passed to function, but already at the first time when a null has been assigned to a non-nullable type. If you have read my article about bugs, you should know that this is the only right thing to do: report the error at as earliest place as possible.

You should also remember that in C++ in some specific situations pointers may well contribute to the “optional” feature needed – in particular in places where a function expects a reference to a variable. Last time I have been researching some code in WebKit and there was some function in JavaScript engine that evaluates the call and returns an information of possible exceptions. It took two arguments: one was of ‘bool&’ type, which was a reference to a variable to be set, and one was of ‘bool’ type, which when true requested that the variable be written accordingly.

This is an example what is the wrong thing to do. What was expected in this case was to write to a variable only when it was requested. This way, the plain pointer was the best thing to be used in this place: when a user passed an address of a variable, it should be used to be written; when there is NULL passed – it means that the user doesn’t want to get this information. Very simple, and conforms to a well known rule that reference types should not be used to pass variables to a function to be written, as this fact is not visible at the call place. While I think this is not the right rule as it allows a null pointer to be passed, in this particular case the null pointer case would be exactly one of expected things.

4. More precise?

Remember: be most precise as possible; if a language does not define tools for you to explicitly state it, at least remember to add some statements in the documentation.

Also, make sure that you are conscious about types’ properties: whether it’s a value type, whether you want it be optional, whether you expect only valid objects. I think I’ll finish with that because a bit more things like that and I’ll say shortly: THINK, STUPID! Well, this is useless because if you think, you’ll come to all these statements without my help, and if you don’t, you just waste your time anyway.

Posted in Uncategorized | Leave a comment

Multimethods considered harmful

(This article is partially a follower for the “Visitors unvisited!” article.)

0. Multimethods and “super hyper advanced object oriented programming”

Multimethods is one of buzzwords in today object oriented programming. It’s considered to be some “higher level” of object oriented method dispatch (which dispatches by multiple objects, in contrast to single dispatch).

My previous article, “Visitors unvisited!” stated in the beginning that bad things have happened to design patterns, as they are used more to torture students than in practical use. The practical use is “how to solve my problem” and the practical use of design patterns is “I can use this design pattern to solve this problem”. Unfortunately, the majority of descriptions of design patterns found on the net just describe what particular design pattern is, however there’s no examples of what kind of problems you can solve with it. And lack of such examples disqualify design patterns in full length: what worth is a pattern for which no practical use can be shown?

I have already written about this, as an example taking the Visitor pattern, for which it’s even rarely stated that this is a way to implement multimethods in languages that offer only single dispatch, but they provide also function overloading by static types.

That’s really bad that Visitor is more a way to torture students and job applicants than a tool to solve practical problems, that is, to implement multimethods.

Although, it’s much worse than that. Multiple dispatch is also a kind of tool, which is widely explained on the net about how it works. For this thing there’s also a big hole of lacking problems to be solved by this tool. In particular, there’s no practical example to show that in this particular case multimethod is the only or the best way to solve it.

I know it may sound astonishing for some people because it’s really obvious that Visitor pattern is a way to implement multiple dispatch (that is, to have a development replacement for multimethods). What can I do – “super educated master developers” creating better and better languages are also “exploring America” by “revelatory” statements that “multimethods make Visitor pattern useless“. Yeah, guys, visitors are useless because they are useless anyway.

These guys should be really treated as extremely well educated. Other guys seem to not even know that Visitor is only for multimethods and using them for anything else is useless even theoretically.

1. What are mutlimethods?

Multimethods generally base on a statement that, in contrast to single dispatch, where for an operation with one object being a “pivot”, and deciding how to perform requested operation, for multiple dispatch there are two or more objects that together decide about it. For example: we can have an operation “op” and would like that an object kept by reference do “op” and in response some function will be called – which one, it depends on the type of the object. For multiple dispatch, we do “op” passing two or more objects (plus maybe some additional arguments) and which function will be called in result, depends on the combination of types of these two objects.

It’s easy to imagine what it means in practice. As in case of single dispatch you had to provide an override for every class that derives some abstract class, for double dispatch you have to provide “a square more” (a cube more in case of triple dispatch and so on). For example, if we have first hierarchy, with class A and derivatives B, C, D, operation op( A ), requires op( B ), op( C ), and op( D ) overrides. If we have a second hierarchy, with class “W” and derivatives X, Y, Z, and operation op2( A, W ) (being a multimethod for A and W), then we need to provide the following overrides: op2( B, X ), op2( C, X ), op2( D, X ), op2( B, Y ), op2( C, Y ), op2( D, Y ), op2( B, Z ), op2( C, Z ), and op2( D, Z ). Total 9 of methods, which is the number of derived classes in both hierarchies multiplied by each other.

I’m not going to focus on implementations. As I have already mentioned, the Visitor pattern is one of the most famous implementation; there are also implementations for Python and Java. It’s not the most important for my article; I’d like to rather focus on finding any practical use of multimethods.

2. The basics of object-oriented programming

Yeah, funny. I know. But what can I do – I must start with reminding what really the object-oriented programming is if I want to explain, what I can see wrong in multimethods.

I have already used the word “pivot”. I am not sure whether this is the best word to describe it (English is not my first language), but as I know its explanation in mathematics and linguistics, the object for which the method is called, can be named with this word. Since generally when you call a function, you pass some arguments to the call and theoretically the object, for which the method is called, is also an argument of this call. (That’s why people who like functional languages would also say that this object is some very artificial and unnatural thing.) So it would be most safe for me to define this object as “pivot”.

This “pivot” is an object, which has one special meaning among other arguments passed to the call, that from this object there is read some characteristic information that allows it to retrieve the correct implementation of the defined operation.

This exceptional role of the pivot relies on the rule that the operation to be performed on an object is abstract. And the pivot is used to “instantiate” this operation, that is, state what “op” means in its particular case.

Note, however, that this too is a kind of a “design pattern”! And, by the same rule, people also are often confused about “well, should I use this pattern here, or not? And how?”. There is a famous confusion about the operation of sending data to the stream. Should a data print itself on the stream or the stream should print the data on itself? I’ll cover that later because this is also one of the most interesting cases in the topic of multimethods.

3. Multiple abstractions

Abstraction in object oriented programming is strictly bound to abstractions in the real world: abstractions base on terms. There is an abstract term and there can be some abstract treat or abstract operation that can be performed for a “live being”, matching this term.

Well, so: are the two types of objects so special that abstractions should be defined on the level “how both of these types would perform this operation together”? Let’s consider, for instance, an abstraction for “a man drives a car”. In abstraction it would be “a drives b”.

Does it make sense? Well, we can try to define it by many ways, but probably it will be most reasonable to have some limitations (base classes) on these two objects that would perform this operation. So we can do it the following way: “Driver drives Drivable”. So, does it make sense? That is, can we think about multiple cases of Driver and multiple cases of Drivable that can “intercommunicate” by various ways?

Well, it may make sense, but what for. We can define the “drive” as a set of operations that can be performed on “Drivable” and as long as the “Drivable” supports them, they can be used by “Driver” – while “Driver” may use these simple operations by various ways, as long as it makes sense for it.

Generally, you can define “drive” operation as set of appropriate accelerating, braking, and turning. If we can define these three operations – we can define the “drive” operation without having any least knowledge on the details of both Driver and Drivable! The “drive” operation can be defined by using only abstract operations of Drivable and Driver. And no matter if the Driver is a man sitting behind the gears or this is a built-in automaton, only the way the operation is being requested from Drivable differs – not the operation itself. So, both cases of Driver should define what they mean by controlling, and all cases of Drivable should define what they mean by accelerating, braking and turning. Now the “drive” operation needs only to cause “controlling” by Driver and cause accelerating, braking and turning by Drivable.

Of course, this can too be defined on the base of multimethods, but the most important question is: what’s the reason? The only difference is that the number of methods you have to cover is in a level of exponentiation more! Look: When you have two Drivers: Man and the Machine, and three Drivable-s: Car, Tank and Horse (no matter how funny sounds an automatic driver for a horse :), you need only to define: Man::control (using gears with hands and legs), Machine::control (access the lowest level control terminals by electronics or engines), Car::accelerate (engine), brake (brakes), turn (steering wheel), Tank::accelerate (engine), brake (engine plus brake support), turn (caterpillar engines differentiation), and Horse::accelerate (run faster), brake (run slower), turn (just turn :). Total number of overrides is 2*1 + 3*3, that is 11. For multimethods in the above equation you’ll have to change + to * in the above equation, so the result is 2*1*3*3 = 18. Note, however, that these method overrides are not equivalent – in case of multimethods one override does things of two methods in single override. It should be then multiplied by two. Even stating that the factor may be 1.5 because simplified information interchange methods can be used due to more direct access, it makes 27 required overrides, which is still more than twice comparing to 11 (plus one general method based only on abstractions) in single inheritance.

But it doesn’t even matter that so many methods you’d have to define. Much more important is that… what’s indeed the difference between “A man rolls the reins of the horse” and “A machine is turning the steering wheel”? If you describe them both as “A driver is using drivable’s means to make a turn”, the difference is none. There may be only a difference about what it means for the driver to use driveable’s means and what it means for the drivable to make a turn – but all these things can be defined on the driver and drivable abstraction levels. The only abstraction, which can be here, is how the data between these two objects are interchanged in this operation. The operation itself isn’t an abstraction at all – what is abstraction here is the information interchange. Do we really need abstraction to express this thing?

Now you can see now how reasonable can be abstractions based on multiple terms. Remember: unless you can define reasonably a term of “multiabstraction”, don’t try to state that multimethods are reasonable. Can you?

Of course, we may have not enough details defined in abstract types and we need them in this operation… bullshit! None of implementations of multimethods or their replacers states that private, inaccessible otherwise data should participate in this operation. This is just the reason, for which abstractions are needed for base classes: let’s state some general forms that must be provided by every derived class.

If you say that you need multimethods because your base class does not provide appropriate abstraction on which you can base your general operation, then maybe you just need to sit down and add them? If your crossing operation needs two concrete objects together, maybe by some reason it’s not enough to have them each other single as concrete class and both together as abstract class – and this “some reason” is “not present appropriate abstraction methods” and “not present appropriate abstract methods for intercommunication of objects”?

4. Multiple pivots?

The problem of pivot is indeed a very similar problem to the problem of inheritance. The object-oriented programming states that there can only be one base class, just similarly, as there can be only one object to decide about the operation’s implementation.

But we have also multiple inheritance. What is multiple inheritance in the terms of object-oriented programming? I describe this topic well enough in another article, I’ll try to be short here then.

First, you must reconsider that inheritance has two aspects: extending (adding new features towards the base class) and subclassing (changing or defining the behavior from the base class). While you can extend multiple classes without problems, it’s a strange thing to have a multiple subclassing.

This is an important difference between extending and subclassing: when B subclasses A, it means that if an operation cannot be defined by class A, it’s redirected to B (whatever it is); while when B extends A, it means that B defines an operation the same way as A unless an explicit override is provided. It means that while it’s normal that the extending B does operation “op” as defined in A, or whatever else derived class defines it, it’s strange what to say when “op” is performed for an object of class B, which subclasses A, X, Y: as B does not define it then the class redirects it to what? All of them? Only those that “understand” it? Only the first one?

The first one, well… a tempting offer. That’s how Smalltalk manages it: it does not offer multiple inheritance, but offers delegation. You can treat it as multiple inheritance, where the first class has special meaning. It can be even treated as a “pivot class”.

And that’s what this “pivot problem” is.

If you have one of the following cases in C++: only one polymorphic class in the hierarchy, two polymorphic classes with one common virtual base, or two polymorphic classes from two separate polymorphic hierarchies – you still have only one “pivot” base class (that is, the base class that defines what to do for particular operation if the derived class did not define it). In first two cases, this polymorphic base class is the pivot, and the last case is the “class composite”, where each composite member has its own pivot.

Only in the case when you have two polymorphic classes coming from the same root as base classes, may you have a kind of “two pivots”. But such a hierarchy is disallowed in object-oriented programming.

(By the way, note that in Java, C#, and many other languages, you always have a class that is a base for all classes, usually named “object”; also there are no “compact” classes or at least they cannot participate in inheritance. It means that the second case – two separate polymorphic hierarchies – is not possible to be done in these languages. In consequence, they do not allow for multiple inheritance.)

The term “method” comes from object-oriented programming and the core meaning of object-oriented programming is “I order object o to do operation p and let object o do this p the way it understands it”. If you are doing something else with the object, it’s not object-oriented programming. How would you then end the sentence “I order objects o and q to do operation p…” ?

What it means for the matter of pivot in case of calling a method, is that the situation of “two pivots” is something unnatural and illogical. Nonetheless, in whatever domain you’d understand the word “pivot”, the “two pivot” term is an oxymoron. So if you confirm that object oriented programming has been inspired by the logics of surrounding world, multimethods is just the point, where this logics ends.

Moreover, in the “multimethod” term the matter is not that objects o and q do operation p. The matter is that object o do operation p adjusted to object q. Or object q do operation p adjusted to object o. Or even “let’s do operation p adjusted to o and q together”. Maybe, then, the problem is with not having something important in the middle?

5. Which one is pivot?

So let’s go back to the problem: when we want to write an object to a stream, should the object print itself on the stream, or the stream should print the object on itself?

Some people say that this problem is unreal. Not exactly. The problem here is the level of information privacy.

However pay attention that this is a typical problem of two-kind polymorphism: there can be many various implementation of a stream, and there can be many types of objects to be printed on the stream. Actually both the stream and the object may implement the way it is being printed!

Isn’t it then the typical problem for which the multimethods have been developed?

So, if this is the typical problem to be solved by multimethods, why it’s never solved by multimethods or any replacement mechanism like visitors?

And how is it solved?

Well, usually the stream have defined methods for sending to it values of some basic types, including string. In other words, there’s something like stream::write(), which is then implemented in specific way. Then, if you want to print your type object to the stream, just convert it to some “printable” type (usually string) and use an existing stream::write(). You may also use several stream::write() methods for multiple parts of the contents of your object. In final effect, all versions are glued together with one common function (in C++ it’s operator<<).

What does this example teach to us?

If the operation requires access to vital data of two objects, there must be some way to convert them into some common (interchangeable) representation. Moreover, in practice every stream should only have the write method for string – for all other data types they are first converted to string and then redirected to the string version. So, it means in practice that this interchangeable representation is string. This way, all that particular stream class has to do in its write() method is to implement write( string ), while all that particular object class's implementor has to do to make it printable on a stream is to convert the contents to string value. That's all. And it really works – since a long time this interface has allowed that objects of various types may be dumped to various kind of streams.

There is also another interesting example, that may be implemented by multimethods: adding two numbers. We have an abstract Number class, and we have operation:

Number add( Number, Number )

where Number may be Integer, Float and Complex (let’s state Complex of Float). You’d have to provide adding Integer to Integer, Float to Float, Complex to Complex, Integer to Float, Integer to Complex, Float to Complex – total 6 of methods. If you add one more Number class, you’d have to add one for both types identical, then one per existing class (3 in this case), so additionally 4 methods. For next one, it will be 5 additional methods. And so on. As both arguments have the same base for the hierarchy, you only have to fill the upper to diagonal – lower to diagonal can be defined by inversion (however it’s the case when you create an operation for two objects of the same abstract class – if they are different abstract classes then usually whole matrix should be filled).

But, unfortunately, in today implementations this is not solved by using multimethods. And it’s not because multimethods are weakly supported. It is because using multimethods is stupid: in the case of Number, much better would be to define only adding for the same types and conversion between types. This uses, however, still some special case: number classes can be ordered from least significant (Integer) to most significant (Complex), that is, an object of less significant class can be converted to more significant class loselessly, however in opposite way only with narrowing (losing part of information). So the other object of the operation is “adjusted” to more significant class of them both and object of this class is returned.

However, whatever example you show to be a good for implementation with multimethods, each such example may have this “something special” that causes that this can be still implemented different way – much simpler in every case. In this case it’s the orderable treat, in case of streams it’s the common data representation. In other cases maybe there should be some specialized object that may have access to private data. Whatever. Implementing this by using multimethods, even in the form of visitor, will make this work always more effort consuming.

6. Are there any typical examples?

Is this a rule? Is there always a method for retrieving data for each object?

There is a faaaamous example of intersection of two shapes. Everyone gives this example as multimethods (including Bjarne Stroustrup in his “Design and Evolution of C++”) – but no-one shows how this intersection methods should be implemented. No-one has even tried. It’s not impossible that if I can see this implementation, I can quickly show, how to do the same, without multimethods or some their replacement, and simpler. Probably some of people who refer to this example, do not realize that this operation is really complicated, and using exactly stated types of two shapes for which we want to test intersection, is no help here.

I have tried to approach (just theoretically – this example really doesn’t look interresting for me and is not practical at all) to implement it. And my first idea was: why not define a method that will do something like “trace shape”, allowing to detect common points of two shapes? Why not having something like “return set of functions that describe parts of the shape”? When you have such methods, you are able to easily implement the intersection… without knowing concrete class of the shape. You can just encode it in the frames of the Shape class. Moreover, in this particular case you don’t have to expose the “functions” method to public because only the Shape class needs to know this information.

And if you say that you would have to reveal too much information to public, try to first verify how deep access to information you would have to ensure if you want to create implementations for all combinations of two (or more) classes. Because if you don’t need details of these classes, then why isn’t it enough for you to encode your operation just once, using abstract virtual methods?

I have tried “mutlimethods example” and “visitor example” in web search engines. The majority of them were explanations how the visitor pattern works or how multimethod works, given some simple examples, usually with – guess what? – and usually so stupid like those mentioned in the beginning. There was one site where the autor was tempted to spell up some really practical examples of multimethods, however the implementation isn’t shown, nor any discussion was provided to at least try to prove that this method seems to be best or that it at least makes sense.

7. Is there any “value added” from multimethods?

All available informations about multimethods make me think about them as about a kind-of scientific aberration, a curiosity, which shouldn’t ever reach the practical software engineering. They are cherished by Java programmers (especially those, who just discovered them), they are implemented in python (a language, for which 80-20 rule becomes 95-5), they are also implemented in many other languages, which for industrial software engineering are niches.

Actually, I cannot find an example, which is using multimethods, which cannot be translated into normal single-dispatch based code. Multimethods are just useless, and I would even make a statement that using them means that the need of using multimethods is simply a design flaw.

So?

To all developers who participate in creating and designing programming languages: guys, I really believe you have lots of much more important problems to solve. Forget multimethods and you’ll really stop wasting your time.

Posted in Uncategorized | Leave a comment

Visitors unvisited!

Do you know the Visitor pattern?

If you don’t, it’s bad. If you do, it’s much worse.

Try to find any explanation of the Visitor pattern on the net. For 99% of probability you’ll find a page that shows you example classes, where we have some “object” classes having “accept” methods and a “visitor” class that has a “visit” method.

Imagine that you don’t know what hammer and nail are and someone is going to explain it to you the following way: “Hammer is a tool that can be used to hammer a nail and nail is something that can be hammered using a hammer”. The majority of explanations what Visitor pattern look exactly like that. Moreover, the same holds about explanations of other Design Patterns for software engineering.

Every thinking programmer would first ask an obvious question: why would I ever need something like that?

Let’s try the example that can be found at Maciek Sobczak’s Inspirel page:

class Hammer;
class Drill;
class Saw;

class Visitor
{
public:
    void visit(Hammer & h) = 0;
    void visit(Drill & d) = 0;
    void visit(Saw & s) = 0;
};

// root of the given hierarchy
class Tool
{
public:
    virtual void accept(Visitor & v) = 0;

    // regular operations of Tool omitted
};

class Hammer : public Tool
{
public:
    virtual void accept(Visitor & v) { v.visit(*this); }

    // regular operations of Hammer omitted
};

class Drill : public Tool
{
public:
    virtual void accept(Visitor & v) { v.visit(*this); }

    // regular operations of Drill omitted
};

class Saw : public Tool
{
public:
    virtual void accept(Visitor & v) { v.visit(*this); }

    // regular operations of Saw omitted
};

class DoSomethingVisitor : public Visitor
{
public:
    void visit(Hammer & h)
    {
        // do something with the hammer
    }

    void visit(Drill & d)
    {
        // do something with the drill
    }

    void visit(Saw & s)
    {
        // do something with the saw
    }
};

vector<Tool *> myToolBox; // filled with lots of tools

void doSomethingWithAllTools()
{
    DoSomethingVisitor v;

    for (size_t i = 0; i != myToolBox.size(); ++i)
    {
        Tool & t = *(myToolBox[i]);

        t.accept(v);
    }
}

The following things come in mind when you look at that example: why do we use those confusing names ‘visit’ and ‘accept’? First, let’s try to analyze this case starting from the top-level entity:

vector<Tool *> myToolBox; // filled with lots of tools

void doSomethingWithAllTools()
{
    DoSomethingVisitor v;

    for (size_t i = 0; i != myToolBox.size(); ++i)
    {
        Tool & t = *(myToolBox[i]);

        t.accept(v);
    }
}

What we want to achieve is: “let an object of class Visitor do what this object states is appropriate with the given object of class Tool”. So the most readable form of this code should look like this:

vector<Tool *> myToolBox; // filled with lots of tools

void doSomethingWithAllTools()
{
    DoSomethingVisitor v;

    for (size_t i = 0; i != myToolBox.size(); ++i)
    {
        Tool & t = *(myToolBox[i]);
        v.invoke( t );
    }
}

Which reads: for every tool, invoke the Visitor object with the Tool object as argument.

Nonetheless, the question is still not answered: why would we need something like this? Moreover that finally the call to ‘accept’ method gets into DoSomethingVisitor::visit() for appropriate argument.

However here we have one important hint: we call it directly as a method of DoSomethingVisitor, a derivative of Visitor, which makes us to think (what is not shown here, though) that probably it will end up with calling this ‘invoke’ (or ‘accept’ as in previous example) using a reference for Visitor, not for DoSomethingVisitor. Effectively then, the most simple form would be:

vector<Tool *> myToolBox; // filled with lots of tools

void doSomethingWithAllTools()
{
    DoSomethingVisitor v;

    for (size_t i = 0; i != myToolBox.size(); ++i)
    {
        Tool & t = *(myToolBox[i]);
        invoke( v, t );
    }
}

So we can treat this in two ways:

  1. more generic, as multimethods
  2. a bit specialized, as invoking a generic action on an object of generic type

Interresting, isn’t it? The Visitor is generally a method of solving the problem of multimethods, but no-one says it directly. It wouldn’t even be so extremely stupid, unless the visitor was never reasonable to be used for any other purpose – but let’s not anticipate:). The same example would be rewritten in even simpler way than Maciek has shown using enumerations:

// First, fill in the mappings:
void doSomething_Hammer( Tool& t )
{
   Hammer& h = static_cast<Hammer&>( t );
   // do this something
}

void doSomething_Saw( Tool& t )
{
   Saw& h = static_cast<Saw&>( t );
   // do this something
}

void doSomething_Drill( Tool& t )
{
   Drill& h = static_cast<Drill&>( t );
   // do this something
}

map<type_info*, void (*)( Tool& )> tool_dispatch_map;

// For convenience
#define REGISTER_TOOL( Toolname ) tool_dispatch_map[&typeid(Toolname)] = \
          &doSomething_##Toolname

void register_tools()
{
    REGISTER_TOOL( Hammer );
    REGISTER_TOOL( Drill );
    REGISTER_TOOL( Saw );
}
#undef REGISTER_TOOL

// root of the given hierarchy
class Hammer : public Tool
{
public:
    // regular operations of Hammer omitted
};

class Drill : public Tool
{
public:
    // regular operations of Drill omitted
};

class Saw : public Tool
{
public:
    // regular operations of Saw omitted
};

vector<Tool *> myToolBox; // filled with lots of tools

void doSomethingWithAllTools()
{
    for (size_t i = 0; i != myToolBox.size(); ++i)
    {
        Tool & t = *(myToolBox[i]);

        void (*handler)( Tool& ) = tool_dispatch_map[&typeid(t)];
        if ( !handler )
        {
             clog << "INTERNAL ERROR: no handler for type " << typeid(t).name() <<
                     "\nPlease review register_tools() function\n";
             return;
        }

        (*handler)( t );
    }
}

What’s better here? Well, first of all it doesn’t need an additional enumeration type. Second, there can be exactly one place in the code where all the expansive definitions reside. However both these examples have one important common thing: they don’t use any “Visitor” class. Guess, why?

Indeed, when we are using the Visitor class, the path through which the invocation goes from the shown ‘invoke’ or ‘accept’ method calls to DoSomethingVisitor::visit() implementation relies on selecting an appropriate implementation basing on two objects – in contrast to usual virtual methods, where the implementation is selected basing on one object (the main object in this case). As languages like C++ or Java don’t have such thing as multimethods, they usually use Visitor pattern to achieve it. (For Java, there is an extension called MultiJava, which supports multimethods; indeed the Visitor pattern is easier to be used in C++ because Java needs that the identical body of method, that redirects accept to visit, be manually repeated in every deriving class.)

But the second way is even more interresting because, as it’s written on the Wikipedia’s Visitor page, it is a way of separating an algorithm from an object structure upon which it operates. Of course the ‘algorithm’ term is not treated as ‘group algorithm’, let’s name it so, as it is used by STL – this is an algorithm that defines what to do with particular object rather than a collection of objects.

So why such a confusing way of getting from the invocation on two generic objects to the actual implementation? This is specific for C++ (and alike) because we want that its mechanisms help us determine the required implementation – in particular, we use the virtual call (indirect call) and overloading.

First, we use virtual call so that the ‘accept’ method is selected for the actual type of particular Tool object, not as of Tool class (although this is the only reason why we override this method in a class derived from Tool because the body is identical as in the base class). Then, this object in ‘accept’ method is calling ‘visit’ on a (given in ‘accept’) generic Visitor, passing itself in concrete type, so that we use overloading. Then, the generic Visitor is virtually calling its ‘visit’ method, overloaded for a concrete type of Tool. Finally, we get into the implementation of ‘visit’ method in a concrete Visitor having the concrete Tool. In other words, this mechanism selects a method that matches only and exclusively this pair of types.

The question is, however, whether this kind of complicated dispatching is really worthwhile. Indeed, the virtual call is nothing else than indirect call, so IOW nothing else than setting a pointer to a function to particular pointer variable and then mapping this variable to the pair of two types! I have already shown, how the mapping a type_info object to a pointer to function can be used to fulfill this.

This is the hint how to solve the problem of multimethods. Visitors seem to be a bit specialized, as I said, because we have the ‘operation’ object and the ‘subjective’ object. We can state some special cases because of that.

The most simple way of creating multimethods is shown in Bjarne Stroustrup’s “Design and Evolution of C++”, where he shows a multimethod selecting an implementation for two objects of the same base class, but various concrete classes. This solution can be used also for two hierarchies and don’t need things like “visitor pattern”. Indeed, the visitor pattern isn’t any better – it holds exactly all the disadvantages of multimethods by virtuality/overloading because it’s intrusive as well, which isn’t wondering given that both solutions base on virtuality and overloading, too.

So, how can we fix this, or better, what problems can be solved? There are two: how we can decrease the number of required changes when adding a new class to any of these hierarchies, and how we can make this solution totally non-intrusive.

The first problem can be solved in C++ the following way: we create an intermediate class that will derive from Tool and needs that the name of the currently created class be put as its parameter. This intermediate class will have appropriate method that redirects to given class, making it first cast to the concrete type (this “concrete type” is passed as a template parameter, which is the same as the class you define). This way you solve the problem of the needs to repeat the same body of redirecting method in every class that derives from Tool. I don’t know how this solution can be provided in Java – maybe. I didn’t try.

Let me show this solution. Note that the class hierarchy here is a bit different, because I’m going to show how Visitors should be really used, so there are two hierarchies with two class each. This is a complete example – you can compile and try.

#include <iostream>
#include <vector>

using namespace std;

class Hammer;
class Saw;

class Action
{
public:
    virtual void invoke_for(Hammer & h) = 0;
    virtual void invoke_for(Saw & s) = 0;
};

// root of the given hierarchy
class Tool
{
public:
    virtual void chain_dispatch(Action & v) = 0;

    // regular operations of Tool omitted
};


void invoke( Action& a, Tool& t )
{
 t.chain_dispatch( a );
}


template <class Derived>
class DispatchActionTool: public Tool
{
public:
 void chain_dispatch( Action& a )
 {
  a.invoke_for( static_cast<Derived&>( *this ) );
 }
};

class Hammer : public DispatchActionTool<Hammer>
{
public:

    // regular operations of Hammer omitted
};

class Saw : public DispatchActionTool<Saw>
{
public:

    // regular operations of Saw omitted
};


class CleanAction: public Action
{
 virtual void invoke_for( Hammer& h );
 virtual void invoke_for( Saw& s );

 // regular operations of CleanAction omitted
};

class UseAction: public Action
{
 virtual void invoke_for( Hammer& h );
 virtual void invoke_for( Saw& s );

 // regular operations of UseAction omitted
};

//////////////////////

void CleanAction::invoke_for( Hammer& h )
{
 cout << "Cleaning hammer\n";
}

void CleanAction::invoke_for( Saw& s )
{
 cout << "Cleaning saw\n";
}

void UseAction::invoke_for( Hammer& h )
{
 cout << "Using hammer\n";
}

void UseAction::invoke_for( Saw& s )
{
 cout << "Using saw\n";
}

vector<Tool*> myToolBox; // filled with lots of tools

void performActionOnWholeBox( vector<Tool*>& toolbox, Action& a )
{
    for (size_t i = 0; i != toolbox.size(); ++i)
    {
        Tool & t = *(toolbox[i]);
  invoke( a, t );
    }
}

struct FDelete
{
 typedef void* argument_type;
 typedef void return_type;

 template <class T>
 void operator()( T* x )
 {
  delete x;
 }
};

int main( int argc, char** argv )
{
 myToolBox.push_back( new Hammer() );
 myToolBox.push_back( new Saw() );

 Action* a1 = new CleanAction();
 Action* a2 = new UseAction();

 performActionOnWholeBox( ::myToolBox, *a2 );
 performActionOnWholeBox( ::myToolBox, *a1 );

 for_each( ::myToolBox.begin(), ::myToolBox.end(), FDelete() );

 return 0;
}

What is important in this example?

So, start from performActionOnWholeBox(). This function walks thru the whole container and performs given action on all objects in the toolbox. What operation will be performed for particular object, it depends on the details of particular action object and particular tool object. I used a standalone function invoke() to show that this is not so an obvious thing which of its argument should be an object for which to call a virtual method.

Now: what you need to add a new Action:
– create a new class derived from Action
– add overloaded invoke_for() methods for all Tool classes as others have
– create implementations for invoke_for() with all available Tools

And now, what you need to add a new Tool:
– create a new class derived from DispatchActionTool<new class> (not from Tool!)
– review all Action classes and add an entry for your new tool class
– add an implementation for invoke_for() method for all Actions with your type

Too much work? You’re kidding. You always have to add an implementation for a crossing of two types; this is such a nature of multimethods. So, if you have a UseAction and CleanAction, and you create a new DestroyAction, you have to add this class with implementations for already existing Tools, so you have to implement DestroyAction for Hammer and DestroyAction for Saw. Similarly, if you want to add a new tool, say, Drill, you have to create a CleanAction for Drill and UseAction for Drill. This is the minumum you have to do stating that you have two interoperating class hierarchies. Minimizing work can be things you can shorten only behind this one.

Note also that this solution provides still the fastest code because the most important part of dispatching occurs at compile time, and any run-time dispatching is only dereferencing pointers. We still can use mappings that will search for the implementation (that is, select appropriate pointer to function) basing on a pair of two types. Types specified in run-time, of course, for example by type_info. This requires nothing more but creating these implementations and mapping them for specified pair of types. Worthwhile, if means really less work – whether it is indeed, evaluate by yourself…

So, is the Visitor over-emphasized? Yes, but stating that even the lazy initialization is called a Design Pattern, let’s say that Visitor is also a Design Pattern. But if you are going to use any Design Pattern it’s best that you first get known of possible problems they solve and possible schemes it can be used for. Treating Visitor as a Design Pattern itself makes a developer more stupid that they can’t even eventually develop in Java language.

(This topic is partially continued in the next article, Multimethods considered harmful.)

Posted in Uncategorized | Leave a comment