First, let me say that I personally do not think there is anything particularly wrong with Lisp. Thus, in this essay, I will not really attempt to answer the rhetorical question of the title. I will, however, attempt to analyze some commonly recurring criticism of Lisp in an attempt to shed some light on the question and why it is often asked. Let me start by saying that, for those who do not know, Lisp is a family of languages, including Common Lisp, Emacs Lisp, and several dialects that are only marginally used today. Sometimes, even the Scheme language is considered a member of this family. Whether that is the case or not depends on what your exact definition of `Lisp' is, which is too complicated (and uninteresting) a question for this essay. I will mostly use Lisp to mean Common Lisp, and sometimes, when convenient, include Emacs Lisp as well.
Lisp has been around for a long time, though it has been transformed considerably since it was first invented (some say `discovered'). Today, it is a modern, multi-paradigm language which has perhaps the most sophisticated features of all general-purpose languages in use (the object system, macros, reader macros, conditions, etc, etc). The first question one might ask is then "If Lisp is so good, why is it not more popular?". People who ask such a question typically assume that "good" implies "popular", and thus look for some part of Lisp that is NOT good, which would explain why it is not popular. However, there is absolutely no reason to assume that "good" implies "popular", so the question is really quite naive. But, instead of just ignoring the question, in parts of this essay, I will momentarily take it at face value and try to explain why Lisp is not as popular as its followers think it deserves to be.
Let me make a short remark, however. Lisp is not THAT unpopular (and now I will include Emacs Lisp for convenience). In a recent line count, Lisp came in as number 4 when it comes to number of source lines of code (SLOC) in the Debian GNU/Linux (Woody) distribution, after C, C++, and shell with some 4 million SLOC (around 4%), before Perl, Python, ML, Fortran, etc. That is hardly an unpopular language, only less popular than C and C++. Granted, the number of SLOC in Debian GNU/Linux is not the only metric of popularity possible, but it is an indicator as any others.
Recently, I saw an article in comp.lang.lisp in which the author seriously "knew" that there must be something wrong with Lisp, because if is so good, the market would find out, and Lisp would start being used in software projects all over. The fact that this has not happened "proved" to that author that there MUST be something wrong with Lisp, even though he did not know what it is. Such naive belief in the abilities of the free market to further the good and suppress the bad is laughable at best, but in fact real sad, because it leads to some very incorrect assumptions and some very wrong decisions.
Though, perhaps there IS indeed something wrong with Lisp after all, because it seems to attract all kinds of crackpots, though maybe I am ignorant and simply do not know that this is the case for other languages as well, or for every other human artifact as far as I know. The way it typically manifests itself in the case of Lisp, is that someone new to Lisp shows up in the comp.lang.lisp newsgroup for the first time, gives the highly-intelligent and very knowledgeable people who hang out there a lesson about what they haven't understood about why Lisp is unpopular, and proceeds to tell them how they should change the language to fix the problem (usually, change the syntax to get rid of the many parentheses), or simply to inform them that they have made a mistake and ought to be using some other language instead.
It is my opinion that these crackpots hold the key to why Lisp is not as popular as we think it deserves to be. Lisp is sufficiently different from what most people already know and expect in a programming language, that they just aren't willing to make the effort. In a different essay, I called these people "performance oriented", and they (unfortunately) make up the vast majority of people in general, though perhaps disproportionally so in software developers. To summarize that essay, they are simply victims of an incredibly strong psychological force (that exist to some degree in all of us) that makes the person attempt to explain away something new as being bad or useless, just in order to avoid the hard work of having to learn it. The stranger the new thing is (like Lisp), the harder the victim estimates the work to learn it, and the more important it becomes to declare it bad or useless.
So, how should we fix Lisp in order to make it more attractive to performance-oriented people? I personally don't think we should try to alter one single thing of Lisp in order to attract these people. Why should all that counts in a programming language be popularity? I often see the very revealing parallel drawn between Lisp and a fine violin. Should we alter violins in order to attract people who are used to playing the accordion, and who are not willing to learn the violin because it is too hard and too different from the accordion? Of course not! The violin has its place, and is a great instrument when played by someone who really masters it. Lisp is a great programming language when used by someone who knows how to use it. Dumbing it down to attract mediocre performance-oriented programmers would be a grave mistake. In some (in particular French) newsgroups, I have been called "elitist" for suggesting such things, but I think the reason for that is that programming is still perceived as an activity that anyone without any training (the less training the better, it seems) should be able to do. We even glorify successful programmers who happen to be high-school drop-outs. Why people think this is normal for software development but not for (say) heart surgery or playing the violin is beyond me.
Some people think that there is something wrong with Lisp, because it is nearly impossible to make software work unmodified on all platforms (combinations of Lisp systems and operating systems), whereas with something like Python or Ruby, this is easy. Clearly, there must be something wrong with Lisp here, right? Not really, no. Lisp is not a single implementation, but an ANSI standard (this is Common Lisp, now). Like many other standards (like the ANSI standard for the C programming language for instance), it does not contain everything you might possibly want when you write applications, such as sockets, XML parsers, SQL libraries, etc. These are instead supplied by separate libraries that, in the case of Lisp, are either supplied by the vendor of the commercial Lisp system or by some free software for the others. The perceived difficulty, then, is to write an application that can work with all these, mutually incompatible, libraries. A developer faced with this problem usually gets very frustrated, complains in comp.lang.lisp that we must fix the Lisp ANSI standard as soon as possible in order to include these libraries. Hold on a second though! Why is this not a problem for languages like Python and Ruby that do not even HAVE an ANSI standard? Why do people not complain loudly that they can't even write a cross-platform loop or assignment statement in Python, because the way to do it is not in the standard? Answer: because these people are comparing apples and oranges, or in this case, a language defined mostly by a single implementation and a standard with multiple implementations. Here is how to see how this comparison is absurd: If I were to write a new implementation of Python that does not have sockets in it, does that suddenly make Python worse off than before? Of course not! Similarly, people who are willing to use a single-implementation-language, and thereby running the risk that the language might change overnight, possibly making large parts of some major investment obsolete, should accept the same thing with Lisp, and pick a single implementation that works on all operating systems. They would be no worse off than with any single-implementation language.
Some people WITHIN the Lisp community think that Lisp is not as popular as it deserves to be because there are flaws in the Common Lisp language. The perceived solution to the problem is thus to create a better Lisp dialect. Frankly, if this were true, then no single other language would have a single follower, given the number of flaws in other languages compared to the flaws of Common Lisp. One typical attempt to create a better Lisp was Dylan (more like Scheme than like Lisp, really) which defines a parenthesis-free syntax for a Lisp-like language, thus essentially depriving Lisp of one of its perhaps greatest advantages, namely the nearly one-to-one mapping between surface syntax and internal representation of the code, an essential feature for making macros possible. And Dylan is no more popular than Common Lisp.
Recently, we regularly see people (usually also within the Lisp community) who have the perfect explanation as to why Lisp is not as popular as it deserves to be, namely that there is not even a single free implementation that works on all operating systems, and that has all the required libraries that languages like Python and Ruby have for web programming, etc (we already saw that the Common Lisp ANSI standard does not). The typical article from one of these people is very condescending. I recently saw phrases similar to "I'm sorry, guys, but command-line-interfaces just don't cut it anymore", "everything is GUI-based these days, and you don't even have a standard GUI library". The general tone seems to accuse some fictive Lisp community of not realizing what it is that Lisp needs in terms of Libraries in order to become more popular. I agree that they are right that there is currently no such implementation. I seriously doubt that fixing this problem would in any significant way influence the popularity of Lisp, though.
Worse, I don't understand to whom these articles are directed. Some of them are no doubt only meant for the author to abandon Lisp for a more popular language while doing so with a clean conscience ("it is not my fault, I HAD to do it because I couldn't get the libraries I needed"). All of them certainly have the effect (whether desired or not) on people who might consider using Lisp to discourage them from trying. For people who don't know Lisp and do not consider using it, they certainly have absolutely no effect whatsoever. Most of the authors of these articles seriously seem to think that somehow they will be taken seriously by the members of the "Lisp community", that these members will realize their own mistakes, and start providing high-quality libraries for web programming and XML parsing for free right away. I don't think that will happen. To see why, we can (for this purpose) divide the members of the "Lisp community" into three kinds of people: those who already spend considerable energy and time writing such libraries, those who don't have the ability to write such libraries (for lack of knowledge, energy, or time), and those who do have the ability to do so, but who don't. The people in the first category are not going to alter their behavior as a result of such an article, except that they might be sad to see that their work is not recognized, and perhaps give up. The people in the second category are not going to suddenly start writing the required libraries. We are thus left with the third category, which typically includes the authors of these articles, mutually accusing one another of not providing the tools that they need. Personally, I think it would be a better use of their time and energy to start writing some of these libraries (as a good examples to the others) than to lament the fact that these libraries do not exist.
I mostly don't care how popular Lisp is. I am not using Lisp in order to score better in a popularity contest. I am using Lisp because it is the best programming language I know for the kind of programs I write. I don't think there is anything particularly or seriously wrong with Lisp. It might be that at the moment it does not provide what some people have come to expect from a programming language (free cross-platform implementation with all the libraries you could possibly want). Whether this will ever happen, I don't know, and I don't care (though I respect that others might). I do think I know that some of the people who want this to happen will have to get their hands dirty and just do it. No amount of lamenting is going to magically create any libraries.