Common Lisp - Myths and Legends

Lisp has been supporting the world's most complex applications since 1958. And Lisp has grown a lot since then. So if you or someone you know harbors fears or concerns about Lisp because of something learned in a class or muttered by a friend 20 or 30 years ago, it's time to take a fresh look. In this paper we're going to survey what Lisp is today.

Partially, Lisp is the stuff of Legends. We hear again and again from our users that writing their system in Lisp cut their investment in developer years by a factor of five or ten. "We couldn't have done this in any other language," they say. Once bitten by Lisp, they don't want to let go. In this paper we'll take a look at some of the reasons for this enthusiasm, at why lisp is the premier language for advanced application development.

There are a lot of myths out there which need to be dispelled, and we've decided to take some of these head-on in this paper. If we leave you with any questions, please feel welcome to ask a LispWorks representative to clarify them. We're not afraid of hard questions; confronting hard questions is our speciality. And it also happens to be what has kept Lisp alive and strong through four decades!

Who's Using Lisp?

You don't always see it, but Lisp is all around you.

The Web is growing as fast for Lisp as for the rest of the industry. Yahoo! Store includes a WYSIWYG editor for editing your online store through a standard web browser. The editor is written in Common Lisp. Public domain Lisp-based web servers include CL-HTTP, architected by John Mallery at the MIT Artificial Intelligence Laboratory.

Some companies consider Lisp technology so valuable that they tend to keep their use of it under wraps; this is often a source of frustration to those wanting to crow about Lisp's successes. But in 1994, in an unusual move, AT&T permitted Harlequin to confirm that they had ``been supplying Lisp consulting services and products to AT&T, in support of AT&T's development of switched virtual circuit capabilities''. Toward this end, Harlequin created a special variant of its LispWorks^® system which offers the realtime response necessary to meet AT&T's rigorous needs, even in spite of being a garbage collected system!

Do you use a credit card? It's quite possible there's Lisp technology used when you present your card for authorization. Fraud detection is another area where Lisp has been used with great success.

Xanalys Ltd uses Common Lisp within application products. Its Xanalys LinkEXPLORER is a LispWorks application.

In addition to the above specific examples involving internet web services, telecommunications, document translation, credit-card fraud detection, and criminal investigation, Lisp is also used in applications involving financial and investment analysis and tracking stock market trends, airline scheduling, space exploration, process planning and scheduling, robotics, chemistry, medicine, physics.

That's a broad range of uses, but it has an important commonality: each of these areas confronts seriously hard problems. So from this we can evolve a checklist of possible reasons for using Lisp.

Should I be using Lisp?

We won't claim that everyone should be using Lisp, but there are some particular situations in which we think you're really hurting yourself if you're not. If you answer yes to any of the following questions, Lisp is definitely for you:

Is the problem very complex or ill-specified?

Experience shows that Lisp excels at dealing with partially specified problems and problems whose essential nature is not fully known at the outset. A programmer can write an approximation to a process and then use the programming to refine his or her understanding of the process interactively.

Does the problem involve multiple representations of data from varying sources?

Because of Lisp's flexible class system and dynamic typing, configuring robust systems is easy. If new data types are added, classes and methods already compiled usually do not have to be recompiled in order to interact properly. Lisp is ideal for large systems that evolve over time.

Do the conditions of the problem change frequently, even dynamically?

Lisp has become highly optimized over the years for `incremental change'. For example:

New function and class definitions can be loaded after Lisp starts, even multiple times to accommodate runtime redefinition of behavior. The default behavior of class redefinition is to update existing data to accommodate the new definition automatically in a fairly general way that usually involves no programming at all, but more refined programmer control of this process is available for cases where it is needed.
Methods on new datatypes can be added to existing generic functions without access to sources for the methods on existing datatypes and without the need to recompile existing methods!
Lisp's automatic storage mechanism, the ``garbage collector'', is constantly on guard to reclaim old storage in the face of redefinition.

These are only some of the many features Lisp has to support incrementality in ways that most other languages do not.

Does the program have to be delivered quickly?

Many features of Lisp support rapid prototyping and application delivery. Lisp provides a comprehensive set of pre-defined libraries. The presence of the garbage collector saves time in writing storage reclamation code, permits more flexible programming styles, and avoids wasted programmer time tracking down obscure memory leaks. Lisp makes it easy to write modular code and to test modules interactively. For more information about the features of Lisp that support rapid prototyping, see: http://www.nhplace.com/kent/PS/Hindsight.html

Will the program have to be frequently modified and updated?

The last thing one wants to have to do is to constantly be recompiling every module of change. As we have discussed already, Lisp is optimized for incrementality and redefinition. Common operations like adding or deleting a method, or adding and deleting a slot don't normally require recompilation of a whole system.

Will the program have to deal with faulty or erroneous data?

The last thing you want is to have programs that think error handling means popping up a bomb box or dumping core. Lisp's error handling capabilities are powerful and give you a lot more options than that. It's easy to control the disposition of a specific kind of error dynamically. Also, the error system of Lisp is active, not passive, so you don't have to be constantly checking functions to see if they returned "error codes" rather than correct results. You can write your programs to assume a correct result will come back because you can arrange for an active transfer of control to happen in the case where an error occurs. This makes programming more modular and also avoids compounding an error by failing to test for it!

The problems we've enumerated above are ones that usually cause other programming languages to fall flat. But they are the very things that Lisp was designed, through its long association with the Artificial Intelligence community, to handle best.

So what doesn't Lisp do? Well, we don't get a lot of call for it to do business accounting systems. Bulk processing of well-understood, very homogenous data is something Lisp could do but is rarely called upon to do because commodity languages usually work fine. But it's almost too bad Lisp hasn't been used for more business systems because among Lisp's many features is a built-in understanding of dates and times which is not susceptible to the Year 2000 bug. And Lisp's dynamically modifiable class and object system would provide excellent support for the changing needs of growing companies.

Oh, and we don't tend to do terribly well on dumping out tiny ``hello world'' executables. So if your only need is a program that does very little and is correspondingly tiny, we're probably not the language for you. But in our experience, one's needs tend to grow with time and we find that ``hello world'' is not typical in size or functionality to real commercial programs. So we've focused our energies on optimizing real programs.

What Does Lisp Offer Me?

In one sentence: Lisp offers you a stunning level of productivity which frees you to write complex, robust, understandable and evolving applications with an order-of-magnitude reduction in the associated pain.

Lisp systems are particularly good at supporting rapid prototyping - you develop your application the same way as you think about it: from the top down. Modules can be constructed in isolation, partially written programs can be tested and debugged interactively. Interpreted code under development can be freely mixed with highly optimized compiled code. Lisp IDEs generally come with a built-in lisp editor - the code you write becomes a fully integrated part of your system as you write it, and you never have to leave your lisp session to rebuild and start over. Your development style ends up reflecting the dynamic powers of the language itself: you can redefine your functions and classes, as often as you want, both in development and in the running application. You don't have to worry about implementing a whole load of primitives just to get yourself going, because the language provides them for you.

Then when the job's done you can deliver either free-running executables or relocatable libraries (DLLs on Windows). Tree-shaking and pruning tools are typically provided to help you cut you application down to size.

ANSI Common Lisp

Lisp is actually a whole family of languages, but in this document we're mostly talking about ANSI Common Lisp, the Lisp dialect of choice for industrial-strength applications development worldwide.

ANSI Common Lisp offers you the following features:

Compiler configurability
Using simple constructs within Lisp itself you get to select where your code will run fast, where it will run slower but "safe" (automatically performing checks on the data as it runs), and where the compilation itself will be rapid. You can specify this by the system, per file, or even on a line-by-line basis when you need to. Further constructs allow you to state which functions should be compiled inline.
Macros
A powerful and elegant macro facility allows lisp code to generate more lisp code. The macros themselves are also written in lisp and can perform arbitrary operations as part of the expansion process. Calls to the macro-expander have the same syntax as function calls - there is no sense of leaving lisp or learning a second language.
Common Lisp Object System
Otherwise known as CLOS, the most advanced object system in the world. It supports:
- dynamic change of an object's class,
- dynamic method recombination for runtime-added methods without access to source code or any need to "rebuild",
- dispatch on classes or object identity,
- dispatch on multiple objects, not just one object,
- user-defined forms of method combination,
- an unofficial but widely accepted "Meta Object Protocol" which allows introspection and even dynamic redefinition of the object system itself.
Built-in types
Lisp provides a rich set of built-in data-types, including
- Symbols (symbolic identifiers)
- Numerical types:
  - Machine integers and arbitrary precision integers
  - IEEE floats (single and double)
  - Rational and complex numbers
- Characters
- Containers:
  - Arrays (simple or indirect, with varying element types)
  - Lists (containing mixed and arbitrary kinds of typed objects)
  - Strings
  - Various kinds of Hash Tables
- Pathnames: portable file naming which accesses the native file system while hiding system characteristics for cross-platform situations
- Functions (including high-order functions) as first class objects
Standardized library
The runtime library for Common Lisp is part of the standard. That makes it fully platform and vendor independent. The library is large and supplies several hundred highly flexible functions. These cover sequence and string utilities, sorting, hash table creation/lookup, flexible file I/O, numerical operations (extremely efficient with declarations), and much much more.
Powerful error-handling facility
No application is perfect, and nor is the real world with which applications interact. Errors do occur, and what Common Lisp does is to provide a rich facility for surviving such "conditions": signalling, handling, restarting - all under program control. As an aid to debugging that elusive error, Common Lisp allows you to trace and even to disassemble your code.

Other features

We listed above the main features of ANSI Common Lisp. Other features of note, but at this time not included within the standard, include:

Introspection tools
Lisp systems provide considerable assistance to writing and debugging an application. The following tools are typical, and tend to be fully integrated: state and values can be exchanged between them with a minimum of fuss.
- Lisp listeners allow interaction with the underlying lisp system - lisp forms (think of them as small programs) are evaluated and the results returned.
- Inspectors allow introspection of Lisp objects - you can present an inspector with an arbitrary object and it will show you the values in all its slots. You don't need to tell the inspector what type the object is - Lisp can figure that out for itself.
- Class tools allow you to browse the accessors associated with any object. In conjunction with inspectors they allow for extremely rapid learning about existing systems.
- Debuggers give the application writer access to the stack in the event of an unhandled error. Arguments to function calls can be inspected, handed to the listener which can perform arbitrary operations on them, or even changed and the function called again with the new values.
Common Lisp Interface Manager
Standardized (but not within ANSI-CL), CLIM gives application users one of the most highly sought-after features around: a combination of both vendor and platform independence in their user interface. It offers:
- optional specification of native look-and-feel, supporting most standard GUI widgets.
- a straight-forward mapping of application semantic components (actions and objects) to GUI components (commands and text/graphics)
- powerful graphics facilities (shapes, fonts, colors, bitmaps)
- incremental redisplay manager
- table formatting, including dynamic layout based on size of cell entries, arbitrary text and/or graphics for cell entries, nesting of tables, etc
- commands trivially made available as keystrokes, pull-down menu entries, command-line style (with completion), and/or associated with pointer gestures on display objects
- automatic prompting for and type-checked parsing of command arguments
- horizontal or vertical formatting of n-ary, cyclic graphs with text or graphics for node labels
- high-level support for common pointer-based operations (like drag-and-drop)
and much more.
Multitasking
Light-weight control of multiple execution strands within a single machine process. On Windows this would typically be implemented with native threads.
System definition
To keep track of build dependencies when it you're managing a large system.

Myths About Lisp

In this section, we'll look at some of the amazing myths that we still hear told about Lisp, and contrast them with the actual reality.

Myth #1: Lisp is slow

The reality is that when all datatypes are appropriately declared, a typical commercial Lisp compiler produces native machine code that is comparable in speed with other languages.

However, something Lisp does which other languages don't is to allow you to run code with no type declarations. In this mode, types are dynamically determined and appropriate behavior occurs based on runtime dispatch. This is slower than heavily declared compiled code would be, but it's faster to write and therefore a big boon to debugging. After all, who wants to write type declarations for a bunch of tentative code they aren't sure they're even going to use? It's not appropriate to compare the speed of such non-type-declared code to the speed of other languages since in other languages, the absence of declarations is fatal to programs in most other languages.

If you want to make a proper comparison, you have to either compare properly declared code in Lisp to properly declared code in other languages. If you do that, Lisp will compare favorably. Otherwise, to be fair, you should compare the speed of undeclared code in Lisp (modest) to the speed of undeclared code in other languages (often zero, since such code in other languages is usually ``incomplete'' and will not run). Once such a proper comparison is made, we again see the Lisp's behavior is quite favorable.

Myth #2: Lisp is big

Big is a moving target. Sometime in the early 1980's, people started complaining about the size of Lisp as an impediment. Lisp was big at that time, compared to other applications of the time because it packed a lot of useful functionality and there was a limit to how small that functionality could be made. But Lisp vendors became very sensitive to the size issue and Lisp has been one of the few programming languages in recent years that has not been allowed to grow by leaps and bounds with every release.

At this point, the size of a typical application in Lisp and its runtime libraries are comparable in size to what a similar application would be written in another language. But if the ``bloating'' trend of other languages increases as it has been, Lisp will soon be seen as the much more compact alternative!

Myth #3: Lisp has no arrays

Lisp has had an array datatype for at least 30 years, but it's common for people who took a ``comparative programming languages'' course not to know this. This is because other languages often have not had Lisp's well-known LIST datatype, and so Lisp is used as a showcase for dealing with linked lists and recursion. Sometimes out of ignorance and sometimes just for sheer lack of time, the discussion of array types in Lisp often receives no attention. But that doesn't mean Lisp doesn't offer powerful support for single and multi-dimensional arrays, arrays of varying element type, and array access that provides transparent access to fixed size arrays, arrays displaced to other arrays, and arrays whose size is expected to dynamically grow and shrink.

Myth #4: Lisp has no compiler

Since its earliest days, now 40 years ago, Lisp implementations have been variously interpreted or compiled, and often both. In fact, much important work in the theory of program compilation has been done using Lisp, and Lisp compilers have benefited enormously by this. No modern commercial Lisp is without a compiler. The fact that modern Lisps often come with an interpreter as well is simply a convenience for some implementations to encourage late-binding semantics and promote program flexibility, including interactive debugging.

The ANSI Common Lisp standard does not require the presence of a compiler provided that an interpreter achieves the defined semantics. It is intended that the marketplace will sort out this issue. The ability to omit a compiler allows certain price/performance points to be achieved, especially among subset and freeware implementations. However, serious commercial implementations invariably offer optimizing compilers as a standard part of their product. Ask to be sure, but don't let ever anyone tell you there's no such thing!

Myth #5: Lisp is not standard

This one is simple to correct. X3.226/1994, the American National Standard for Programming Language Common Lisp, not only exists but in fact was the first ANSI standard for an object-oriented programming language.

A webbed adaptation of the ANSI Common Lisp standard, The Common Lisp HyperSpec^TM (CLHS) is available from LispWorks Ltd free of charge from our downloads page. CLHS contains a comprehensive glossary and myriad programming examples. It is heavily cross-indexed (105K hyperlinks) and is optimized to use low graphics for very fast browsing.

Common Lisp is not only a standard but it places a heavy emphasis on program portability. This allows you to smoothly deploy the same program on quite different platforms. With additional help from CLIM, the Common Lisp Interface Manager, an application can be developed which uses the same code to take on a native Motif look-and-feel under X Windows or a Windows look-and-feel under Windows.

The design of the Common Lisp language planned for a wide variety of potential platform variances (character set, machine word size, filename syntax, interaction style) and will serve its users well into the future without the need to make costly program upgrades required by other languages.

Myth #6: Lisp doesn't talk to other programs

While the ANSI standard for Common Lisp doesn't require it, a serious vendor of Common Lisp such as LispWorks Ltd goes well beyond what the standard requires in the way of potential connections between Lisp and the rest of your computer system. And in future releases, we'll be adding more because we know this is a key issue in the modern, heterogeneous computing environment.

Foreign Interface. Data and program interfaces to C can be declared and called in a way that is transparent to Lisp programs.

Network interfaces. Lisp supports easy access to TCP so that programming TCP-based interfaces is easy. Also, third-party software such as CL-HTTP (mentioned earlier), allows Common Lisp to talk to the web. And, of course, access to other network facilities for which there is no pre-packaged access in Lisp is still readily accessible using the Foreign Interface.

CORBA Interface. A binding of CORBA to Lisp allows natural Lisp programming style to be used when interacting with CORBA interfaces.

Database Interfaces. Flexible accesses to external database through SQL and ODBC interfaces make database entities conveniently available as Lisp data.

DDE Client. A DDE client interface is available for programs requiring Inter-application communication under Windows.

COM and Automation. COM client/server and Automation modules are also available on Windows.

The reality is far from what the myth suggests because, in fact, Common Lisp from LispWorks Ltd provides a flexible and growing set of interconnection options.

Myth #7: Lisp syntax is painful

If you haven't seen it before, the Lisp notation for what other languages might write as ``5*a+3'' is ``(+ (* 5 a) 3)''. And we'll admit this is to some degree a matter of personal taste. However, in spite of the fact that it may look initially a little funny to the unaccustomed, there are some sound technical reasons why Lisp syntax exists and is preferred by most Lisp programmers, and we've tried to enumerate them here:

It's easy to teach. There are no complicated precedence rules waiting to trip you up. Grouping of operators and associativity is manifestly obvious.

It's easy to parse. A few simple rules apply throughout. If you extend the language to add a new operator, you don't have to train the Lisp parser (provided to Lisp programmers as the function called READ) to understand your new operator. You can focus immediately on the semantics, which is what's important anyway.

It maps naturally to an underlying data structure. When teaching someone about macros, it's obvious what the internal representation of a Lisp expression is because it looks just like Lisp program data. This greatly simplifies the writing of macros and ``automatic programming'' facilities. It means that when you extend the language, you don't have to worry about system operators looking different than the ones you define. This breaks down the system/user distinction and keeps the system operators from looking more important than the ones you write yourself.

Text editors can provide better support. A text editor, such as GNU Emacs or the built-in editor that comes with products like Common LispWorks, can automatically provide useful support for indentation based solely on parenthesis level without having to understand the specifics of your program. They can also provide commands for moving forward and backward conveniently over expressions, which in turn allows more flexible use of ``keyboard macros'' to perform automated program manipulation that is much harder to do in infix languages because the beginning and end of an infix expression must always be designated explicitly (usually by mouse motion to establish a ``region'').

Myth #8: Lisp GC is slow

Generational GC. First, LispWorks offers a modern ``generational'' garbage collector. Such collectors are based on a layering theory of data that says that when one wants to get back some storage, one should look ``nearby'' for recently discarded objects. If not enough are found, additional work is done to find slightly older objects. And so on. The idea is that the nearby search is quickest and if that succeeds, very little work will have been done. Objects that survive for a long time in the innermost circle are eventually ``aged'' and become part of a second generation. In practice, this kind of garbage collector can be highly efficient, especially in an environment in which virtual memory is used, because they generally don't have to touch all of memory in order to reclaim enough storage to succeed.

Realtime GC. Customers whose applications can't even tolerate the speed of a generational GC should talk to LispWorks Ltd about its realtime implementation of Lisp. So far, this has not been packaged as a shrink-wrapped product but we might be able to provide it to you under some sort of consulting arrangement. Even if you're not sure you want to get involved in a consulting arrangement, please do contact us about your needs so we can be aware of your interest and help you determine how best to proceed.

Myth #9: Lisp needs special hardware

No.

Lisp's performance can certainly be optimized by special hardware. It's hard to imagine a language that couldn't. Various machines from Digital Equipment Corporation in the 1970's, notably the PDP-10 processor, contained numerous instructions which helped Lisp work efficiently. Later, in the 1980's and into the very early 1990's, there were several companies that built custom ``Lisp chips'' which ran Lisp very efficiently. But any language can benefit from a hardware assist; that doesn't automatically mean it requires special hardware to run that language. In the 1970's, the C language grew up on the PDP11, and yet now it is used widely. Lisp grew up on a wide variety of machines, most of them general purpose.

LispWorks Ltd Lisp products were originally developed on stock hardware. They have never been ported to Lisp-only platforms. Today they run on Unix workstations and on the PC under Linux, Solaris, FreeBSD and MS Windows and on Apple Macintosh hardware running Mac OS X.

Myth #10: Lisp is expensive

This is wrong on several levels.

First, some people believe the expense comes from the need for special purpose hardware. As mentioned already, that's not necessary. LispWorks will run comfortably on a low to mid-range home PC - that's about as general purpose as you can get.

Second, features such as dynamic redefinition, powerful debugging tools and automatic memory management mean Lisp programmers are highly productive.

Third, Lisp is available at a variety of prices. For example, a person who's just getting his or her feet wet with Lisp might want to obtain the LispWorks Personal Edition, which is available for free. Hopefully that's not too expensive! Of course, it comes with some limitations that are not present in the Professional Edition and Enterprise Edition. And even these other editions are quite reasonably priced; see the products or contact a LispWorks Ltd sales representative for details.

Or, if you're not ready to actually order a copy of Lisp, feel free to browse the Common Lisp HyperSpec^TM to get a feel for the language.

Lisp tools from LispWorks Ltd

LispWorks Ltd brings the best of Common Lisp to the Windows, Macintosh, Linux and UNIX platforms. All versions include optimizing native code compilers, advanced garbage collection, and excellent platform integration. LispWorks implements the ANSI Common Lisp Standard and adds numerous libraries for GUI creation, foreign function and enterprise integration, and expert system programming. The LispWorks integrated development environment further simplifies software development.

Once your application is developed, LispWorks supports optimized application delivery. There is no charge for runtime distribution on the Windows, Macintosh, Linux, FreeBSD and x86/x64 Solaris platforms. On UNIX platforms, runtime distribution is available for a modest fee.

LispWorks IDE

The LispWorks IDE provides a smooth and comfortable workflow, allowing you to incrementally write, test, and extend your software while it is running. Features of the LispWorks IDE include:

Interactive Lisp listener, for compiling and executing expressions,
Debugger, tracer and source code stepper,
Object inspector,
Browsers for classes, generic functions, and compilation errors,
Execution time profiler,
Integrated extensible editor and ability to use external editors (e.g. EMACS),
Build system manager,
Incremental compiler and dynamic loader,
Source code location and cross-referencing tool,
Complete on-line documentation in hypertext format.

Language Extensions

Our Common Lisp products include a number of extensions to the language standard, further increasing your productivity. Additional libraries and features include:

CAPI portable GUI toolkit, supporting both Windows and OSF/Motif look and feel,
CLIM 2.0, the Common Lisp Interface Manager,
CORBA interface for creating distributed components,
KnowledgeWorks^® and Prolog for expert system programming,
Integrated database access,
Support for internationalization through Unicode,
Symmetric Multiprocessing,
Programmer-extensible I/O streams,
TCP socket streams,
Object finalization; weak vectors and hash tables.

Platforms

LispWorks Ltd provides Common Lisp implementations on a variety of operating system and hardware platforms. The use of portable language constructs lets you maintain your software on all of these platforms simultaneously.

LispWorks for Windows provides a full native implementation of Common Lisp on all modern x86/x64 Windows computers.

LispWorks for Windows, LispWorks for Macintosh, LispWorks for x86/x64 Solaris, LispWorks for Linux, LispWorks for FreeBSD and LispWorks for AIX are available in several editions. All editions include an advanced Common Lisp compiler, runtime system, and Common LispWorks IDE. LispWorks Ltd does not charge runtime license fees for applications developed with the HobbyistDV, Professional and Enterprise Editions.

The Personal Edition is intended for personal and educational Lisp programming. As a contribution to the Common Lisp community, LispWorks Ltd makes the Personal Edition of LispWorks available free of charge. While the Personal Edition includes the full Common Lisp compiler and development environment, it does limit program size and duration and it does not support application delivery.

The Professional Edition includes everything you need for commercial Common Lisp software development and application delivery. CLIM 2.0 is included to further increase program portability.

The Enterprise Edition (with a 64-bit version now available) includes all the features of the Professional Edition, and provides further support for software needs of the modern enterprise, including:

Database access through object-oriented SQL/ODBC libraries.
Portable distributed computing through CORBA.
Expert system programming through our KnowledgeWorks product and an embedded Prolog compiler.

For UNIX platforms, we offers two products: LispWorks and Liquid Common Lisp (formerly Lucid Common Lisp). Both include the full LispWorks integrated development environment and foreign function interface. Product delivery and specialized libraries are available as separate add-ons. The following platforms are supported:

Compaq Tru64 Unix (OSF 1)
PA-RISC / HP-UX
Sun SPARC / Solaris

Acknowledgements

How Can I Contact LispWorks Ltd?

If you have more questions, or if you want to place an order, please contact Lisp Sales.