Memory Management in the MS-DOS Environment

The average MS-DOS box has three kinds of memory.  One is RAM, which means
the kind that you try to RAM your programs into (because that's the only way
you can get them to fit).  Another is ROM, which is the kind that you don't
RAM the programs into.  And the third is the kind that sits in your clock
chip that viruses hide in, allowing them to survive reboots, power cycling,
full system reinstalls and comet-impacts and still erase your hard disk.

The RAM memory itself is divided into three types: conventional memory,
extended memory, expanded memory and the other kind that sits between 640k
and 1 meg, and doesn't exist (which is why there are only three different
kinds).  Except sometimes you can use the kind that doesn't exist.  But
mostly it just sits there, laughing, saying "So ya thought that just using
the x=C000:C400 argument would let ya use yer video card AND run yer program
at the same time, did ya?  Whatta nerd!"

The reason this memory is there is that 640 is not a power of 2.  When the
RAM chip makers wanted to make a larger RAM they skipped directly from 256
to 1024, failing in the process to make a 640k-by-1-bit chip. Bill Gates had
tried to prevent this from happening by lobbying the Reagan Administration
to officially declare 640 a power of 2, but they were going through the
Iran-Contra problem at the time and didn't want to do anything to make
people think they were tampering with the separation of powers.

The thing to realize is that all DOS applications require 640k of memory.
Actually any program, if given the opportunity, will use an infinite amount
of memory.  Every computer system must adopt some strategy to prevent the
programs it runs from doing this (though admittedly one component of almost
every strategy involves not purchasing infinite amounts of memory).

The approach used by the UNIX operating system was once considered very
promising but is beginning to show its age.  The idea is that if a program
starts getting too big, you "swap" it out to disk.  Most programs think that
if they are on disk, they've been terminated, are not doing anything, and
don't need to get more memory.  Usually they'll just sit there quietly.
Besides, most programs originally found the experience of being swapped out
to disk and rotated at 3600 rpm in mid-execution quite bewildering. However,
for the user to get any work done, at least a tiny bit of the program has to
be in physical memory (this is another term for RAM, since RAMming things is
pretty physical).  With time, programs began to realize that they had NOT
actually terminated when they were swapped out to disk, and some even began
to enjoy the experience (one, which started out as a set of "editing
macros," now comes close to overwhelming the resources of most UNIX systems,
and continues to grow at an exponential rate).

The DOS approach, on the other hand, took more of a brute force tack.  The
DOS designers simply declared that no program would ever need more than
640k.  So there.  Since this was an order of magnitude more memory than the
amount they had previously declared nobody would ever need more than, people
believed them.

They also made one major design decision that proved very successful: they
made it almost impossible to USE more than 640k.  Note that users were able
to INSTALL more memory; they were just unable to USE it. Programs that tried
to expand beyond 640k simply found that there wasn't any there (actually
they found things like ROM and video memory, which wasn't much better).

Everything would have been, well, perhaps not 'fine' but at least maybe
somewhat short of hellacious, if it had not been for the TSR.  To explain
how this all came about, it is necessary to delve back into the prehistoric
days when IBM first realized that there were such things as small computers.

It seems that one day an IBM engineer read that Intel Corp. was selling 16-
bit computers for $5. Except they weren't really computers. And they weren't
really 16-bit.  But they were $5, and one out of three wasn't bad.  So the
IBM engineer ordered about 35,000 of them out of petty cash, just to play
with.  Then he got hit by a truck.

When this shipment of 35,000 8088 chips showed up on the doorstep, nobody
knew quite what to do with them.  Finally they bought a data sheet for $.75
and put something together to see if the chips were any good.  They even put
in a BASIC interpreter so that they could play games on it.  Unfortunately
it turned out to run the games slower than a Commodore 64, so they got bored
with that pretty quickly and went back to their 360s.

But then one cold March afternoon after a beer-and-pizza lunch, the ratio of
which was skewed perhaps a bit too much towards the former, a group of
engineers decided to make up a mock proposal to sell the things they had
cobbled together.  They submitted it at the beginning of April and sat back
to await the laughter.  Unfortunately the mail-room took 2 weeks to deliver
it and they'd forgotten to put a date on it.  You can only imagine their
horror when the proposal was accepted by upper management, and they were
told to have something ready by the end of the year.

In those days it was considered acceptable to ship computers with only BASIC
interpreters installed.  However, once upper management got hold of the
benchmarks showing that the PC ran BASIC slower than the Commodore 64, they
demanded that the engineers make it do something FASTER than a Commodore 64.
Fortunately it was well known that the Commodore 64 had the slowest disk
interface in the entire computer industry (slower, in fact, than some
cassette tape drives), so their job was relatively easy.  Once they had
added a disk interface, however, they were forced to get a disk operating
system for it.  (They considered just taking the Commodore 64 approach of
putting the DOS in the BASIC interpreter, but there were already too many
comparisons being drawn between the C64 and the PC, and they wanted to try
to distance their product from the Commodore line).

They decided to throw money at the problem, and wandered off to Digital
Research to get them to port CP/M to the PC. However, the head of Digital
Research was late for the appointment (rumors that he was out flying his
private plane are false; he just hadn't put on a tie for so long that he had
forgotten how to tie one, and he spent the whole afternoon trying to
remember how to do it right).

This left a brief window of opportunity for Bill Gates.  Unfortunately, the
only program Bill Gates had ever written was a BASIC interpreter (and there
are some doubts about that).  So he ran out and bought an operating system
that ran on the 8088 from a guy he met on the streets of Seattle (it's
amazing what you can find if you run down the street waving hundred dollar
bills in the face of everyone you see).

In no time flat, the PC was running PC-DOS.  Careful benchmarking determined
that the PC ran the new DOS faster than the C64 (which didn't) and so PC-DOS
became a reality.

One might expect, however, that a DOS bought from some guy you ran in to on
the streets of Seattle might be lacking a few features.  This was, in fact,
the case.  One problem was especially prevalent.  It turned out that most
people familiar with Microsoft products expected the 'print' command to do
something, perhaps actually even print.  In early versions of PC-DOS what it
did was to print an error message, and many found this unacceptable.  So the
problem was given to a new-hire (Microsoft was just beginning to reap some
of the benefits of its association with IBM, and was expanding quickly).

Unfortunately, Microsoft had decided that since they were now in the
operating-system business they should hire people who knew about operating
systems.  The new-hire, who was just out of college, needed a job, and
Microsoft didn't know any better, so they hired him.  Thus from simple
beginnings do major disasters come.

This programmer, who had never programed in BASIC (!) decided that he would
make the print command do something useful, like print files.  To the
printer.  In the background.  While other programs were running.  After all,
how hard could it be to wait for the printer port to tell you it's ready and
shove a character out?  Just a few bytes of assembly language....

Thus people began typing the 'print' command and seeing the cryptic,
tantalizing message, "Resident part of PRINT installed."  Then the DOS
prompt would come back.  But the file would keep on printing!

This took some time for people to get used to.  A number of users actually
had exorcisms performed on their computers until they found out that the
behavior was normal.  Eventually, though, a few clever people discovered
that it was possible to make other programs stay 'resident' (that is, in
memory) and active, even though they had officially 'terminated.'  The TSR
(terminate and stay resident) program was born.

At the time, DOS programs were still taking between 100-200k of RAM.  The
TSR programs typically took about 50-60k of RAM, so a computer that could
handle 640k of memory seemed to be sufficient.  But one could already see
the handwriting on the wall.  It turned out to be possible to run many TSRs
at the same time, as long as you installed each one last (that is, after all
other TSRs you tried to run).  At the same time, all programs were growing
inexorably.  Finally the day came when normal programs reached the 640k
limit, and at the same time TSRs became ubiquitous and indispensable.

One might expect that this would simply cause the industry to abandon the
PC-DOS architecture.  A number of the original IBM engineers, all of whom
had carefully guarded their anonymity, were hoping this would happen.
Unfortunately they did not reckon with the cleverness of their user base.

It turned out that there was a technique known as 'bank-selection'.  This
technique had been developed when people realized that users would, in fact,
need more than 64k of RAM. It was a technique for allowing a computer with a
limited address space to access more memory by switching multiple banks of
memory in and out of a fixed place. This approach was applied to the PC-DOS
architecture and that misbegotten spawn, expanded memory, saw the light of
day.

Of course it was quickly acknowledged that for expanded memory to be useful,
hardware and software developers would have to agree on a standard method
for implementing it.  Recognition of this fact was immediately followed by
the development of several incompatible standards and the sale of numerous
products implementing the various standards.  After about five years all
these standards except one had perished, along with the companies that
created them.  The single remaining standard was known as the Lotus-Intel-
Microsoft (LIM) standard.

The IBM engineers were initially dismayed, but soon they realized that even
after the standard was agreed upon, it was too clunky and slow to really
use.  Besides, if a program depended on it, then computers that didn't have
it would not be able to use the program, thus decreasing the potential sales
of the program.  At the same time, other computers were beginning to emerge
that could use much more memory than the PC, and some even offered features
such as window interfaces and multitasking.

However, Intel Corp. had suffered much the same fate as Microsoft: it had
become rich and successful.  It engaged in a running battle with Motorola,
which had developed a clean, fast architecture in the early 80s.  Each time
Intel would come out with a new chip, Motorola would soon follow with a
faster chip that had more features and was easier to program.

Unfortunately the Motorola architecture was not upward compatible with the
8080. Nor had it been the victim of a misfired practical joke. Worst of all,
it was not DOS-compatible (since it could access more than 640k of memory).
As a result, it could not achieve the market penetration of the Intel
architecture.

Intel had developed a number of chips that allowed PCs to address memory
addressed above 1 meg.  This memory is called extended memory.  However, due
to various factors, while they could ADDRESS this memory, they could not USE
it while running DOS.  Intel was forced to ensure that all of its processors
were backwards-compatible with the original 8088, and this required that
they run in something called 'real' mode, as in, "You want to access more
than 640k?  Come on, get real!"

Nevertheless, the resourcefulness of long-suffering DOS users again came to
the fore.  After a few false starts (the '286, for example), Intel developed
a chip that could cleanly switch between 'real' mode and 'protected' mode.
Protected mode allowed the processor to access extended memory.  Eventually,
using this ability to switch, it turned out to be possible to use extended
memory to emulate expanded memory.  So if someone wanted to use a program
that could use expanded memory, because real memory wasn't sufficient, then
he could install an expanded-memory emulator that made the computer think
its fast, easily accessible extended memory addressed above 1 meg was
actually slow, clunky expanded memory addressed above 640k but below 1 meg.
(Really, I'm not making this part up!)

A few hackers, with their usual distaste for elegant, well-thought-out
solutions, insisted on writing programs that would simply put the system
into protected mode and proceed from there.  These programs had a hard time
gaining user acceptance because they required that the system be booted with
a special disk that did not use any memory management utilities.

Another feature IBM added, as almost every computer now had at least 1 meg
of memory, was the ability to access the RAM that lay in the address space
just above 1 meg.  This memory was called the 'high memory area' because it
was the lowest part of extended memory.  Ordinarily, as you remember,
extended memory can only, at best, be used as expanded memory if you've got
the right utility.  Otherwise it normally goes unused.  However, since the
RAM was there, someone decided it would be nice to use it for something.  So
they figured out a way to switch it in to the address space.  To do this,
they used the keyboard processor.  This was the last gasp of the original
engineering team.  They hoped that with this the PC community would finally
see the joke, laugh, and all go out and buy Amigas.  Unfortunately all the
'clone' makers immediately rushed off and made their systems capable of
addressing this 'high memory' as well.  To make sure their systems were 'IBM
compatible' most of them also used the keyboard chip.

This brings us to the current state of things.  A tad untidy perhaps.  Some
people never do figure out the difference between expanded and extended
memory (several AI systems are on the market to aid in this task).  But
Microsoft, in the spirit of its can-do attitude, has come to our rescue.  It
has produced a set of utilities that allow the user to manage the various
types of memory, INCLUDING THE NON-EXISTENT KIND, no matter what keyboard
chip your computer uses!  These utilities are called HIMEM and EMM386.
Information about these utilities, each containing a small BASIC interpreter
to parse its command-line options, is helpfully sprinkled throughout the
documentation Microsoft supplies with its products.  A number of
universities also offer graduate-level courses on their use.

With the use of these utilities, you can finally use every kind of memory on
your system.  As long as your program doesn't need more than 640k.