PERFORCE/CLEARCASE benchmark 
January 4, 1996

SCENARIO

These comparisons were done on a single unloaded Sparc 10 with 32MB of
memory on a single 1GB disk partition.  Some numbers would be different
if the client and server were on separate hosts.  Both the ClearCase
VOB and PERFORCE depot were created new for the test.

The test consisted of adding a set of files into the system, checking
out and in a set of files, reading all the files using "cat" twice
(once to get the initial load and the second time to get the warmed-up
speed), branching a set of files, and labelling a set of files.

Speed is computed both for ClearCase and for PERFORCE and both in terms of
files/second and KB/sec, since some operations are sensitive to the
sizes of the files.  The final section has the performance ratio of PERFORCE
to ClearCase in terms of both KB/sec and files/sec.  The ratio for
KB/sec is lower than file/sec for checkout/checkin, which is probably
due to the fact that the files used in the PERFORCE sample included a lot
more small files.  Since the checkout/checkin speed for small files is
dominated by per-file costs, handling many small files drives down the
KB/sec rate.

Ideally, we would have used the exact same number of files for both PERFORCE
and ClearCase, but that would have meant either unusably low times for
PERFORCE or very long waits for ClearCase.  Labelling all 750 files took
somewhere around a second with PERFORCE which doesn't give very many significant
digits.  On the other hand, we we had to interrupt the initial load
into ClearCase in order to get the tests done in the limited time we had.

Some activities don't match exactly between ClearCase and PERFORCE most
notably "cat(1)-ing" a bunch of files for the first time.  The purpose
of this test is to see how fast the content of files can be produced.
This is similar to the performance for compiling or searching a tree of
source files.  With the ClearCase MVFS you just invoke cat: on the
first call the MVFS populates its cache and produces the files; on the
second call it produces the files from the cache.  With PERFORCE you must
first populate the client tree with "get."  Then both first and
subsequent calls to cat are just against the local file tree.  Thus
for PERFORCE the time for "Cat" and "Cat, second time" are the simply the
same figures relabelled.

The most suprising figure is that local file access is 8 times faster
than accessing cached files through the MVFS.  This doesn't jibe with
Atria's claim that MVFS performance is 80% of native.  We must assume
the 80% number only applies when the client and server are on separate
machines, when the file transfer is limited by network performance.

We welcome attempts to reproduce or refute these figures.  We wouldn't
have believed them either if we hadn't seen them with our own eyes.

CLEARCASE TIMING

    Activity               # files   size KB      time    KB/sec files/sec
    ----------------------------------------------------------------------

    Add new files              547      2300      5580      0.41      0.10

    Checkout                    35       420        62      6.77      0.56
    Checkin                     35       420        56      7.50      0.62
    Checkout+Checkin            35       420       118      3.56      0.30

    Cat, first time            547      2300       281      8.19      1.95
    Cat, second time           547      2300        35     65.71     15.63

    Label                      547      2300       103     22.33      5.31

    Branch                      35       420        40     10.50      0.88


PERFORCE TIMING

    Activity               # files   size KB      time    KB/sec files/sec
    ----------------------------------------------------------------------

    PERFORCE add               750      3900        42     92.86     17.86
    PERFORCE submit of add     750      3900       118     33.05      6.36
    Add new files              750      3900       160     24.38      4.69

    PERFORCE edit              299      1300         4    325.00     74.75
    PERFORCE submit of edit    299      1300        94     13.83      3.18
    Checkout+Checkin           299      1300        98     13.27      3.05

    PERFORCE get               750      3900        47     82.98     15.96
    Cat                        750      3900         6    650.00    125.00

    Cat, first time            750      3900        53     73.58     14.15
    Cat, second time           750      3900         6    650.00    125.00

    Label                      750      3900         1   3900.00    750.00

    PERFORCE integ branch      750      3900        10    390.00     75.00
    PERFORCE submit branch     750      3900        27    144.44     27.78
    Branch                     750      3900        37    105.41     20.27


PERFORCE/CLEARCASE RATIO      

    Activity                      by KB  by files
    ---------------------------------------------

    Add new files                  59 x      48 x       < How many times faster
    Checkout+Checkin                4 x      10 x       < PERFORCE is than 
    Cat, first time                 9 x       7 x       < ClearCase at various
    Cat, second time               10 x       8 x       < SCM activities.
    Label                         175 x     141 x       < These numbers
    Branch                         10 x      23 x       < are multipliers!