PERFORCE/CLEARCASE benchmark January 4, 1996 SCENARIO These comparisons were done on a single unloaded Sparc 10 with 32MB of memory on a single 1GB disk partition. Some numbers would be different if the client and server were on separate hosts. Both the ClearCase VOB and PERFORCE depot were created new for the test. The test consisted of adding a set of files into the system, checking out and in a set of files, reading all the files using "cat" twice (once to get the initial load and the second time to get the warmed-up speed), branching a set of files, and labelling a set of files. Speed is computed both for ClearCase and for PERFORCE and both in terms of files/second and KB/sec, since some operations are sensitive to the sizes of the files. The final section has the performance ratio of PERFORCE to ClearCase in terms of both KB/sec and files/sec. The ratio for KB/sec is lower than file/sec for checkout/checkin, which is probably due to the fact that the files used in the PERFORCE sample included a lot more small files. Since the checkout/checkin speed for small files is dominated by per-file costs, handling many small files drives down the KB/sec rate. Ideally, we would have used the exact same number of files for both PERFORCE and ClearCase, but that would have meant either unusably low times for PERFORCE or very long waits for ClearCase. Labelling all 750 files took somewhere around a second with PERFORCE which doesn't give very many significant digits. On the other hand, we we had to interrupt the initial load into ClearCase in order to get the tests done in the limited time we had. Some activities don't match exactly between ClearCase and PERFORCE most notably "cat(1)-ing" a bunch of files for the first time. The purpose of this test is to see how fast the content of files can be produced. This is similar to the performance for compiling or searching a tree of source files. With the ClearCase MVFS you just invoke cat: on the first call the MVFS populates its cache and produces the files; on the second call it produces the files from the cache. With PERFORCE you must first populate the client tree with "get." Then both first and subsequent calls to cat are just against the local file tree. Thus for PERFORCE the time for "Cat" and "Cat, second time" are the simply the same figures relabelled. The most suprising figure is that local file access is 8 times faster than accessing cached files through the MVFS. This doesn't jibe with Atria's claim that MVFS performance is 80% of native. We must assume the 80% number only applies when the client and server are on separate machines, when the file transfer is limited by network performance. We welcome attempts to reproduce or refute these figures. We wouldn't have believed them either if we hadn't seen them with our own eyes. CLEARCASE TIMING Activity # files size KB time KB/sec files/sec ---------------------------------------------------------------------- Add new files 547 2300 5580 0.41 0.10 Checkout 35 420 62 6.77 0.56 Checkin 35 420 56 7.50 0.62 Checkout+Checkin 35 420 118 3.56 0.30 Cat, first time 547 2300 281 8.19 1.95 Cat, second time 547 2300 35 65.71 15.63 Label 547 2300 103 22.33 5.31 Branch 35 420 40 10.50 0.88 PERFORCE TIMING Activity # files size KB time KB/sec files/sec ---------------------------------------------------------------------- PERFORCE add 750 3900 42 92.86 17.86 PERFORCE submit of add 750 3900 118 33.05 6.36 Add new files 750 3900 160 24.38 4.69 PERFORCE edit 299 1300 4 325.00 74.75 PERFORCE submit of edit 299 1300 94 13.83 3.18 Checkout+Checkin 299 1300 98 13.27 3.05 PERFORCE get 750 3900 47 82.98 15.96 Cat 750 3900 6 650.00 125.00 Cat, first time 750 3900 53 73.58 14.15 Cat, second time 750 3900 6 650.00 125.00 Label 750 3900 1 3900.00 750.00 PERFORCE integ branch 750 3900 10 390.00 75.00 PERFORCE submit branch 750 3900 27 144.44 27.78 Branch 750 3900 37 105.41 20.27 PERFORCE/CLEARCASE RATIO Activity by KB by files --------------------------------------------- Add new files 59 x 48 x < How many times faster Checkout+Checkin 4 x 10 x < PERFORCE is than Cat, first time 9 x 7 x < ClearCase at various Cat, second time 10 x 8 x < SCM activities. Label 175 x 141 x < These numbers Branch 10 x 23 x < are multipliers!