lighty 1.5.0 and linux-aio

lighty 1.5.0 and linux-aio14

Posted by janon Sunday, November 12, 2006

1.5.0will be a big win for all users. It will be more flexible in thehandling and will have huge improvement for static files thanks toasync io.

The following benchmarks shows a increase of 80% for the new linux-aio-sendfile backend compared the classic linux-sendfile one.

The test-env is

• client: Mac Mini 1.2Ghz, MacOS X 10.4.8, 1Gb RAM, 100Mbit
• server: AMD64 3000+, 1Gb RAM, Linux 2.6.16.21-xen, 160Gb RAID1 (soft-raid)

Theserver is running lighttpd 1.4.13 and lighttpd 1.5.0-svn with a cleanconfig [no modules loaded], the client will use http_load.

$ ./http_load -verbose -parallel 100 -fetches 10000 urls

I used this little script to generate 1000 folders, with 100 files each of 100kbyte.

for i in `seq 1 1000`; domkdir -p files-$i;for j in `seq 1 100`; dodd if=/dev/zero of=files-$i/$j bs=100k count=1 2> /dev/null;done;done

That’s 10Gbyte of data, 10 times larger the RAM size of the server as we want to become seek-bound on our disks.

The Limits

2 Seagate Barracuda 160Gb disks (ST3160827AS) are building a RAID1 via the linux-md driver. The 7200 RPMs will give us 480 seeks/s max (7200 RPM = 120 r/s, .5 rotations avg. per seek, 2 disks).

Each disk can send 30Mbyte/s sequential read, combined 60Mbyte.

The Network is 100Mbit/s, we expect it to limit at 10Mbyte/s.

lighttpd 1.4.13, sendfile

A first test run against lighttpd 1.4.13 with linux-sendfile gives use:

$ iostat 5avg-cpu:  %user   %nice %system %iowait  %steal   %idle0.99    0.00    4.77   86.68    0.20    7.36Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtnsda              35.19      3503.78       438.97      17624       2208sdb              33.40      4052.49       438.97      20384       2208md0             119.48      7518.09       429.42      37816       2160avg-cpu:  %user   %nice %system %iowait  %steal   %idle0.60    0.00    4.61   78.36    0.00   16.43Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtnsda              31.46      3408.42       365.53      17008       1824sdb              30.06      3313.83       365.53      16536       1824md0             104.21      6760.72       357.52      33736       1784

./http_load -verbose -parallel 100 -fetches 10000 urls--- 60.006 secs, 1744 fetches started, 1644 completed, 100 current--- 120 secs, 3722 fetches started, 3622 completed, 100 current--- 180 secs, 5966 fetches started, 5866 completed, 100 current--- 240 secs, 8687 fetches started, 8587 completed, 100 current10000 fetches, 100 max parallel, 1.024e+09 bytes, in 274.323 seconds102400 mean bytes/connection36.4534 fetches/sec, 3.73283e+06 bytes/secmsecs/connect: 51.7815 mean, 147.412 max, 0.181 minmsecs/first-response: 360.689 mean, 6178.2 max, 1.08 minHTTP response codes:code 200 -- 10000

lighttpd 1.5.0, sendfile

The same test with lighttpd 1.5.0 using the same network backend: linux-sendfile.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle0.40    0.00    3.60   85.60    0.00   10.40Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtnsda              33.80      4606.40       564.80      23032       2824sdb              37.00      4723.20       564.80      23616       2824md0             136.00      9368.00       553.60      46840       2768avg-cpu:  %user   %nice %system %iowait  %steal   %idle0.80    0.00    4.80   81.80    0.00   12.60Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtnsda              33.40      4198.40       504.00      20992       2520sdb              30.60      4564.80       504.00      22824       2520md0             123.60      8763.20       496.00      43816       2480avg-cpu:  %user   %nice %system %iowait  %steal   %idle0.80    0.00    5.19   81.24    0.00   12.77Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtnsda              36.53      4490.22       493.41      22496       2472sdb              32.34      4784.03       493.41      23968       2472md0             126.75      9274.25       483.83      46464       2424

--- 60 secs, 2444 fetches started, 2344 completed, 100 current--- 120.003 secs, 4957 fetches started, 4857 completed, 100 current--- 180 secs, 7359 fetches started, 7259 completed, 100 current--- 240 secs, 9726 fetches started, 9626 completed, 100 current10000 fetches, 100 max parallel, 1.024e+09 bytes, in 246.803 seconds102400 mean bytes/connection40.5181 fetches/sec, 4.14906e+06 bytes/secmsecs/connect: 55.5808 mean, 186.153 max, 0.24 minmsecs/first-response: 398.639 mean, 6101.44 max, 9.313 minHTTP response codes:code 200 -- 10000

This is minimal better, but has still the same problems. We are maxed out by the disks and not by the network.

lighttpd 1.5.0, linux-aio-sendfile

We only switch the network-backend to the async io one:

server.network-backend = "linux-aio-sendfile"

... and run our benchmark again:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle8.38    0.00   10.18   38.52    0.00   42.91Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtnsda              42.91      7190.42       526.95      36024       2640sdb              36.93      6144.51       526.95      30784       2640md0             205.99     13213.57       517.37      66200       2592avg-cpu:  %user   %nice %system %iowait  %steal   %idle0.80    0.00    9.84   48.39    0.20   40.76Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtnsda              50.40      8369.48       573.49      41680       2856sdb              44.18      7318.88       573.49      36448       2856md0             241.77     15890.76       563.86      79136       2808avg-cpu:  %user   %nice %system %iowait  %steal   %idle0.60    0.00    8.38   44.91    0.00   46.11Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtnsda              50.10      7580.04       720.16      37976       3608sdb              47.50      7179.24       720.16      35968       3608md0             242.12     14558.08       710.58      72936       3560

The client said:

--- 60.0001 secs, 3792 fetches started, 3692 completed, 100 current--- 120 secs, 8778 fetches started, 8678 completed, 100 current10000 fetches, 100 max parallel, 1.024e+09 bytes, in 137.551 seconds102400 mean bytes/connection72.7004 fetches/sec, 7.44452e+06 bytes/secmsecs/connect: 66.9088 mean, 197.157 max, 0.223 minmsecs/first-response: 226.181 mean, 6066.96 max, 2.098 minHTTP response codes:code 200 -- 10000

Summary

Using Async IO allows lighttpd it overlapfile-operations. We send a IO-request for the file and get notifiedwhen it is ready. Instead of waiting for the file (as in the normalsendfile()) and blocking the server, we can handle other requestsinstead.

On the other side we give the kernel to reorder the file-requests as it wants to.

Taking this two improments we can increase the throughput by 80%.

Onthe other side we don’t spend any time in wait in lighty itself. 64kernel threads are handling the read()-calls for us in the backgroundwhich increases the idle-time from 12% to 40%, a improvement of 230% .