Friday 17 May 2013

Suricata transaction engine re-designed - Increased performance, better accuracy.


For quite sometime we wanted to improve the stateful detection engine inside suricata.  The previous detection engine although worked fine in a way, had its issues, some of them being these -

  • Repeated inspection of same app state, transactions included, and as a side effect of this we had the pattern matching engine carrying out redundant runs on already matched buffers.
  • FPs resulting from re-inspection of state.
  • FNs.
  • FPs from cross transaction matching.  For example the following sig would FP with the old engine for the below scenario.  Assume we have a flow with 2 requests -
        GET /one.html HTTP/1.1
        GET /two.html HTTP/1.1

        alert http any any -> any any (content:"one"; http_uri; content:"two"; http_uri; sid:1;)


We have now re-designed the way we carry out inspection(master branch), and all the above issues mentioned have disappeared, along with noticeably improved performance.

The corresponding commits being -

commit b0f014124dbf44829ba04ed9d090ff268f7cb0ae
Author: Anoop Saldanha <anoopsaldanha@gmail.com>
Date:   Fri May 3 20:34:58 2013 +0530

    Transaction engine redesigned.....

commit e71de3f98f713fd4fe6cbccf42c51e59b0fca848
Author: Anoop Saldanha <anoopsaldanha@gmail.com>
Date:   Fri May 3 10:03:48 2013 +0530

    Track transaction progress separately......

commit 6ebd360c225ccffab0ec65099e4f0b4882945b25
Author: Anoop Saldanha <anoopsaldanha@gmail.com>
Date:   Fri Apr 12 13:18:17 2013 +0530

    hsbd mpm and packet mpm share same mpm ctx id.....

Here are the stats from a private pcap containing 12826 http requests and which alerts 4033 times with the old engine.

1. The no of times the pattern matching engine was called on buffers

----------------------------------------
mpm - old engine : new engine
----------------------------------------
uri - 179k : 13k
http client body - 222 : 173
http header toserver - 179k : 13k
http header toclient - 174k : 117k
http method - 143k : 10.5k
http cookie - 10.5k : 6.5k
http raw uri - 143k : 10.5k

As you can see the pattern matching runs has drastically reduced.

2. Alert accuracy has been improved, with FPs and FN's disappearing.


-Default yaml-

Old engine - 40.5 seconds
New engine - 33.5 seconds
Performance increase  - 17.28%

I then modified the HOME_NET and EXTERNAL_NET to any, to increase the flows inspected by the engine, and to also increase the no of alerts.  These are the numbers obtained -

-Modified yaml-

Old engine - 70.5
New engine - 51.5
Performance increase - 27%

Suricata users with http heavy traffic and a fairly http heavy ruleset, should see the increase in performance as well.


Please do note the code update requires some rigorous testing, so keep an eye out for missed alerts, segvs and other bugs.  Any form of testing for alert, stability, and performance is appreciated.

=Future Work=

Effort continues to further improve the detection engine for better performance and a lot of cool new features.  Keep an eye out for our mailing list.