vectorscan: new package for speeding up regex ops

Vectorscan is fork of Hyperscan, a high-performance multiple regex
matching library. It follows the regular expression syntax of the
commonly-used libpcre library, but is a standalone library with
its own C API.

Currently ARM NEON/ASIMD and Power VSX are 100% functional. ARM
SVE2 support is in ongoing with access to hardware now. More
platforms will follow in the future.

The performance difference of snort3 compiled against this is
sizable for aarch64 confirmed on two different SoCs:

Test SoC #1 flogic/glinet_gl-mt6000
IDS mode:
Download speed wo/ vectorscan: 91.2 ±0.21 Mbit/s (n=3)
Download speed using vectorscan: 331.0 ±27.34 Mbit/s (n=3)
Gain of 3.6x

IPS mode:
Download speed wo/ vectorscan: 30.0 ±0.06 Mbit/s (n=3)
Download speed using vectorscan: 52.9 ±0.78 Mbit/s (n=3)
Gain of 1.8x

Notes:
* Data generated on snapshot build on 12-Apr-2024 using kernel
  6.6.26, snort 3.1.84.0, vectorscan 5.4.11.
* Speedtest script hitting the same server.
* Snort rules file of was 37,917 lines/22 MB.
* In all cases, single core CPU saturation occurred which
  speaks to the efficiency gains supplied by vectorscan.

Test Soc #2 bcm2712/RPi5B

IPS mode:
Download speed wo/ vectorscan: 164.3 ±0.64 Mbit/s (n=3)
Download speed using vectorscan: 232.8 ±0.26 Mbit/s (n=3)
Gain of 1.4x

Notes:
* Data generated on snapshot build on 13-Apr-2024 using kernel
  6.1.86, snort 3.1.84.0, vectorscan 5.4.11.
* Google fiber speedtest (https://fiber.google.com/speedtest/)
  hitting the same server.
* Snort rules contained 39,801 rules/22 MB.
* In all cases, single core CPU saturation occurred which
  speaks to the efficiency gains supplied by vectorscan.

Build system: x86/64
Build-tested: flogic/glinet_gl-mt6000, bcm2712/RPi5B, x86/64-glibc
Run-tested: flogic/glinet_gl-mt6000, bcm2712/RPi5B, x86/64-glibc (Intel N150 based box)

Co-authored-by: Tianling Shen <[email protected]>
Co-authored-by: Jeffery To <[email protected]>
Signed-off-by: John Audia <[email protected]>

diff --git a/libs/vectorscan/Makefile b/libs/vectorscan/Makefile
new file mode 100644 (file)
index 0000000..2cbfee2
--- /dev/null
+++ b/libs/vectorscan/Makefile
@@ -0,0 +1,92 @@
+#
+# This is free software, licensed under the GNU General Public License v2.
+# See /LICENSE for more information.
+#
+
+include $(TOPDIR)/rules.mk
+
+PKG_NAME:=vectorscan
+PKG_VERSION:=5.4.12
+PKG_RELEASE:=1
+
+PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.gz
+PKG_SOURCE_URL:=https://codeload.github.com/VectorCamp/vectorscan/tar.gz/$(PKG_NAME)/$(PKG_VERSION)?
+PKG_HASH:=1ac4f3c038ac163973f107ac4423a6b246b181ffd97fdd371696b2517ec9b3ed
+PKG_BUILD_DIR:=$(BUILD_DIR)/$(PKG_NAME)-$(PKG_NAME)-$(PKG_VERSION)
+
+PKG_MAINTAINER:=John Audia <[email protected]>
+PKG_LICENSE:=BSD-3-Clause BSD-2-Clause BSL-1.0
+PKG_LICENSE_FILES:=LICENSE
+CMAKE_INSTALL:=1
+PKG_BUILD_FLAGS:=no-lto
+PKG_BUILD_DEPENDS:=ragel/host python3/host boost/host
+
+include $(INCLUDE_DIR)/package.mk
+include $(INCLUDE_DIR)/cmake.mk
+
+DEPENDS_COMMON:=@(x86_64||aarch64)
+
+# With at least version 5.4.12, Neon/ASIMD is required for Arm support
+ifeq ($(CONFIG_CPU_NEON),)
+  PKG_BUILD_FLAGS := skip
+endif
+
+CMAKE_OPTIONS += \
+       -DCMAKE_INSTALL_PREFIX=/usr \
+       -DCMAKE_INSTALL_LIBDIR=lib \
+       -DBUILD_SHARED_LIBS=ON \
+       -Wno-dev
+
+ifeq ($(CONFIG_USE_GLIBC),y)
+       CMAKE_OPTIONS += -DFAT_RUNTIME=ON -DBUILD_BENCHMARKS=ON
+else
+       CMAKE_OPTIONS += -DFAT_RUNTIME=OFF -DBUILD_BENCHMARKS=OFF
+endif
+
+define Package/vectorscan-headers
+  CATEGORY:=Libraries
+  SECTION:=libs
+  TITLE:=Vectorscan Headers
+  URL:=https://github.com/VectorCamp/vectorscan
+  DEPENDS:= $(DEPENDS_COMMON)
+endef
+
+define Package/vectorscan-runtime
+  CATEGORY:=Libraries
+  SECTION:=libs
+  TITLE:=Vectorscan Runtime
+  URL:=https://github.com/VectorCamp/vectorscan
+  DEPENDS:= +libstdcpp +libsqlite3 $(DEPENDS_COMMON)
+endef
+
+define Package/vectorscan-headers/description
+  This package contains the headers for Vectorscan.
+  A fork of Intel's Hyperscan, modified to run on more platforms.
+endef
+
+define Package/vectorscan-runtime/description
+  This package contains the shared objects for Vectorscan.
+  A fork of Intel's Hyperscan, modified to run on more platforms.
+endef
+
+define Build/InstallDev
+       $(INSTALL_DIR) $(1)/usr/include/hs
+       $(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/include/hs/* $(1)/usr/include/hs/
+       $(INSTALL_DIR) $(1)/usr/lib
+       $(CP) $(PKG_INSTALL_DIR)/usr/lib/libhs* $(1)/usr/lib/
+       $(INSTALL_DIR) $(1)/usr/lib/pkgconfig
+       $(INSTALL_DATA) $(PKG_BUILD_DIR)/libhs.pc $(1)/usr/lib/pkgconfig/libhs.pc
+endef
+
+define Package/vectorscan-headers/install
+       $(INSTALL_DIR) $(1)/usr/include/hs
+       $(INSTALL_DATA) $(PKG_INSTALL_DIR)/usr/include/hs/*.h $(1)/usr/include/hs/
+endef
+
+define Package/vectorscan-runtime/install
+       $(INSTALL_DIR) $(1)/usr/lib
+       $(CP) $(PKG_INSTALL_DIR)/usr/lib/libhs* $(1)/usr/lib/
+endef
+
+$(eval $(call BuildPackage,vectorscan-headers))
+$(eval $(call BuildPackage,vectorscan-runtime))